Welcome to the first blog in a series exploring the concept and implementation of a Metadata Warehouse, hereafter referred to as MDWH.
You are most likely already familiar with concepts that align with an MDWH, such as a data warehouse or a data lake. An MDWH is quite similar to a data warehouse, but focuses on exported metadata stored in another database—specifically, Ataccama’s metadata. In this scenario, there is no need to integrate data from multiple systems, because there is only one Ataccama implementation. (After all, why use more than one metadata application?😉)
What are the drivers for having an MDWH?
At Stater, our MDWH serves the following purposes:
- Reuse of metadata
Ataccama is the golden source for metadata, so when details are required in different applications, they should not differ from those in Ataccama. Examples where reusing metadata is essential include- Data models
Many data models contain common entities/tables and attributes/columns, both across models and within a single model. These models have metadata such as definitions. Rather than facing the major challenge of keeping metadata consistent everywhere, it is easier to have definitions in one place and reuse them. - Data products
For some of our data products, metadata is provided. Naturally, our clients expect the same definitions, etc., if a data element appears in multiple data products.. - Contracts accompanying APIs
The same applies to contracts that accompany APIs, where definitions and other metadata are provided.
- Data models
- Easy access to this metadata
One way to query metadata is via APIs using GraphQL. For most users, this is a bridge too far, as you must understand the underlying data structure and, perhaps more importantly, GraphQL is not easy to learn.
Plain SQL, on the other hand, has been widely adopted as a simple query language for decades and provides far more functionality.
- (Meta)data governance
Metadata management contributes to data governance, ensuring you can demonstrate control over the health of your data. Implicitly, the quality of your metadata must also be sufficient. And which tool is great at measuring data quality? Right: Ataccama! So this requires a data source that contains metadata, which you can define as a source in Ataccama.
In the following articles, I will describe various topics related to building the MDWH, so stay tuned!