We are using dBT for our data pipelines, and for the tables and fields, there is description maintained in dBT that we would like to export (json) and update Ataccama Data Catalog.
Is there a known, trusted, supported way to automate this?
I already do have a desktop plan i created that can update catalog objects and attributes based on a excel input file. I may be able to modify that if thats the only option. Iam reaching out to the community to see if there is a preferred way to get descriptions and documentation from dBT to Ataccama.
Hi Prasad,
You should be able to update the descriptions for catalog items and attributes using Metadata Writer step.
First you would need to read the ID’s and names for all the existing CI’s and Attributes using Metadata Reader step, then you can join the metadata from the platform with another data stream from dbt which would contain the names of CI’s and attributes and relevant descriptions and then you would update existing CI’s and Attributes using Metadata Writer step.
Below are examples of the configuration of Metadata Reader\Writer steps:
And here’s a high level example of how the plan logic might look like:
Updating CI descriptions is rather straight forward task but in case of attributes you need to make sure that you reference the right parent catalogItemId for each attribute id.
To parse the data in json format coming from dbt you should be able to use something like Json Reader or Json Parser step in IDE.
I hope this will be helpful.
Ivan
