How are you using Ataccama API’s in your project. I am explaining the most common scenario which we have in our project. I have a need to use Ataccama API part of data pipelines, i.e right after reading the source. execute the ataccama jobs. Per my understanding the Ataccama API is capable of doing below items thru API calls:
Running DQ Evaluation
Profiling
Run Catalog item
Run Monitoring etc
This will not be truly data pipeline usage, For eg) I wanted to run a DQ Evaluation and based on the result of the DQ Evaluation, I would like to capture the results Success/Failure + failed records(if there are any failures). Is it possible with Ataccama API’s? Have any one tried similar approach or if not want to understand the API usage in your projects
Best answer by anna.spakova
@KP_DQ In 12.6.2 there is something called IDE component that can be configured in the monitoring project to also post-process the DQ results, or you can configure a CSV file export. As for the IDE component, as far as I remember, there was this issue, that when a Project was published (e.g. after performing changes), the IDE configuration was erased. And as for API, there is a REST API and all the calls are described here: https://support.ataccama.com/home/docs/aip/12.6.1/one-web-application/configuration-reference/one-rest-api-reference
@Siva_Madhavan thank you very much for the feedback, we are aware of the current limitations the post-processing components have and our product team is working on improvements for future versions.
@Siva_Madhavan - I am also looking for similar dataset . I’m interested in failed detailed records with rule instance name. Pls let me know, If you already found the way to extract . TIA
Hello @Siva_Madhavan , if I correctly understand your need - you would like to process detailed results from a monitoring project (DQ evaluation). There is an option to create a post-processing component (https://support.ataccama.com/home/docs/aip/latest/development-guides/one-desktop-development-guide/working-with-ataccama-one-platform/post-processing-plans) that is triggered after DQ evaluation and will give you all records of a monitored data set with information about failed DQ checks, which you can then process as you desire (export the failed records into CSV/database, create a summary report, ...). There is no other way how to obtain these detailed results on a record level.
Any other information (like the aggregated DQ results for instance), can be obtained using the Ataccama API (https://support.ataccama.com/home/docs/aip/latest/development-guides/one-development-guide/one-api) as you mention - you can use either the ONE metadata reader steps in ONE Desktop or more general GraphQL, based on your specific needs. API can also be used to trigger actions like monitoring, profiling etc.
Thanks @anna.spakova . yes that answered my question. However, I would like to highlight,this is limitation with the tool & a lack of feature for API. The need of API arise when people want to use for their data engineering purpose & truly we are redirected to use Post processing step in this case.
Secondly, Post processing plans are not built correct, the option doesnt export table names, Monitoring project etc and the CSV generated is not in human readable format and it includes both success & failure without proper format which needs lot of manual work.
@KP_DQ In 12.6.2 there is something called IDE component that can be configured in the monitoring project to also post-process the DQ results, or you can configure a CSV file export. As for the IDE component, as far as I remember, there was this issue, that when a Project was published (e.g. after performing changes), the IDE configuration was erased. And as for API, there is a REST API and all the calls are described here: https://support.ataccama.com/home/docs/aip/12.6.1/one-web-application/configuration-reference/one-rest-api-reference
@Siva_Madhavan thank you very much for the feedback, we are aware of the current limitations the post-processing components have and our product team is working on improvements for future versions.
I enabled the IDE in the reports section in the project and it looks like below “IDE integration Status To be configured in IDE”.
In IDE ,connected to Ataccama one web sever and navigated the project and able to change the output report by replacing “trash” to “output file writer” . In output file writer , I have specified the local system path (i.e- one of the path from my computer path).
Saved and went back to one web and published the project and ran manually.
nothing happened. Still the In the project report section- I am seeing the IDE status as still “IDE integration Status To be configured in IDE”.
When I ran the project manually nothing written on the file.
Am I missing any steps?. Pls help with detailed steps. Thanks in Advance
Hello @KP_DQ , you are trying to do this locally or on some instance installed in your company? Because the path has to be reachable by the server where the runtime is installed. Can I ask you to reach to our support? https://support.ataccama.com/ They can navigate you better as I don’t remember the exact place where to configure the shortcuts.
We use 3 different kinds of cookies. You can choose which cookies you want to accept. We need basic cookies to make this site work, therefore these are the minimum you can select. Learn more about our cookies.
