How to upload DQ results from my monitoring project into a database?
Hello everyone!
I created a VCI (Virtual Catolog Item) and created a Monitoring Project with Rules in Atacama One, using a Post Processing Plan I wanted to upload the result of the monitoring project to databases, in Atacama One we have a csv file with the results in the Export tab, but I can't upload it to the database, the database is empty. Сreated a Global Runtime Configuration, but there is no result. Who has faced such a problem?
Page 1 / 1
Hi @FCB developer , thank you for posting! I’m checking your question with the team and will be back with an update
Hello @Cansu, thanks for the feedback! I'll be looking forward to it.
Hi again @FCB developer the team needs more information on your set up so would you be able to create a support ticket, so that they can take a closer look into your question? Thank you!
Good day @Cansu, no problem, thanks for the advice and answer
Did you change you Post processing plan output to DB Object and updated the runtimeconfig file? if yes, then restart DPM
Hello, I’m not sure if this has already been resolved, but there are a couple of ways you can write data to a database from within the post-processing plan. You should start by replacing your text file writer step with a JDBC writer step (similar to the screenshot below).
In the post-processing plans, you can use the connections defined in the Ataccama ONE web application. Please see the screenshots below, where we navigate to the Ataccama One Source connection page. Here, you can get the connection id from the URL (as highlighted). And then, you can use this id as a Data Source Name in the JDBC Writer step.
Don’t worry about the red X icon here as the plan will be execute from Ataccama One and it will recognize this id there.
Hopefully this helps you
@pdanpoonkij Good day! It helped me a lot, thank you!
A post processing component has a different deployment model to main pipeline code deployed via git. If you want to write just one main pipeline to read all monitoring projects’ results and write them to a database it can be done, albeit complicated, via a ONE Metadata Reader.
Hi @pdanpoonkij , thank you for your response on this as I was facing a similar issue as well exporting issues from post-processing plans to the database. Does the step you indicated require updating the runtime config file and restarting DPM server?
Hello @mbiteye if you specify the data source name in the JDBC writer with a UUID (as shown in my previous comment), there's no need to do anything with DPM as the data source detail is already on Ataccama One. However, if you choose to specify the data source name with DB Object name, then you'll have to update the DPM's runtime config with this data source detail and give it a restart. Let me know if this helps or if you need more details on the latter approach.
Hello @mbiteye if you specify the data source name in the JDBC writer with a UUID (as shown in my previous comment), there's no need to do anything with DPM as the data source detail is already on Ataccama One. However, if you choose to specify the data source name with DB Object name, then you'll have to update the DPM's runtime config with this data source detail and give it a restart. Let me know if this helps or if you need more details on the latter approach.
Thank you for the prompt response, this was really helpful!