Skip to main content
Solved

How to upload DQ results from my monitoring project into a database?


Forum|alt.badge.img+1

Hello everyone!

I created a VCI (Virtual Catolog Item) and created a Monitoring Project with Rules in Atacama One, using a Post Processing Plan I wanted to upload the result of the monitoring project to databases, in Atacama One we have a csv file with the results in the Export tab, but I can't upload it to the database, the database is empty. Сreated a Global Runtime Configuration, but there is no result. Who has faced such a problem?

Best answer by pdanpoonkij

Hello, I’m not sure if this has already been resolved, but there are a couple of ways you can write data to a database from within the post-processing plan. You should start by replacing your text file writer step with a JDBC writer step (similar to the screenshot below).

In the post-processing plans, you can use the connections defined in the Ataccama ONE web application. Please see the screenshots below, where we navigate to the Ataccama One Source connection page. Here, you can get the connection id from the URL (as highlighted). And then, you can use this id as a Data Source Name in the JDBC Writer step.

Don’t worry about the red X icon here as the plan will be execute from Ataccama One and it will recognize this id there.

Hopefully this helps you 😀

View original
Did this topic help you find an answer to your question?

11 replies

Cansu
Community Manager
Forum|alt.badge.img+3
  • Community Manager
  • 625 replies
  • April 3, 2023

Hi @FCB developer , thank you for posting! I’m checking your question with the team and will be back with an update 👍


Forum|alt.badge.img+1
  • Author
  • Data Pioneer
  • 31 replies
  • April 4, 2023

Hello @Cansu, thanks for the feedback! I'll be looking forward to it.


Cansu
Community Manager
Forum|alt.badge.img+3
  • Community Manager
  • 625 replies
  • April 4, 2023

Hi again @FCB developer the team needs more information on your set up so would you be able to create a support ticket, so that they can take a closer look into your question? Thank you!


Forum|alt.badge.img+1
  • Author
  • Data Pioneer
  • 31 replies
  • April 4, 2023

Good day @Cansu, no problem, thanks for the advice and answer👍🏻


Forum|alt.badge.img+3

Did you change you Post processing plan output to DB Object and updated the runtimeconfig file? if yes, then restart DPM


Forum|alt.badge.img+4
  • Data Pioneer L2
  • 28 replies
  • Answer
  • April 12, 2023

Hello, I’m not sure if this has already been resolved, but there are a couple of ways you can write data to a database from within the post-processing plan. You should start by replacing your text file writer step with a JDBC writer step (similar to the screenshot below).

In the post-processing plans, you can use the connections defined in the Ataccama ONE web application. Please see the screenshots below, where we navigate to the Ataccama One Source connection page. Here, you can get the connection id from the URL (as highlighted). And then, you can use this id as a Data Source Name in the JDBC Writer step.

Don’t worry about the red X icon here as the plan will be execute from Ataccama One and it will recognize this id there.

Hopefully this helps you 😀


Forum|alt.badge.img+1
  • Author
  • Data Pioneer
  • 31 replies
  • April 18, 2023

@pdanpoonkij Good day! It helped me a lot, thank you!😃


Forum|alt.badge.img+1

A post processing component has a different deployment model to main pipeline code deployed via git. If you want to write just one main pipeline to read all monitoring projects’ results and write them to a database it can be done, albeit complicated, via a ONE Metadata Reader.

 

 


  • Data Enthusiast
  • 3 replies
  • August 1, 2024

Hi @pdanpoonkij , thank you for your response on this as I was facing a similar issue as well exporting issues from post-processing plans to the database. Does the step you indicated require updating the runtime config file and restarting DPM server?


Forum|alt.badge.img+4
  • Data Pioneer L2
  • 28 replies
  • August 2, 2024

Hello @mbiteye if you specify the data source name in the JDBC writer with a UUID (as shown in my previous comment), there's no need to do anything with DPM as the data source detail is already on Ataccama One. However, if you choose to specify the data source name with DB Object name, then you'll have to update the DPM's runtime config with this data source detail and give it a restart. Let me know if this helps or if you need more details on the latter approach.


  • Data Enthusiast
  • 3 replies
  • August 2, 2024
pdanpoonkij wrote:

Hello @mbiteye if you specify the data source name in the JDBC writer with a UUID (as shown in my previous comment), there's no need to do anything with DPM as the data source detail is already on Ataccama One. However, if you choose to specify the data source name with DB Object name, then you'll have to update the DPM's runtime config with this data source detail and give it a restart. Let me know if this helps or if you need more details on the latter approach.

Thank you for the prompt response, this was really helpful!


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings