Skip to main content
Solved

Data Quality Monitoring Project Result Download


Forum|alt.badge.img
  • Universe Traveller
  • 2 replies

I have a data quality monitoring project to which I’ve deployed a post-processing component. Results are written to a CSV file, which I presume is stored within Ataccama’s object storage. When I navigate to the data quality monitoring project and select the Export tab, I see the below screenshot, which appears to include a file path along with a download button. 

I’m trying to determine how to automate download of the below-depicted CSV file. Is there a way to determine the fully-qualified path to the file and download from Ataccama server?

 

Best answer by anna.spakova

Hello @jbrant , the files are stored in Minio, which is a mandatory component installed with the platform. Usually, the UI is accessible by https://minio-console.<rest of the url to one>.  You should have the link and credentials from the platform onboarding. If not, please contact your Ataccama contact person or our support (in case of the PaaS).

You can then automate the download using e.g. the orchestration server. You would define the connection to Minio as a server connection and use that in a plan to read the data from the files.

You can define the connection to minio in ONE Desktop:

and then use it with resource://minio/export/<rest of the path to the data> in a file path in the Text File Reader step. The bucket in your case is the shared one.

Let me know if this helps or if you have further questions.

Kind regards,

Anna

View original
Did this topic help you find an answer to your question?

2 replies

anna.spakova
Ataccamer
Forum|alt.badge.img+3
  • Ataccamer
  • 144 replies
  • Answer
  • March 12, 2024

Hello @jbrant , the files are stored in Minio, which is a mandatory component installed with the platform. Usually, the UI is accessible by https://minio-console.<rest of the url to one>.  You should have the link and credentials from the platform onboarding. If not, please contact your Ataccama contact person or our support (in case of the PaaS).

You can then automate the download using e.g. the orchestration server. You would define the connection to Minio as a server connection and use that in a plan to read the data from the files.

You can define the connection to minio in ONE Desktop:

and then use it with resource://minio/export/<rest of the path to the data> in a file path in the Text File Reader step. The bucket in your case is the shared one.

Let me know if this helps or if you have further questions.

Kind regards,

Anna


may_kwok
Star Blazer L3
Forum|alt.badge.img+4
  • Star Blazer L3
  • 82 replies
  • March 12, 2024

@jbrant , if your organisation has s3 bucket, ADLS Gen2 storage, or google cloud storage, I’d recommend trying to save the file directly over to those locations. This way you have your own dedicated space, and you won’t need to store things alongside critical application components in minio.

I personally prefer to touch minio as little as possible - if you accidentally move / delete a component on minio it could have very bad consequences.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings