Data Quality Monitoring Project Result Download

@jbrant , if your organisation has s3 bucket, ADLS Gen2 storage, or google cloud storage, I’d recommend trying to save the file directly over to those locations. This way you have your own dedicated space, and you won’t need to store things alongside critical application components in minio.

I personally prefer to touch minio as little as possible - if you accidentally move / delete a component on minio it could have very bad consequences.

I am currently trying to retrieve files from the Minio storage. In Desktop I have a workflow which gets a file from Minio using the above method, which works as expected. I've exported the runtimeConfig from desktop and put it on the orchestration server. When I trigger the workflow on the server, the workflow can't find the file and gives the following error:
 

As I mentioned, in One desktop it works as expected with the exact same configuration of server and workflow.

​@anna.spakova You mentioned it can be used for automation on the orchestration server, so I assumed this would work.

Are there any other settings I have to take into account? Or any changes to the runtime config to make it useable on the runtime server perhaps?

I could think of 2 things to check:

1. Is the minio you have set up on your ONE Desktop the same environment you have set up for the runtime server?
2. Is the file still there?

I’m inclined to believe that the minio connection on the runtime server is a valid connection, otherwise it would say connection not found. So it’s connected to some minio.

Also, when you tested it to “retrieve” the file, was it a MOVE or a COPY? Just in case you ran the workflow on ONE Desktop and it MOVE the file which means the file won’t be there anymore when you test it on the server?

You should be able to verify the file is there on your ONE Desktop (connect to minio on ONE Desktop and browse, or connect to minio on web browser), then trigger the workflow on runtime server.

EDIT: There was a typo in input of the variable and I just kept on copy-pasting the same input. Everything works as expected.

Hi May,

Thanks for your response.

1. Minio is indeed set to the same environment. Double checked everything just to be sure, but the orchestration server is the same env. as the minio server. Also checked port connections from orchestration server to minio server, which are all good.
2. The file is still there, I'm just using copy. 

While typing my response I seem to have found the issue. Leaving my answers in here though.

It seems that variables in the workflow are not supported (?)

The workflow is set up to be able to iterate over several files on a daily basis, so I have a variable for the date and a variable for the filename. This makes the defined path in the workflow as follows: 

resource://MINIO/export/${today}/${filename}.csv

When the workflow fails, it does however show the correct path in the error message, which would imply that the variables work as intended, so I still don't quite understand why it fails.