Skip to main content
Solved

Ataccama Desktop Iterate Tool Erros

B
Data Voyager
Forum|alt.badge.img

Hello!
Does anyone have experience using the Iterate tool within the Ataccama Desktop, when creating a workflow? I’ve found a couple of old community posts referring to it, along with the files they’ve used, but I can’t get it to work in my environment. see the following:

Iterating rows of a table (Record-by-record - SQL Iterator) in a workflow

Based on that post and my testing, it seems like the Iterate tool needs to call another workflow, so I’m calling another .ewf file that just has the Run DQC tool, which runs the .plan file. I’m able to succeed in running the .plan file from the Run DQC .ewf, however i can’t seem to get the Iterate tool to run, without getting Configuration errors, like the following:

12.03.2025 15:30:51 [INFO]     Task 'Iterate' : failed with exception: com.ataccama.adt.internal.core.runtime.EwfConfigException: Configuration contains errors, exiting
java.lang.RuntimeException: com.ataccama.adt.internal.core.runtime.EwfConfigException: Configuration contains errors, exiting
	at com.ataccama.adt.runtime.ProcessorWorkflowInvocationService.runWorkflow(ProcessorWorkflowInvocationService.java:55)
	at com.ataccama.adt.task.exec.EwfForEachTaskInstance$SerialExecutionStrategy.run(EwfForEachTaskInstance.java:143)
	at com.ataccama.adt.task.exec.EwfForEachTaskInstance.run(EwfForEachTaskInstance.java:84)
	at com.ataccama.adt.internal.core.runtime.EwfTaskProcessor$ThreadSlot.run(EwfTaskProcessor.java:372)
Caused by: com.ataccama.adt.internal.core.runtime.EwfConfigException: Configuration contains errors, exiting
	at com.ataccama.adt.internal.core.runtime.EwfWorkflowTools.createWorkflowRunThread(EwfWorkflowTools.java:222)
	at com.ataccama.adt.runtime.ProcessorWorkflowInvocationService.runWorkflowAsync(ProcessorWorkflowInvocationService.java:83)
	at com.ataccama.adt.runtime.ProcessorWorkflowInvocationService.runWorkflow(ProcessorWorkflowInvocationService.java:53)

Ideally i would be passing files through to iterate on, but I’ve been unsuccessful in getting that to work, or even work when not passing anything through the iterate, just iterating at all. I’ve tried iterating on Set and Files based on the tooltips available, but never been able to get this to work!
If anyone has had success with this, please let me know!

Best answer by Adrian Anderson

Hi ​@bobparry

Thanks heaps for sharing the config files. I think I now understand what you're trying to achieve.

Hopefully, the answer is a bit simpler than it initially looked.

But first a few follow up questions/clarifications:
1. The updates that we want to apply to the Catalog Items are in separate XML files. Is this correct?
2. How are the XML files produced and how do they arrive into the filesystem?
3. In the provided examples, the only difference between the three component (.comp) files is the Xml Reader step's File Name. Is this correct?
4. In the lookup (.lkp) files, are the Code values coming from ONE? If so, it might be easier/more dynamic to use additional ONE Metadata Reader to retrieve the codes from ONE in real-time. This way the same plan can be used across multiple environments without needing to recreate any lookup files.

To process a folder of (XML) files, we can use a workflow to iterate over each XML input file and using a single parameterized component that accepts the XML filename as input.

Attached (community_dq_1670.zip) is one way of doing this.

Depending on how and where the XML files are generated, and how the lookup values are managed, this process could be simplified further. 

Kind regards,
Adrian

View original

A
Ataccamer
Forum|alt.badge.img+1

Hi ​@bobparry .

Iterating workflows can be a bit tricky to set up the first couple of times for sure.

There are some helpful example tutorials built into ONE Desktop.

Have you had a chance to take a look at these?

 

Let us know if this helps.

Kind regards,
Adrian

B
Data Voyager
Forum|alt.badge.img
  • bobparryData Voyager
  • Data Voyager
  • March 17, 2025

Hi Adiran!
Thanks for this, I’d never seen that part of the Tutorials before! That clarifies where i was going wrong, my assumption was I could loop through a single .comp file many times with a range of inputs, but from how the tutorial is set up this is more like a batch file, limited to just iterating through a series of .comp files. Is that correct? 

Thanks again, huge help!
Best, 

A
Ataccamer
Forum|alt.badge.img+1

The workflows can be configured very flexibly, especially when parameters are used effectively.

These parameters can be used inside components as column values and even column names, allowing dynamic replacement inside expressions and most steps configurations. Note: this doesn’t work in plans.

A couple of non-obvious hints:

  1. To set parameters in workflow, double-click on an empty part of the workflow’s canvas or right-click and select Edit properties…
  2. To use parameters in a step’s configuration in components, right-click on the name of the setting and select Map as Parameter.

There are other options as well but, hopefully, this helps for now.

If not, could you please provide more details about what you’re trying to achieve?

Kind regards,
Adrian

B
Data Voyager
Forum|alt.badge.img
  • bobparryData Voyager
  • Data Voyager
  • March 18, 2025

Hi Adrian, 

Thanks for this, i was able to replicate your steps, but I’m not sure how to iterate through values in the parameter. 

For context, the overall goal is just to pass metadata (titles, descriptions, etc) from our source systems through to Ataccama One, allowing users to update their metadata at source, and see that same metadata reflected in the Data Catalogue. Our metadata is stored in XMLs, associated with every item in the catalogue. Using the methodology from this community post: importing catalog item descriptions and attribute descriptions from dbT docs, i can join the metadata from the XML to the Data Catalogue item that already exists, and write to Ataccama One. 

 

However, i have thousands of catalog items with metadata stored this way, and didn’t want to manually create each plan, nor run each plan individually. The plan as of now, based on what you’ve said about iterating above, is to have a unique .comp file for each catalogue item, store all the comp files in the same folder, and iterate through them with the Iterate tool. 

To make all of the .comp files, I’m just going to duplicate a template .comp file, and edit the JSON of each .comp file, replacing the name and .XML file it reads, to match each of the XMLs in a separate XML folder. I think we can do all of that using python, but that’s where I’m testing now. 

If there is a way to iterate through the same plan, using the input .XML as a parameter, and passing the file name of each the XMLs through the parameter, and iterate the one plan for as many XMLs as there are in the XML folder, that would be easier, but based on the above instructions/what I’ve seen online I don’t know if that is possible. 

I’ve attached a zip file with the .ewf files, and a sample of 3 .comp files they refer to, and the XMLs they write to for your reference. It refers to ID’s for our workspace/is connecting to our instance of Ataccama One, but otherwise should work on your end. Once configured for an environment you can work with, just running the SDWMetadataIterator.ewf should run through each of the .comp files and update the metadata for them. 

Let me know if there’s a better or more efficient way of doing this!
Best, 

 

A
Ataccamer
Forum|alt.badge.img+1

Hi ​@bobparry

Thanks heaps for sharing the config files. I think I now understand what you're trying to achieve.

Hopefully, the answer is a bit simpler than it initially looked.

But first a few follow up questions/clarifications:
1. The updates that we want to apply to the Catalog Items are in separate XML files. Is this correct?
2. How are the XML files produced and how do they arrive into the filesystem?
3. In the provided examples, the only difference between the three component (.comp) files is the Xml Reader step's File Name. Is this correct?
4. In the lookup (.lkp) files, are the Code values coming from ONE? If so, it might be easier/more dynamic to use additional ONE Metadata Reader to retrieve the codes from ONE in real-time. This way the same plan can be used across multiple environments without needing to recreate any lookup files.

To process a folder of (XML) files, we can use a workflow to iterate over each XML input file and using a single parameterized component that accepts the XML filename as input.

Attached (community_dq_1670.zip) is one way of doing this.

Depending on how and where the XML files are generated, and how the lookup values are managed, this process could be simplified further. 

Kind regards,
Adrian

B
Data Voyager
Forum|alt.badge.img
  • bobparryData Voyager
  • Data Voyager
  • March 21, 2025

Hi Adrian,

Huge improvements on mine, work on my end as well! Thanks so much!

  1. The updates are all in XML files, however those files are just from a table in a MSSQL Database, so I think I can switch the iterator to a Sql Row iterator, and pull them from there? I’d only tried to do everything the XML’s in a folder as I thought I’d need to for the iterator to work, so that would save additional steps.
  2. The XMLs are produced through a spatial data application called ArcGIS Pro, its how our organization manages and works with spatial data. They have a catalog interface that allows users to enter metadata for any table, and that metadata is then saved to an XML corresponding to the table.
  3. Correct! The only change was the input file, using your technique it works. I had tried doing more or less what you had done but I was putting the parameter name in the File Name entry box in the XML Reader, rather than the .comp name, hadn’t occurred to me that would be the way this works.
  4. Great idea on having the Lookups saved in ONE, and that means I only have to update the lookup in one location as we use the same lookup in a few dq rules as well.

I won’t be able to share the connection string/credentials for the database, but if I can get that working I’ll share again with the database connection removed for others to use as a template. Any other Ataccama customer who store spatial metadata within an ArcGIS system should be able to use this! 
As well I’ll be out of office for the next few weeks, but I’ll post an update when back. Thanks again Adrian!

A
Ataccamer
Forum|alt.badge.img+1

Hi Bob.

Glad to hear we're heading in the right direction with this.

As mentioned, depending on where the XML is sourced from, the entire process could be simplified further.

If the XML is being read from a DB, and the process executed from Desktop, then workflows aren't required at all.

In the component, simply replace the Xml Reader step with a Jdbc Reader step followed by an Xml Parser step (see example screenshot). The iteration now occurs in the Jdbc Reader step.

Note: The Xml Parser step's config will be almost identical to the Xml Reader's, just needing to change the Data Stream name from 'out' to anything else, e.g., 'metadata'.

Enjoy your break!

Kind regards,
Adrian

Reply


Community Terms of Use & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings