Hi Team,
I have few data loads that read data using API’s and writing to Snowflake tables. Often the data is incomplete & inconsistent. Im checking if I can use Attacama to compare the data between different snowflake schema’s : eg) Raw vs Curated? I understand we can write SQL Execute (select col1 from Tab A minus select col1 from Tab b and create a plan. But i have 500+ jobs and needs a lot of plans to be created. Looking for correct way of doing this?
Best answer by Vani
Hello
I understand that you need to compare the data in table between Schema A and B for 500 different data sets. As you already knew ,the parameterization may not work in SQL Execute to pass the entire SQL query. As a workaround, we may need to store the queries in a master lookup table/file and create the component for each dataset dynamically using a step called “Complex XML writer” step which would require some support from the Ataccama consultant. So I would suggest you to reach out to your Engagement Manager for further discussion about this use case.
Hello
I understand that you need to compare the data in table between Schema A and B for 500 different data sets. As you already knew ,the parameterization may not work in SQL Execute to pass the entire SQL query. As a workaround, we may need to store the queries in a master lookup table/file and create the component for each dataset dynamically using a step called “Complex XML writer” step which would require some support from the Ataccama consultant. So I would suggest you to reach out to your Engagement Manager for further discussion about this use case.
Thanks Vani for your reply.
I wish there is an sample plan in Ataccama IDE Tutorial’s for ‘Complex XML Writer” but thanks for highlighting this STEP ill do further research.
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
