I am working on the usecase of comparing two different columns in two different csv files. I need to compare these column, remove any duplicates and merge them into single column. I am trying to use join component but not sure about this approach will work or any better solution is available to achieve this use case - If anyone can suggest a better apprach it will be helpful. - Thanks.
Solved
Compare two different columns from two different files
- Data Pioneer
- 12 replies
Best answer by AKislyakov
Hi
Join will work, especially if you have any additional columns, that you need to bring to resulting data set. For this case you need to use Join Type = OTHER to preserve un-matched records from both inputs.
Another approach, especially if you need to remove duplicates within input data files as well is to use Union step to join two sets together, followed by a Group Aggregator step, to remove duplicates. Here is an example of how result may look like based on 05.06 Union.plan from the Tutorials project.
- Data Pioneer
- 12 replies
The output will be coming from two different multiplicator which consists of those two columns.
- Ataccamer
- 150 replies
- Answer
Hi
Join will work, especially if you have any additional columns, that you need to bring to resulting data set. For this case you need to use Join Type = OTHER to preserve un-matched records from both inputs.
Another approach, especially if you need to remove duplicates within input data files as well is to use Union step to join two sets together, followed by a Group Aggregator step, to remove duplicates. Here is an example of how result may look like based on 05.06 Union.plan from the Tutorials project.
- Data Pioneer
- 12 replies
+3- Community Manager
- 635 replies
Hello
- Data Pioneer
- 12 replies
Reply
Most Liked this week
Login to the Ataccama Community
No account yet? Create an account
For Ataccama Customers and Partners
or
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.