Solved

Duplicate detection with aggregation results config

  • 26 April 2024
  • 1 reply
  • 21 views

  • Data Voyager
  • 0 replies

I have created a rule that detects if a record is in the table more than once based on aggregation of 2 attributes in OneWeb application.  This gives a proper result in Invalid samples data and exports giving 2 errors if the record is in the table 2 times etc.  But the customer requires that only 1 record should be visible in Invalid samples and also in exporting to db.  It does not matter which one.  Is it possible to configure the result to do this?

icon

Best answer by joyce 26 April 2024, 18:36

View original

1 reply

Userlevel 3
Badge +2

Hi @Mel! If you’ve configured a ONE Desktop plan (i.e. post-processing plan) to export the invalid records to a database, you can add a Representative Creator or Group Aggregator step to group based on the two attributes to ensure one instance for each duplicate record. For example, if I’m grouping on code + id I will add those columns under the Group By section of the Representative Creator. The out_merged stream will include the new grouped records.

For more information on how the Representative Creator or Group Aggregator works, select the magnifying glass icon (top right) & type in “Representative Creator” to search through the ONE Desktop documentation. 

If you’re not using a plan to export the invalid results to a database, please elaborate more on how you’re exporting to the database. Let me know if you have any questions!

Reply