Sometimes we have monitoring project results where a DQ check returns 1000 records because of so many rule failures. We want to limit the amount of records in the monitoring project export file for each DQ rule. If I only want to show X invalid samples per DQ check, how can I go about doing this? I initially thought of using a sequence step then filtering the final result, but this does not work because each rule needs its own sequence or count. Basically, I think I may need to group each DQ rule and apply a sequence to each group separately. This is how I am thinking of solving the problem. Any suggestions on what I can do? Thank you in advance for your help!
Solved
How to limit number of each DQ rule in invalid sample exports from One Desktop
- Universe Traveller
Best answer by AKislyakov
Are you sure it is Group Selector and not Group Aggregator.
Here is a sample Group Selector configuration. It selects up to 3 first (by Id) records from each group defined by the value in the ruleName column.
And test result.
- Ataccamer
Hi
I think Group Selector step might work for you.
Step description
This step creates groups of records based on specified key definitions. The step will select records from each group based on specified criteria. Records can be selected multiple times based on different criteria. The resulting records of selection criteria for each group may be dependent on sort order. For example when selecting the first N records, the sort order of the group determines which records will be selected.
Though depending on configuration records failing multiple rules at once might be considered a separate group and will get their own count.
- Universe Traveller
Hi
Thank you for the response! I’m not sure if Group Selector will work in this case. Unless I’m doing something incorrectly. See the image for what I’m trying to accomplish. Group Selector allows me to group things with a filter/expression. What I need is to have a sequence for each rule. So for rule1 it starts a sequence, but once it hits rule2, it should reset and so forth. Then the final step will be to add a filter so I can limit the amount of records per rule.
- Ataccamer
Are you sure it is Group Selector and not Group Aggregator.
Here is a sample Group Selector configuration. It selects up to 3 first (by Id) records from each group defined by the value in the ruleName column.
And test result.
- Universe Traveller
Reply
Login to the Ataccama Community
No account yet? Create an account
For Ataccama Customers and Partners
or
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.