Skip to main content

I see column level quality score in monitoring project. Do we have row level quality score?

@anna.spakova 


Hello @KarthikaSPillai ,

 

you can see the row level quality (FAIL/SUCCESS) when you export the whole data set using post-processing components. The export tells you if a row satisfied all DQ checks (SUCCESS) or if it failed for at least one of the DQ checks (FAIL). Otherwise in the web app you can always see only aggregated DQ results. (with exception of the samples).

https://support.ataccama.com/home/docs/aip/latest/development-guides/one-desktop-development-guide/working-with-ataccama-one-platform/post-processing-plans

Kind regards,

Anna


Bear in mind that post-processing components are per each catalog-item in the MP...


Thanks @anna.spakova and @Maxim Kim My requirement is we have to send only valid records ( valid row only when ALL columns in a row is a success) to output DB. I am looking for a solution to filter the rows with only valid data.


@KarthikaSPillai This can be done using the post-processing component. The component is in fact a plan in ONE Desktop where you can use all the functionality it offers - including filtering etc. The incoming data in the post-processing component contain three additional columns (besides your data):

  • valid_rules (list of DQ rules the record successed)
  • invalid_rules (list of DQ rules the record failed)
  • invalid_rules_explanation (explanation messages for the failed DQ rules).

It is then enough to use filter like: invalid_rules is null to get only records with no DQ issues.


Reply