Question

do we have row level quality score?

  • 5 January 2023
  • 5 replies
  • 39 views
K
Userlevel 3
Badge +3

I see column level quality score in monitoring project. Do we have row level quality score?

5 replies

K
Userlevel 3
Badge +3

@anna.spakova 

anna.spakova
Userlevel 4
Badge +7

Hello @KarthikaSPillai ,

 

you can see the row level quality (FAIL/SUCCESS) when you export the whole data set using post-processing components. The export tells you if a row satisfied all DQ checks (SUCCESS) or if it failed for at least one of the DQ checks (FAIL). Otherwise in the web app you can always see only aggregated DQ results. (with exception of the samples).

https://support.ataccama.com/home/docs/aip/latest/development-guides/one-desktop-development-guide/working-with-ataccama-one-platform/post-processing-plans

Kind regards,

Anna

Maxim Kim
Userlevel 3
Badge +6

Bear in mind that post-processing components are per each catalog-item in the MP...

K
Userlevel 3
Badge +3

Thanks @anna.spakova and @Maxim Kim My requirement is we have to send only valid records ( valid row only when ALL columns in a row is a success) to output DB. I am looking for a solution to filter the rows with only valid data.

anna.spakova
Userlevel 4
Badge +7

@KarthikaSPillai This can be done using the post-processing component. The component is in fact a plan in ONE Desktop where you can use all the functionality it offers - including filtering etc. The incoming data in the post-processing component contain three additional columns (besides your data):

  • valid_rules (list of DQ rules the record successed)
  • invalid_rules (list of DQ rules the record failed)
  • invalid_rules_explanation (explanation messages for the failed DQ rules).

It is then enough to use filter like: invalid_rules is null to get only records with no DQ issues.

Reply


Terms & ConditionsCookie settings