Skip to main content
Solved

do we have row level quality score?

K
Data Pioneer
Forum|alt.badge.img+1

I see column level quality score in monitoring project. Do we have row level quality score?

Best answer by anna.spakova

@KarthikaSPillai This can be done using the post-processing component. The component is in fact a plan in ONE Desktop where you can use all the functionality it offers - including filtering etc. The incoming data in the post-processing component contain three additional columns (besides your data):

  • valid_rules (list of DQ rules the record successed)
  • invalid_rules (list of DQ rules the record failed)
  • invalid_rules_explanation (explanation messages for the failed DQ rules).

It is then enough to use filter like: invalid_rules is null to get only records with no DQ issues.

View original

K
Data Pioneer
Forum|alt.badge.img+1

@anna.spakova 

anna.spakova
Ataccamer
Forum|alt.badge.img+3

Hello @KarthikaSPillai ,

 

you can see the row level quality (FAIL/SUCCESS) when you export the whole data set using post-processing components. The export tells you if a row satisfied all DQ checks (SUCCESS) or if it failed for at least one of the DQ checks (FAIL). Otherwise in the web app you can always see only aggregated DQ results. (with exception of the samples).

https://support.ataccama.com/home/docs/aip/latest/development-guides/one-desktop-development-guide/working-with-ataccama-one-platform/post-processing-plans

Kind regards,

Anna

DannyRyan
Maxim Kim
2 people like this
  • DannyRyan
    DannyRyan
  • Maxim Kim
    Maxim Kim
Maxim Kim
Ataccamer
Forum|alt.badge.img+3
  • Maxim KimAtaccamer
  • Ataccamer
  • January 8, 2023

Bear in mind that post-processing components are per each catalog-item in the MP...

K
Data Pioneer
Forum|alt.badge.img+1

Thanks @anna.spakova and @Maxim Kim My requirement is we have to send only valid records ( valid row only when ALL columns in a row is a success) to output DB. I am looking for a solution to filter the rows with only valid data.

anna.spakova
Ataccamer
Forum|alt.badge.img+3

@KarthikaSPillai This can be done using the post-processing component. The component is in fact a plan in ONE Desktop where you can use all the functionality it offers - including filtering etc. The incoming data in the post-processing component contain three additional columns (besides your data):

  • valid_rules (list of DQ rules the record successed)
  • invalid_rules (list of DQ rules the record failed)
  • invalid_rules_explanation (explanation messages for the failed DQ rules).

It is then enough to use filter like: invalid_rules is null to get only records with no DQ issues.

DannyRyan
Maxim Kim
2 people like this
  • DannyRyan
    DannyRyan
  • Maxim Kim
    Maxim Kim

Reply


Community Terms of Use & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings