Skip to main content
Solved

Data Quality Check for No Records Populating

T
Data Pioneer L2
Forum|alt.badge.img+2

Hey all!

I am working through a use case that wants to check if data is being populated. The issue I am facing is that when there are no records being populated, the case statements I am using will not work (due to no records being in the table) and no rule seems to work as well due to the fact that when no records are passed through a monitoring project, it “passes” but shows that there are 0 records. Is there a default value that Ataccama will show on the back end that I can check for on a rule?

I also want to add that they stated they do not want to use anomaly detection due to the fact that we cannot be 100% guaranteed that it will catch no records being populated.

Has anyone else faced this issue previously? Looking for any assistance here.

Thanks!

Best answer by may_kwok

Is your data source an SQL database?

If so you can create an SQL catalog item which could be something like:

select count(1) as row_count,
'my_table' as table_name
from my_table

And the rule would be “invalid if value = 0”

Then you put this SQL CI into your project, mapping the rule to attribute row_count.

 

There are also aggregation rules in Ataccama ONE DQ but I haven’t tested it to see if it would run into the same problem that you stated - no records could mean no evaluation.

View original

may_kwok
Star Blazer L3
Forum|alt.badge.img+4
  • may_kwokStar Blazer L3
  • Star Blazer L3
  • March 16, 2024

Is your data source an SQL database?

If so you can create an SQL catalog item which could be something like:

select count(1) as row_count,
'my_table' as table_name
from my_table

And the rule would be “invalid if value = 0”

Then you put this SQL CI into your project, mapping the rule to attribute row_count.

 

There are also aggregation rules in Ataccama ONE DQ but I haven’t tested it to see if it would run into the same problem that you stated - no records could mean no evaluation.

Cansu
1 person likes this
  • Cansu
    Cansu
T
Data Pioneer L2
Forum|alt.badge.img+2
  • ThomasProData Pioneer L2
  • Data Pioneer L2
  • March 18, 2024

@maykwok_hamilton  This worked!! I am seeing the record count as 0 when there are no records being populated, and having a rule to check for 0. Thank you so much!! I was previously trying to include as a single catalog, but makes much more sense to break this out as two separate catalogs. 

Reply


Community Terms of Use & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings