Solved

Data Quality Check for No Records Populating

  • 15 March 2024
  • 2 replies
  • 20 views
T
Badge +1

Hey all!

I am working through a use case that wants to check if data is being populated. The issue I am facing is that when there are no records being populated, the case statements I am using will not work (due to no records being in the table) and no rule seems to work as well due to the fact that when no records are passed through a monitoring project, it “passes” but shows that there are 0 records. Is there a default value that Ataccama will show on the back end that I can check for on a rule?

I also want to add that they stated they do not want to use anomaly detection due to the fact that we cannot be 100% guaranteed that it will catch no records being populated.

Has anyone else faced this issue previously? Looking for any assistance here.

Thanks!

icon

Best answer by maykwok_hamilton 16 March 2024, 02:55

View original

2 replies

maykwok_hamilton
Userlevel 2
Badge +2

Is your data source an SQL database?

If so you can create an SQL catalog item which could be something like:

select count(1) as row_count,
'my_table' as table_name
from my_table

And the rule would be “invalid if value = 0”

Then you put this SQL CI into your project, mapping the rule to attribute row_count.

 

There are also aggregation rules in Ataccama ONE DQ but I haven’t tested it to see if it would run into the same problem that you stated - no records could mean no evaluation.

T
Badge +1

@maykwok_hamilton  This worked!! I am seeing the record count as 0 when there are no records being populated, and having a rule to check for 0. Thank you so much!! I was previously trying to include as a single catalog, but makes much more sense to break this out as two separate catalogs. 

Reply


Community Terms of Use & ConditionsCookie settings