Skip to main content

Hello Ataccama Team,

I am using Ataccama One Web Application (v15.4.1) and need help setting up data quality rules to monitor data loads for three different tables.
Each table has a Row create datetime column to capture the load times.


Load schedules:
•    Table A: weekly once (On Tuesday)
•    Table B: 5th of the month
•    Table C: 1st and 15th of month

What is the recommended data quality syntax for rules that check date freshness against weekly/monthly/semimonthly schedule?
•    Table A: Verify the data was loaded during the current week (on Friday or Saturday)
•    Table B: Check by the 10th of the month, to confirm data was loaded on / after 5th of month.
•    Table C: Check by the 5th for the 1st month load and around 20th for the 15th load.

I initially tried using datePart, but it looks like this function is not supported in our environment. Is there any alternative recommended approach for dynamically calculating offsets without hardcoding dates?
I am using maximum(toDate(ROW_CREATE_DATETIME)) >= dateAdd (TODAY(), -6, 'DAY')] for weekly load (planning to run this on Sunday)

What is the best approach for sending notifications when these checks run?
I want to send email notifications when a load has failed (no need to send for successful runs).
Can the email notification be customized with additional details such as latest load data or row counts?

Project setup
Given the different schedules, is it best practice to create three monitoring projects each with its own schedule or is there a way to manage this in a single project.


Any examples or best practices from others who have implemented similar monitoring would be helpful.

Thank you!!
 

Hello ​@senaiswarya ,

thank you for your question. I would recommend checking out parameters https://docs.ataccama.com/one/15.4.0/data-quality/create-dq-rule.html#inputs→ that way you don’t have to hardcode the values in the rule, you can create it once and set the specific value directly in the monitoring project when applying the rule to the table: https://docs.ataccama.com/one/15.4.0/data-quality/create-dq-rule.html#inputs

As for the schedules, if we go with this approach using the rules and projects, easiest is probably to have the projects separate to be able to schedule it accordingly, in case you’d want to have it in one, I think you’d need to include some condition to check the date of the run against the expected “refresh” of the data (that value would be provided through parameter to each of the rule mapping). But not sure if this isn’t too complicated. 

But besides projects, have you considered the data freshness functionality? Not sure what is your technology, but there should be an option to customize it as well: https://docs.ataccama.com/one/15.4.0/data-observability/data-observability-dashboards.html#freshness,  https://docs.ataccama.com/one/15.4.0/data-observability/data-freshness-custom-configuration.html#custom-freshness Maybe that would suffice for your needs?

As for notifications in the projects, you can set up notifications only in case the data quality is below threshold: https://docs.ataccama.com/one/15.4.0/monitoring-projects/results-reports-notifications.html#notifications and yes, you can also customize the template to certain extent: https://docs.ataccama.com/one/15.4.0/notifications/notification-templates.html#email

Let me know if anything from this is useful or if you need more information.

Kind regards,

Anna

 

Reply


Community Terms of Use & ConditionsCookie settings