Last week, we had the pleasure of hosting Jessie Smith, VP of Data Quality, and Edris Ibrahim, Senior Sales Engineer, who shared the newest update to the Ataccama product with the community: Data Quality Gates.
But for those who missed the webinar, fear not! We haven’t forgotten about you. In this blog, we’ll go over some of the highlights and open the floor for your questions.
But first, let’s cover the basics: What is Data Quality Gates?
Data Quality Gates is a lightweight Python library and can be integrated into your data pipelines to perform live data quality checks as data flows through. This allows for a proactive, not reactive, response to potential issues, addressing problems during the data pipeline process, rather than after everything is completed.
With Data Quality Gates, users can define clear rules and automations that notify them when data fails to meet quality standards and enables teams to catch and fix bad data before it reaches the business and affects decisions or analytics.
Data management and quality are big deals, but how does this work within your organization?
The great thing about Data Quality Gates is that, as a Python-based SDK, it can be embedded directly into your Snowflake or DevDBT instance, and will be supported for Databricks and Python API soon. This enables the implementation of data quality checks directly within your systems.
From there, you can build out new rules and manage data quality directly in the pipeline. We’ve also integrated new AI features to help generate various inputs, allowing users to easily test and validate new rules. Governance is applied to all rules and packages to ensure that the right people have the appropriate permissions to create, edit, and publish rules in your environment.
But what happens when data or rules fail? Can you audit or trace the issue?
As Jessie confirmed during the webinar, DQ Gates generates an audit log of everything that passes and fails. This log is available wherever you’re running DQ Gates such as inside Snowflake.
There’s even more to explore about Data Quality Gates, so we’ve posted a recording of the webinar for anyone interested. Be sure to check it out and if you have any questions, let us know!