Data Quality & Catalog ⚙️

Term Detection Rule: Based on metadataRules based on metadata use information about asset, e.g. description, Catalog Item or data source name.Is there an option to expand this function to detect our “add custom metadata”?  For example:  On the Catalog Item level: Data Classification, Criticality, and Data Asset Type  

Hello Community. I have a vendor who shared their data through Snowflake Data Sharing. We added it as a data source and scanned the source to add the objects into my Data Catalog. I even have a sccheduled job that scans this source everyday (end of day) for sample profiling.Today, we had an issue that 10 of the tables that were previously present in the shared data source, were dropped, two new tables created, and some of the tables, have new fields added, existing fields removed, some fields renamed.This source is about 300 tables, but some of the tables ahve about 700+ fields. So, I would like to know best way for me to get a report/alert/notification onany new tables added to the source any existing tables dropped any existing table modified with new fields added any existing table modified with existing fields dropped/removed. What is the best way to accomplish this?For the dropped tables that previously existed, my end of day profile job is showing error in the DPM (because the ta

Data Quality checks on catalog item can they automatically come into monitoring project when that item is added to MP? Here is my challenge:I have catalog items with 20 or more checks. If i add them to monitoring project - it does not automatically show as “Data Quality” for that catalog item itself. And when i add the checks to the Catalog item, they are not coming to monitoring project when i add that catalog item to the MP.Iam needing to add/configure all the rules separaely - both at Catalog item level (to get DQ percent score for that object), and also at the Monitoring project level. Is it possible to add the DQ rules - just to the Catalog item, and when I add that catalog item to the MP, it should automatically bring the existing rules on that object into the MP?​​​​

HiWe are interested to export metadata information into excel or CSV using “Export list of Items” option (see attached image).However, this is disabled in some screens.While there are ways to use desktop components to read metadata and export to CSV, this process is highly cumbersome and still doesn’t solve the purpose.Are there anyother alternatives?

Hello: I have a data source with 300+ tables. I have a daily profiling job (sample data).Sometimes, when a table gets changed, and a new field added/existing field dropped, when i open that table in Data Catalog, and in the “Data” tab, it shows that “N” number of metadata changes happened that are marked with a “yellow triangle” symbol (warning/alert).But, Iam able to see the list of fields that got changes, but iam not able to figure out if it is a new field added or existing field dropped or existing field changed??How can i get this data? 

How can I enable connection page to show glossary/business terms along with other properties. I tried editing the page template but not sure what details to add for fetching those details. 

Hello Dears,Who went through migration from 13.9 to 14.5  or even to 15.2? How did it go?We were planning to upgrade soon to 14.5, but were surprised realizing how much manual activities it will take from our side. Such as recreating all monitoring projects, lookup files, post processing… My question: will migration to newer version always required so much manual intervention or it is only when it comes to migration from 13.9 to 14.5? Thank you in advance.Kind regards,Maria 

Is it possible to customise the subject (screenshot below) and add things like DQ rule or Project name by any chance? Currently it just returns “Project run finished”.  I have some groups with several monitoring projects and would like to know the project name in the subject line itself.And having the ability to add overall Data QUality results of that MP in the subject line will be a definite plus.For Ex: “Project run finished for Security Master Start of Day at 5:05AM PST on 4/2/2024 with overall Data Quality of 84%”(the bold and underlined above would be variables available for that monitoring project)

We need  the change datetime of rules, catalog items, attributes and monitor projects. How can we extract these using one Desktop. In One Web for these assets the history can viewed, but I don't see where I can get these change dates from. Where is the version information stored and how can we read that info?

Can we create a lineage in Ataccamma which shows the lineage from source till the target?There are many stages involved before reaching the target. This lineage should help me in doing root cause analysis when a column has a different value in the target(Delivery). can someone help me on this?

Hi everyone! This week we continue to explore different features of Monitoring Projects ✨Each day this week, we will share a best practice post on Monitoring Projects. We are kicking it off with a popular feature - notifications 🔔 General NotificationsThere are two types of notifications available: General and Specific. General notifications cover various scenarios such as monitoring failures, critical structure errors, new results availability, data quality issues, and anomalies. Specific notifications offer even greater granularity, allowing you to set custom data quality thresholds and filter conditions for triggering notifications.Adding General NotificationsTo set up notifications for predefined scenarios:Navigate to the Notifications tab of your project.   Under General, select Add Notification to create a new notification or Edit an existing one. Provide a name for the notification. Choose the events that should trigger notifications (e.g., Monitoring failed, New version of re

Hi Community, Is it possible to customise the subject (screenshot below) and add things like DQ rule or Project name by any chance? Currently it just returns “Project run finished”.  

Dear community,In the overview screen of an entity, like the business terms screen, each column has some predefined width. Some of the term names are long and are cut off because the column is too small.Is there a way to define the width of a column yourself?Thanks and kind regards!Albert

I have an Ataccama workflow (EWF file) developed in ONE Desktop that executes after an ETL job. The workflow kicks off multiple data quality monitoring projects, extracts and aggregates their results, generates a Tableau hyper file from those results and publishes it to Tableau server. I’m currently executing that workflow on a dedicated server, but would prefer to deploy it to Ataccama server and execute it via an API call.Is there a way to deploy and execute workflows on Ataccama server?

We are on 14.5.1.We have a custom property on our terms, that currently is mandatory.We want to find out if we can change it to non-mandatory. When I click on edit on the property, it only allows me to edit the name.Is it possible to turn off the mandatory flag?

If I create a component with a step that maps an expression as a parameter, and call that component from a workflow, I get a “Model is not valid” exception. The attached provides a minimal example of this behavior. The component, param_test.comp, uses a random record generator with a single column, the expression of which is mapped to a parameter named “param.” The parameter value is set to “param_value.” The workflow, param_test.ewf, calls the component, passing the parameter, “param,” with value “param_value.”If the component is executed in isolation, execution completes successfully. If the component is executed from the workflow, the below error message is printed and execution halts. Is this a supported scenario? If not, how can I map a parameter value to a column within a component that is called from a workflow?25.03.2024 01:20:25 [INFO] Processor started25.03.2024 01:20:26 [INFO] Task 'Run DQC' : IN_QUEUE25.03.2024 01:20:26 [INFO] Task 'Run DQC' : RUNNING25.03.2024

Hey all!I am working through a use case that wants to check if data is being populated. The issue I am facing is that when there are no records being populated, the case statements I am using will not work (due to no records being in the table) and no rule seems to work as well due to the fact that when no records are passed through a monitoring project, it “passes” but shows that there are 0 records. Is there a default value that Ataccama will show on the back end that I can check for on a rule?I also want to add that they stated they do not want to use anomaly detection due to the fact that we cannot be 100% guaranteed that it will catch no records being populated.Has anyone else faced this issue previously? Looking for any assistance here.Thanks!

Hello Community: Iam on version 14.5.1. WE have created a custom field for Catalog items to flag (a boolean field) whether the data is certified for consumption. I would like to update from a batch file to the few hundred catalog items. Is there a template anyone has  - to update catalog items ? I have previously used a component for updating catalog attributes (like fields in a table), but here i need to update the catalog item itself (and not the attribute). appreciate anyone sharing a template of a component that i can adopt and use. 

Hello,I'd like to set up a rule for the 'transaction_date' column to verify if the latest date in the column equals to previous work day. How can I do this? Is there any function for that?  If there isn’t , we also have an additional calendar table where  previous work day indicated for each date. But how can I indicate another table inside a rule?Thanks in advance.

Short question here: I'm creating a debug step that writes to a database table. In that I would like to put the name (or path?) of the component that was running. Is there some kind of system variable (similar to $ewf_workflow_id in workflows)? 

Hello,I have a monitoring project, and inside, there is a table (operations). This table is populated by ETL daily, and every day the ETL job finishes at a different time. That's why I can't assign a specific schedule to this monitoring project. There is another table (job_status) that shows the condition of the ETL job. Inside that table, there is a column named 'Status of ETL JOB'. If the value in that column is 'Completed,' then it means the job has completed, and now the monitoring project can be run. How can I create such a schedule for that MP?I would be grateful if you can help me. Thanks in advance.

Hi everyone! We hope that you’ve found our Getting Started with Data Governance best practice series useful! This week we are continuing the Getting Started series with Data Observability 🔭 In this post, we will introduce you to the key components of Data Observability and guide you through the process of configuring it for your data source. Understanding Data Observability Data Observability expands on the existing data quality monitoring capabilities in Ataccama. It allows you to gain insights into the state of your data by monitoring data quality, detecting anomalies, and tracking schema changes. The key components of data observability are:Data quality monitoring: Regularly checks data against predefined data quality rules, providing results and trends based on business terms or domains. AI-powered anomaly detection: Utilizes AI to identify unpredictable data quality issues, such as sudden drops in data volumes or average values in specific columns. Data discovery: Monitors and tr