Data Quality & Catalog ⚙️

Hello all,I am running into an issue when trying to grab terms from attribute items. My current use case is as follows. I have a list of catalog item that are tagged with a term (that we refer to as a jurisdiction) and I am attempting to first find all attributes associated with these catalog items and then find all terms that are tagged to these attributes. The order is as follows:Find all catalog items with term → get all attributes associated with catalog items → extract all terms on attributesMy plan looks like this: The issue that I am running into is that these plans take a very long time to run sometimes upwards of 30min to an hour depending on the term. I would like to know if there is a more efficient way to do this (either through graphql or some other components etc) or if I am simply stuck waiting with the current speed of the plan. Best Regards,Andrew

Hi,Previously we added Data Quality rules on Attribute Terms via Settings. The outcomes was not something we expected as customers could see the organisational Data Quality score. Where we expected they could see the DQ score on Attribute terms which were only available to them. Hence we removed the DQ rules on Attribute Terms, but the DQ score is still visible in certain Catalog Items. My question: how to get rid of this?  

Hi,I added a DQ Rule on a boolean data type attribute inside a Monitoring Project (MP). After the attribute changed datatype I had to add a new DQ rule with string attribute, but the old DQ rule vanished. How do I get rid of the error?  

Hi ​@anna.spakova ​@Lisa Kovalskaia ​@Ataccama Community Admin I looked into the documentation for Ataccama ONE (the Ataccama data-management platform) regarding source/connectors and could not find a clear statement that it directly supports a connector to Google Analytics 4 (GA4).Could you please confirm the possibility?

Can we create interactive fields like “upvotes” or “likes” directly inside the Catalog UI for user interaction instead of using comments sections?

Hi ​@anna.spakova I am trying to write invalid data into DB using JSON Call. For that I want to first get the invalid values from the “Show Invalid Samples page”Not like any other array in JSON, the values array does not have key - value structure. Instead, it is like values: (array) 0: <no key> only values. When I run ONE Desktop Plan, I am only getting blank rows. Please help Thanks.  

Looking for ways to access the backend data storage for Ataccama ONE. We’re currently developing an internal data quality dashboard that needs to pull data from ONE to build visualizations in Power BI.Thanks!

I have created monitoring projects in Ataccama ONE . I don’t want those reports to be stored into MinIO. Instead, I would like to export those reports in csv format through Ataccama Desktop platform. How can I do that?

I have a data quality monitoring project to which I’ve deployed a post-processing component. Results are written to a CSV file, which I presume is stored within Ataccama’s object storage. When I navigate to the data quality monitoring project and select the Export tab, I see the below screenshot, which appears to include a file path along with a download button. I’m trying to determine how to automate download of the below-depicted CSV file. Is there a way to determine the fully-qualified path to the file and download from Ataccama server? 

One of our teams need to use ataccama as a self service tool and catalog the excel file. How to do that?​@anna.spakova 

Hello ​@Lisa Kovalskaia ​@anna.spakova I have a requirement to do DQ of few tables that have names changing as per schedules of a release. e.g., Table Name could be Table1_release_1 etc. The database and schema are constant. The scope/structure of DQ attributes is mostly same.For now, I think it will need manual intervention to onboard new objects → Add DQ on attributes and then add monitoring projects manually. Is there any way we can automate this process?I am using v15.4

Hi Community,For a catalog item you can add one or more linked instances (other catalog items). With a Desktop plan I would like to make visible which linked catalog items exist, but I cannot find an object in the metadata model that reflects the linked instances.Does anyone have an idea which metadata entity I should use?Kind regards,Albert

Hi I have tried looking into an article on how can we do data reconciliation on Ataccama, but I only found that can be achieve in Ataccama ONE Web, not on ONE Desktop. How can we achieve the same in ONE Desktop?My use case is, I want to know the number of rows between two tables from two different databases - Oracle and Snowflake. I tried doing that in ONE Web but it took a lot of time to load.

Hi everyone,I’ m having an issue with missing DQ results  in one of my dashboards.The dashboard had been running successfully for a long time, but it failed on July 27. After investigating, I fixed the issue on October 16, and it has been running normally since then.  However, the DQ results  between July 28 and October are missing, most likely because the dashboard didn’t run during that period or it failed for some reason. Is there a way in Ataccama to rerun DQ jobs or recalculate results for specific past dates?Or is there another recommended approach to fill this gap in the DQ dashboard? Thanks is advance for your help!Ece

Hello! One of the engineers on my team noticed that when trying to configure notifications for when Anomalies are detected on a monitoring project that it only works if the Data Observability for the source is also configured for Anomaly Detection and to send emails. Is this behavior anyone else has experienced in V 16.2? Is this intentional? It seems odd that these would be linked, I would think that users would want to be able to configure Anomaly Detection independently either at the source or the project level, and that the audiences for those anomalies might be different.We’re currently working through a bug where this is working this way in our dev instance and not working at all in our production environment. To clarify, this is not about the general Monitoring Project Run Completed email, which includes info on anomalies, data quality, etc., this is just about the “Anomalies Detected” notification on monitoring projects. This has occurred in multiple instances in Dev and has no

Hello folks,I am creating a new connection on web application for OneDrive/SharePoint using Microsoft Entra ID; while doing so there's this field (Drive Id) which is optional if you have multiple drives and you want to limit the connection to a specific drive, we enter the ID of the drive you want to connect to.Without Drive Id I am able to access the root folders and their content but I want to target a specific child folder.We tried the actual drive ID, folder name, web URL for that particular folder but faced below mentioned errorError - com.ataccama.extension.onedrive.connector.api.OneDriveConnectorException: Unable to connect to OneDrive  This is the error message I get if we provide Drive Id 

Adding more details to a list view and providing additional filtering options can help business users in their daily tasks as you scale Data Governance across multiple teams, geographies, or business functions.In this article we’ll customize the Business Glossary to show additional term attributes as columns in the list view and to enable filtering by those attributes. This approach generally works across the Ataccama ONE Web app, so you can use this article as a guide to customize other pages, for example Rules, Catalog Items, or Monitoring Projects.You don’t have to be a developer to follow this guide. If you come across any concepts you’re not familiar with, have a look at our documentation. Adding columns First, navigate to the Business Glossary’s Terms list. In the top right, find the 3-dot menu which features the option to Edit page. In the side bar that opens up, go to the Layout tab. You’ll see the JSON that describes how the objects are shown on this page when the web app load

Hello, I am trying to understand better where timeliness dimension can be applied with all the possible case scenarios. The documentation explains about the configuration of the dimension but with no real time examples. Could you please help with some use cases from the real world?For something like below , what do we need to take into consideration when implementing the rule? Attribute must be updated Daily ; effective date= Today-1. Load dates are Sun-Sat excluding Holidays.

Hi everyone,We have created several monitoring projects in Ataccama, each dedicated to a specific catalog item. Our approach is to create special tables per clientId and measure those within the monitoring project-effectively one monitoring project per catalog item.Could you please share your thoughts on this approach? Is this a best practice or are there more efficient ways to handle multiple client-specific data quality checks?Additionally, we want to create individual postprocessing plans for every rule in a monitoring project. For example, if a catalog item has 20 rules, we want 20 separate postprocessing outputs (Excel files) listing the bad records per rule. This is to share these sheets with data stewards who will then fix the data and upload it back.However, we are encountering a limitation: Ataccama currently does not allow more than 10 postprocessing plans per monitoring project.What are your recommendations or best practices to handle this? Should we split monitoring project

I need to create a complex duplicate check rule.  I am familiar with the aggregation rule; however, in this case it won’t work for me.  Here is the logic to determine duplicates:If the name, unit of measure, location, and type are all the SAME but the code (not a PK) is DIFFERENT then it is considered a duplicate.These would be considered duplicates:Code Name UOM Location Type 123 Trek LBS Houston Bike 234 Trek LBS Houston Bike            This is not considered a duplicate because we can extend products to different locations instead of creating a new row: Code Name UOM Location Type 678 Razor LBS Houston Scooter 678 Razor LBS San Antonio Scooter            When I write the rule, I use the aggregation rule to select name, uom, location type is not unique; however, that can give me false positives for the scenario above (Code = 678).  I don’t see a way to say these fields are the same while this field is different. 

Hi,I have created a simple monitoring project with 12 catalog items for which I do only one dq check per catalog item + anomaly detection for this one attribute. However it seems that for this the system runs a full profile of every catalog item. Therefor this project is very heavy for the system to process. Can we adjust some settings so that the profiling is only run for this one attribute per table? Seems like a waste of resources this way.kind regards, Jur Dördregter

I have a requirement in which I am supposed to check completeness and data freshness on a catalog item at row level and at the same time I have business rules that are table level validations like - checking maximum sales value for previous batch/load is in allowed limits (95 to 105%)e.g., if aggregated Sept value is e.g, sum(sale) = 100 when month = max(month_field) then next load e.g., October value can have value sum(sale) = 105 or 95 anything other than this is fail of my dq check.I am using Ataccama web v15.4 Issue here is I cannot store the max and second max values in same catalog item and apply the rule, any suggestions? I am thinking of creating a sql catalog item to get the subset of data and then apply business rules that need aggregated values.Please suggest the right approach/alternative here?

Hii folks,We have a requirement to do DQ evaluation on a table that has few billions of records (> 10 billions). We have to perform various checks on this table that includes aggregations for uniqueness check, completeness, data format etc. Making a subset of data also doesn’t help in reducing volume to desirable/executable range as it still has total count in billions.What would be the ideal way to to DQ on such tables? Your inputs are highly appreciated here. Thanks! ​@anna.spakova ​@Lisa Kovalskaia  any suggestions here?

I want to create a sql catalog item by joining one table from snowflake schema and other from one data file uploaded as catalog item. I don’t see the table available for selection from SCI creation screen. Is it not possible to do?

Hi,  is it possible to remove a specific run of the data observability? We had to rerun data observability today, but now we see a lot of anomalies because part of the data had remained the same between two runs. I want to trow away both runs and then rerun again to keep things clean. kind regards, Jur Dördregter