Data Quality & Catalog ⚙️

hi Team,I would like to understand a couple of things related to implementation of slices directly on catalog item v/s filtering out records in sql or virtual catalog items and then using these items in monitoring projects (MPs) I am adding a data slice on a data catalog item X having e.g., 300 millions of records in total, sliced subset is having records around 30-35 millions -Question: when I run the MP, will it first pull all 300 records and then filter out those 30 million records based on slice and then do the DQ evaluation? Also, In case I use SQL catalog item (SCI), I am already cutting down undesired records, so I’m directly applying DQ evaluation on 30 million records, will it be faster in this case than above scenario?In any case, Slice / SCI/ VCI - how are we calculating the overall quality of the data catalog item here? Are there any guidelines on calculation of DQ on the whole dataset using output/score from DQ of subset of data.Really looking forward for your inputs here!

Hey Community!I would like to create an overview with One Desktop that provides me insight in which assets have been shared with which groups/persons with which type of access. I have been checking the metadata model, but cannot exactly locate the relevant entities that I could use for this purpose. Does anyone have an idea? Maybe only possible with GraphQL?Kind regards,Albert

I need some assistance creating a AQL query for finding a table within a database and schema. How would this be written? 

I have few monitoring projects in web but I would want to add them as components in workflows and run from desktop on schedule.Is it possible and if so what are the steps to do?  Are there any pre-requisites for this?Thanks in advance!

Hi,Via ONE Desktop I created an invalid records component file which can be extracted in the Monitoring Projects → Export tab. It is in regards to a Float attribute. However, when I download the .xlsx file and open to check the invalid records it shows blanks. While if I check the database I see values.How do I show Float attributes within the .xlsx file?

Our team has just started using reconciliation project in Ataccama ONE UI, just wanted to confirm if we can apply a DQ rule on reconciliation projects, you can take a requirement e.g. if source data is having Wed as a record in an attribute and target is having Wednesday as a record in an attribute with same name, it shouldn't be highlighted as a different record.

We have environment with 14.5 and now we have established totally new environment with new Ataccama 15.4. We want to migrate all the object from one environment to another without manually re-creation of objects.Is possible to use metadata database & Audit Database of 14.5 in 15.4 Environment with some upgrade process so that all the objects can be accessed in 15.4 without any issues?What are the steps to be taken for this?

In Ataccama Monitoring projects, the email notifications currently provide summary details (e.g., number of failed rules, scores, etc.). Is it possible to customize the email notification content so that it also includes a snippet or sample set of failed records directly in the email body (instead of just attaching or linking to error tables)?If yes, what would be the recommended approach or configuration to achieve this in V15.4.1?

I have two BDE DPEs pointing to different Hadoop clusters. They have different cluster ids set in the configuration. In the DPM they both show up separately with their own configuration. In ONE Web I have a connection that uses DPE 1, which is working fine when only DPE 1 is connected. However, when both DPEs are connected I get the following error when I try to browse this connection:It seems to me that ONE Web can't seem to differentiate between the two DPEs. I have tried all kinds of different constraints in the application.properties files of both DPEs, but nothing has worked so far.Does anyone know if this is possible and if so, how I can get this done? Please let me know.

In One Web, with invalid sample records uploaded to ONE Data, we would like to filter data that failed on a particular DQ (Data Quality) rule and download it. However, at present, there is no way to download a filtered view.The system currently only allows us to download the entire table. This means that when we download the whole table, all the DQ rule filters are lost, and there is no way to figure out which rows failed on which DQ rule. This is quite problematic as we want to be able to download records that errored on a particular rule so we can send them to the appropriate team for corrections specific to that rule.Also, we often perform DQ evaluations on spreadsheets and setting up post processing or remediation plan on an excel sheet is not the most efficient approach. This is because each time there is a new version of the excel data, we would need to rebuild the whole plan again. Considering that the download functionality is already available, I believe adding an option to do

Hi ​@anna.spakova Is it possible to download the list of rules (maybe using metadata reader if not from UI)?Also, anything/any links related to functionality of Export List Of Items option could help here.Thanks! 

Hi I see we can configure PowerBI as data source in connections but want to know if it’s possible to use the BI report to be treated as catalog item to apply DQ checks and profile these.

Hello community,I’m creating a Power BI source on Ataccama One and have a few queries:Is the built-in Power BI connection allow One to retrieve a Power BI datasets? Or this is strictly for only bringing the Power BI reports to One? The imported catalog items from this connection shows “0” number of attributes and “-” number of records (screenshots below). This is also presents in the the screenshot from the Ataccama support page (link). Is this behave as intended or am I missing anything here?Thank you in advance.Ben

I have created new Email template and wants to use that template while send Project Result notification. Please guide how to link new email template instead of default template? Do I need to change any property?

Hello!Does anyone have any ideas for how to make a welcoming and useful Home page for Data Users coming to Ataccama just for data discovery? All of the widgets seem to be focused on Data Stewards, but the majority of users arriving on to Ataccama for us are looking to find data to work with, so widgets like Tasks, Processing Centre, Notifications, and even the Getting Started ones are just confusing to them. I’m looking into making a Report that just acts as a welcome page for those new users, but haven’t been able to make anything that nicely introduces the platform without putting too much in front of them. Let me know if anyone’s done something like this before!

Hi Team, I have a requirement where I need to write the post processing results (post DQ, failed records) into snowflake database in a dedicated schema.If I set up a user X to perform this write operation in snowflake schema, does it also need to have read access to the data objects that I am monitoring for data quality.Let’s say data objects are on schema s1 of database ABC (snowflake) and post processing results are to be saved on schema s2 of database PQR (snowflake)Thanks for your response!

Hi all,I cannot find the filter pane (Filter) in version 15.4. Is there any chance it doesn't appear due to my access permissions?Also, is it possible to search for a custom value within the schema? If it exists, I need to identify the corresponding table, field, and row number.Thank you!

 Hello,I am new to Ataccama One and had a question around using the DQ rules in other ETL tools and integrations. It’s my understanding that a DQ rule returns whether the inputs are valid\invalid and it can be called as a web service using DQ firewalls. Is there a way to extend this logic to apply a fix to the input and returning this fix for invalid inputs ? Can this be exposed and reflected in the DQ firewall ? Do we need to build a component rule for this or can this be achieved using rule variables \ parameters ? It is not clear to me when parameters and variables are typically used in rule configuration. Regards,Ihab

Could anyone help me understand how to best create a collection or visualization for the dashboards? Currently when I go to create a new collection I cannot see any tables/attributes (that exist in the Knowledge Catalog) to select as part of the collection. Is this permission based or do I need to enable a setting for the data somewhere? Any help would be greatly appreciated, thanks!  I’m currently using Ataccama Web v15.4.0

HelloI am looking to create a One Desktop plan to extract records which are out of sync. I need to compare tables coming from different databases and sources. Or better suggestion to achieve the same.Need help with this. 

I have created a post processing plan in One Desktop, and when running the Monitoring Project in One Web, I’m unable to see any sheets contains results.  

Hi All, We’ve had a request from our data consumers to have the Last Update from the Data Freshness check appear in the catalog search. Users want to know if an item is stale or is receiving updates, and i think this would be the best way to do so.I can find the property of the the last update under modifyTime column of the freshnessChecks attribute in the catalogItem entity, however I can’t get that to show in the catalog Search. It doesn’t error out which makes me think the property is correct, but nothing appears in the cell for each catalogItem even when there is a freshness check.  "modifyDate": { "name": "modifyDate", "dataPath": [ "modifyDate" ] }Has anyone had any success doing this, or anything similar?

Hello Ataccama Team,I am using Ataccama One Web Application (v15.4.1) and need help setting up data quality rules to monitor data loads for three different tables.Each table has a Row create datetime column to capture the load times.Load schedules:•    Table A: weekly once (On Tuesday)•    Table B: 5th of the month•    Table C: 1st and 15th of monthWhat is the recommended data quality syntax for rules that check date freshness against weekly/monthly/semimonthly schedule?•    Table A: Verify the data was loaded during the current week (on Friday or Saturday)•    Table B: Check by the 10th of the month, to confirm data was loaded on / after 5th of month.•    Table C: Check by the 5th for the 1st month load and around 20th for the 15th load.I initially tried using datePart, but it looks like this function is not supported in our environment. Is there any alternative recommended approach for dynamically calculating offsets without hardcoding dates?I am using maximum(toDate(ROW_CREATE_DATET

Is Mongo DB data source supported to connect and profile?

Hi Community people!I have a question regarding data observability. I am especially interested in noticing changes in record volume and applying this observability functionality in the process of a data delivery. This delivery is orchestrated with a tool, Jams in our case, starting with data retrieval and ending with the actual data delivery. Somewhere in this data flow orchestration I want to check for anomalous changes in record volume. The orchestration tool could then, for instance just before file creation, trigger an observability job to check the underlying database tables for record volume.But when configuring data observability I see a scheduler, which does not match with the idea of an orchestrated flow.So question 1 is if data observablity can be triggered.Maybe in a scenario where the orchestration tool starts an Ataccama job via a command, like a workflow that starts the observability and reads the observability results and returns an OK or Not OK to the orchestration tool