Data Quality & Catalog ⚙️

METADATA MODEL The metadata model is used to configure the application entities and their properties. The Metadata Model describes the entities and their properties in the application. Web Application) use the Metadata Model to create the frontend user interface. The metadata model is also responsible for the configuration of the left navigation menu of the application. ONE METADATA PROPERTY REFERENCES Entities consist of properties of Simple types, such as strings and integers, and Object types, which link the entity to other entities. ADDING NEW ENTITY TO MMM MODEL If you are creating an entity using the user interface, you must then add it as a property of the root entity metadata and then publish For example, if we need to add a new entity under Business Glossary named (TestEntity) to the MMM model Click on the Global settings on the left panel Choose Metadata Model under Maintenance Center We must create a new entity under Term so click on create entity Fill in the name of the Ent

During bundle deployment I see this error. I have tried different combination of Update to old bundle. Manually update the component again on the existing old bundle and upload Copying extended unification Updating OmnigenServer folders (bundle,cmd,dbms) with the backup. restarting services as needed. Any thoughts or suggestions? Thank you.

The matching component is slow at the extended unifications step for a incremental load. When a record like ‘WALGREEN%’ is loaded and there are 20K records with similar name on the instance, the records are all considered through the process of matching. Hence the process is slow when processing the increment and even if the incremental load has 11 records to process 30K records have the transaction Id and omni_modified date updated with the work order related on the instance table. When I check history, nothing changes on these records except the omni_modified_date and transaction_id to that of the work order. Tweaking the union keys to create combination of name and key factors or adding separate groups for the high count records is not helping. Any thoughts or suggestions?

Hello everyone, I created a profile of my catalog item using ONE desktop and I would like the output(.html) to be accessible to business analyst with read access. The html is only accessible on my local computer. Where’s the best / secure location to store output reports of profiling? Please chime in with any thoughts. Thank you! 😀

Hello everyone! I have an interesting task related to the desktop, we have a table combined with certain attributes and with full names, full names are divided into columns (firstname, lastname, midname) and they need to be compared with other columns (firstname1, lastname1, midname1) it is necessary to output non-matching full names. I can't compare one table between columns, I need your help. Do you know how to implement this? I will be very grateful for your help

Is it possible to Set up the detection rule, which would allow to detection of new table(s) within the Snowflake schema and trigger the required potential alerts

Imagine the scenario where you have Ataccama ONE Data Quality Monitoring Projects running regularly. Any Data Quality (DQ) issues are already exported to our DQ Issue Tracker (DQIT) module. You can also see trends of DQ over time within the Ataccama ONE Web application. You start noticing that the score for some DQ rules has had a sharp drop in quality, and you want someone to investigate this. One of our customers decided that they want to integrate Ataccama with Jira at this point. Have Ataccama raise a ticket into Jira when this happens automatically. In this article, we will explore: Preparing to integrate with Jira The Json Call step Preparations Business Understand your end users. What do they expect to see in their Jira ticket? Know which fields in the Jira ticket are mandatory Have an idea of what extra information you should supply in the detailed description of the ticket. Think about formatting and presentation. Good to understand Jira markdown notation (link) Jira Know whic

Hi Team, Our Aim is to Avoid duplication of the same data coming from different requirements (project trains) - in case of identified duplication DQ Brach alert should be reported. 1.How to identify if Somone ingested the same table in a different Snowflake schema ? Is it possible to detect duplicate table through ONE web application or ONE Desktop by creating a rule? Thanks in Advance!

I have a sql select where in I wanted to pass multiple data source names. how to make it dynamic or parameterized so that It can read all the names one by one.

Hi, For our monitoring projects post processing we use fairly standardized (sub-) components. Now for every MP / CI combination these sub-components are all the same. If an MP Postprocessing is published these components get published also, right? Is there a way to have a single version of a component stored? This would make maintenance and life-cycle-management on these components a lot easier. In stead of having to go in the PostProcessing main component, we'd just have to change one. A plausible solution would be to split things and do things asynchronously, maybe. But that is a more of a workaround. Any ideas?

Hi, I’m trying to modify our view of our business terms - currently we have a lot of information showing in the default view that our end users may not care about. What I really want to do is modify the view, so that it only shows Name, Abbreviation, Description and Synonym. I have figured out how to add the columns, but I can’t figure out how to remove unwanted columns. For reference here is the default screen layout in Ataccama One via the Debugging Tools. { "_type": "entity.page.listing", "children": { "_type": "relationships.hierarchyListing", "enableSelection": true, "columns": { "type": { "dataPath": [], "name": "Type", "after": "name", "renderer": { "_type": "glossary.termType.tableCell" } } } } }

Use Case: As part of the Data Governance Program KPIs reporting, we need to report on unique user usage count and details of who within the Ataccama Web on a monthly, month-to-date, and year-to-date. Can this be easily captured using OpenSearch Dashboards historically or elsewhere?

I’ve been working on a use case recently which I think may be of interest to the community here. Some reference data is stored in a single table DATA_KEY_ID column tells us what reference data set it is DATA_VAL column stores the actual reference data in the format of a json array Goal is to extract the values from the json array, and use it as a lookup item in Ataccama ONE, which in turn is used in Data Quality Validation rules To start with, you need to have some preview access to the data and understand the schema of the json. For the example in this post, I have very simple “id” and “name” properties for my json. You will need to analyse that data before starting this, so you know what to put into the Json parser, or which values you want to get out from the json object. Solution 1: Virtual Catalog Item (VCI) Bring the reference data table into Ataccama ONE Build a VCI, with a Catalog Item Reader of the reference data table Use a filter step to filter the exact set I need (it will

I’m having this import error note, how to fix this? thank you.

I tried to run monitoring project at morning, but it is forever running. Please let me a know a way to stop it. Thanks! @anna.spakova @Catherine

Introduction This post was created to answer the following questions: When moving data through data pipelines, how can the Ataccama ONE DQ platform support Data Integration platforms other than its own. How do other Integration Platforms access the DQ rules? How do other Integration Platforms flag records for Data Stewards to review? DQ platform in context of integration processes There are three key patterns that are typically used for integration of the ETL process with Data Quality: Option 1 The integration platform calls Ataccama DQ rules exposed as real-time API, sending in batches or micro-batches of data for evaluation. Option 2 The Integration platform loads data into a staging/landing layer and triggers (via API call) a batch process in Ataccama, Ataccama evaluates/improves the data and populates the cleansed layer in the data lake, including DQ scores and flags. Option 3 Ataccama can be connected to a data stream, such as Kafka, and on the fly evaluating data in the stream -

Hello, community I’m currently working on a project which requires a workflow on the runtime server to be triggered after the monitoring project on Ataccama ONE is finished. Can anyone recommend a solution to achieve this? Thank you in advance

I see column level quality score in monitoring project. Do we have row level quality score?

Hello, I am new to DQ and Ataccama. Our code is in development environment and soon to be deployed in Test/UAT for UAT testing and later to Prod once we get UAT signoff. My question is what is the way to export DQ monitoring project and its related components (like rules, etc) from Dev environment to UAT/Test environment? Is there any export capability? I have also heard that we may need to recreate it again in UAT environment. Really appreciate any direction in this. Thanks in advance.

I want to export DQ Evaluation Rules from Ataccama ONE and import them onto another Ataccama environment. To my knowledge, this will require an export plan. However, I’m unsure how to configure the JSON file for the plan. Could anyone guide me on this, please? Thank you in advance

Hi, We'd like to compare two string-fields with several words. Is there a fuzzy matching or approximate string matching step for a plan? In this case it is not reference of lookup data. They are name fields. One is in the DB the other in a file. Kind regards, Marnix

I’m running a monitoring project and it is failing. Here is what I am receiving as a message: Caused by: java.io.IOException: strange number of lookup header sections: 0 What does this mean?

You can optimize your plans by simple things. In this case I would like to show how the number of columns/fields influences the performance of the processing. Test conditions local environment, stable conditions 14M records generated by Random Record Generator 78 fields containing fixed strings or dates No parallelism level set Test Cases One stream – simple reading - BASELINE Overall progress: 117k/s Duration: 3:08min Split into 4 streams, same number of fields – you really need ALL fields in ALL streams Overall progress: 91k/s Duration: 3:42min Split into 4 streams, reduced number of field to 1 – maybe you just need 1 field in each stream Overall progress: 109k/s Duration: 3:19min Split into 4 streams, 3 of them with reduced number of fields to 1 – you might not need all fields in other streams Overall progress: 101k/s Duration: 3:33min In this example it’s just about simple transformation flow however in case you use JOINs, RepresentativeCreators, Aggregators etc. and you reduce the

Use Case; to configure a scheduled profiling job daily for a specific schema of a data source within Ataccama Web application and not via Desktop?

I need to connect to an AS400 database. I have an odbc connection working for that but I can use ODBC with DQA. No OBDC driver is available. it is not like what is displayed on the FAQ support page when they explain how to connect to MS access using ODBC. it seems the driver has disapeared :-(