Skip to main content

Data Quality & Governance ⚙️

Discuss, learn, ask, suggest, share all things DQG 🔍

  • 553 Topics
  • 1,337 Replies
553 Topics
C
Data Voyager
ckaushikData Voyager
asked in Data Quality & Governance ⚙️
Temp Data Files Purging in Ataccama

DPM and DPE - Ataccama ONE Gen2 Platform LatestOnce DPE connects to a data source, it creates metadata and optional sampling of records that do not satisfy DQ rules.Could you please answer the following questions or direct me to a link that can answer the below questions?1. Does DPE create other temporary files as a result of processing as well?2. What happens to the metadata, sample records that do not satisfy DQ rules and any temporary files generated during processing after the connection to the data source get closed?3. How does purging of data works in Ataccama?

1131
anna.spakova
Ataccamer
1 year ago
M
Data Pioneer L2
MayKwokData Pioneer L2
published in Data Quality & Governance ⚙️
Creating JSON String From DataBest Practice

In Ataccama ONE Desktop, we have steps like JSON Reader, or JSON Parser, to read JSON data and put them into a data flow.We also have a JSON Call step that allows you to construct a JSON template to send JSON Calls outwards.But what if you wanted to transform the data into a JSON string but not necessarily send it anywhere? If that’s what you want to do, read on…Our dataLet’s take an example of this data:I have some fruits:id name 1 Apple 2 Orange 3 Pear 4 Grape  I have also some fruit variety names:id variety fruit_id 1 Gala 1 2 Golden Delicious 1 3 Jazz 1 4 Pink Lady 1 5 Navel 2 6 Blood 2 7 Seville 2 8 Conference 3 9 Comice 3  Join them together as our base data:(See json_1.comp in zip)Output looks like this: 💡 I want to now turn this into a JSON string. Variety should be an array within each fruit.To achieve this, we first need to aggregate the variety back into fruit, and then a final aggregation to get the whole set

3620
M
Data Pioneer L2
1 year ago
A
Data Pioneer
aiwaliaData Pioneer
asked in Data Quality & Governance ⚙️
Monitoring Project Export Rule Names

What does the id mention in invalid_rules column in Monitoring Project Export refer to?the alphanumeric term within() after rule name as in the image belowIs this the ruleId of the rule? If not, what does this refer to?Is there a way I can configure Metadata Export step to give rule id?

23310
anna.spakova
Ataccamer
1 year ago
A
Data Pioneer
aiwaliaData Pioneer
asked in Data Quality & Governance ⚙️
Divide results from Invalid Rule column in Monitoring Project Report Export into multiple columns

The idea is to divide the invalid_rules column as obtained in export for Monitoring Project into two columns- one containing the rule name and the other containing the explanation; and implement this for all the rules mentioned in a cell.For example, The below cell would be divided into invalid_rule_name column containing two records-validation [USA] State and calidation[Noth America] Phone Numbers, and invalid_rule_explanation column containing explanation for the respective rules.The challenge I am facing here is that the rule ID mentioned in Monitoring Project report export is not the same as the rule ID mentioned in the list of rules I export; which I am trying to use to identify the rule in invalid_rules column from Monitoring Project report.Please help me find a solution for this

1482
A
Data Pioneer
1 year ago
M
Data Voyager
manzuraData Voyager
posted in Data Quality & Governance ⚙️
Manage Nodes Access, Access Level in v.14.2 (VS v.13.9)

​Hello everyone! Foreword and problem statementThere are several subcategories of terms. I need to set up access for manager and steward of each subcategory so that:The steward of certain subcategory is able to create new terms, edit old ones in his subcategory only. However, he can not publish any changes The manager, in addition to above, can publish changes, but also in his own subcategory only.I have the list of questions concerning access management in v.14.2:Is it possible to grant ownership to node? And if it’s yes, how can I implement it? Do you know any way to solve my manager-steward task? (there are my understanding of the solution below) In States of approval workflowwhat are the “User roles” : governance or identity provider ones? (looks like the second ones, but doesn’t work) what is the “OWNER” for this version of workflow?_______________________There is a long prelude to where the questions came from. Version: 13.9There is the ability to create a special capability to b

932
M
Data Voyager
1 year ago
S
Ataccamer
SathyaAtaccamer
published in Data Quality & Governance ⚙️
Adding New Property to Catalog Item in MMM ModelBest Practice

METADATA MODEL The metadata model is used to configure the application entities and their properties. The Metadata Model describes the entities and their properties in the application.  Web Application) use the Metadata Model to create the frontend user interface. The metadata model is also responsible for the configuration of the left navigation menu of the application. ONE METADATA PROPERTY REFERENCESEntities consist of properties of Simple types, such as strings and integers, and Object types, which link the entity to other entities. ADDING NEW PROPERTY TO THE MMM MODELGo to left panel of the Webapp and choose Global settings   Choose metadata model under Maintenance CenterChoose the Custom filed where you need to add the new property to CatalogItem, Glossary and sources. In our case we adding to CatalogItem   Click on the Custom field CatalogItem. We can see General Information and Property. Click add property and type in the property to be added.   Fill in the name and choose the

1150
S
Ataccamer
1 year ago
E
Data Voyager
Evgeny ChernoskutovData Voyager
asked in Data Quality & Governance ⚙️
How to increase the speed of profiling data for using Oracle DB?

Hello, I have a speed of profiling data for using Oracle DB of about 4000-7000 rows per minute.I use Ataccama DQ Analyzer (Version: 11.1.1.ga-2017-06-14).  Please, can you take me to advise me on how to increase the speed of profiling data for using Oracle DB?I hope to take the profiling speed to about 100 000 rows per minute...  Connation config:   Log:17.03.2023 12:56:04 [INFO] Switching to logging setup from C:\Users\ECHERN~1\AppData\Local\Temp\loggingConfig3307204922660014311.tmp17.03.2023 12:56:04 [INFO] Using following licenses:17.03.2023 12:56:04 [INFO] C:\Ataccama DQ Analyzer 11\runtime\license_keys\license.plf17.03.2023 12:56:04 [INFO] Using DQ Analyzer configuration file C:\Ataccama DQ Analyzer 11\workspace\sqb_prof_pilot\data\out\sqb_all_table.plan17.03.2023 12:56:04 [INFO] Using runtime configuration file C:\Users\echernoskutov\AppData\Local\Temp\runtimeCfg1065525328424309714.tmp17.03.2023 12:56:04 [INFO] Changing CPU affinity to 4 CPU(s).17.03.202

1372
E
Data Voyager
1 year ago
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
published in Data Quality & Governance ⚙️
Catch OneWeb GraphQL in Chrome/Edge test and useBest Practice

We are using some GraphQL to ‘do stuff’. Turns out the easiest way to find the GraphQL is using the Developer-tools in Chrome or Edge. See enclosed. 

1303
M
Star Blazer L2
1 year ago
anna.spakova
Ataccamer
anna.spakovaAtaccamer
published in Data Quality & Governance ⚙️
Export metadata model into ExcelBest Practice

Hello everyone, Anna here! I work as a projects consultant in Ataccama and today I put together a guide for you on how to export a metadata model definition from Ataccama ONE.Ataccama ONE allows to export metadata model definition from the platform into a JSON file. However, the JSON file is not an ideal format for business users. For that reason, parsing of the JSON into more user-friendly format, for instance Excel, is desired. The current JSON parser step in Ataccama DQC, however, is not able to parse this JSON structure.This article introduces a Python script that was developed to parse the JSON structure of the metadata export and provide the metadata model in Excel and CSV formats. It is supported for v13 of Ataccama ONE.As a first step, the metadata model has to be exported from Ataccama. Next step is to run the Python script on top of the exported metadata model to obtain the CSV and Excel format. Chapter JSON parser in Python describes in detail the Python script as well as th

2860
anna.spakova
Ataccamer
1 year ago
S
Data Voyager
smunotData Voyager
asked in Data Quality & Governance ⚙️
MCQs in Certification

What kind of MCQs are asked in Certification exam?

881
DannyRyan
Ataccamer
1 year ago
S
Data Pioneer
sgillaData Pioneer
asked in Data Quality & Governance ⚙️
NULL Valued fields on DQS

How are NULL’s considered? I have a case statement which gives the right value matching measure -used debug to test. But when it comes to matching rules, it skips the rule. Example:Name      NPIRamona  12345678NULL       12345678the 2nd row should match the 1st on NPI value and considering on of them is null.Even the scenarioName      NPINULL       12345678NULL       12345678these two should match together based on NPI. I do not see a special character. If I replace NULL with a value ‘A’ , they match with rule Matching on NPI. but does not work for NULL.  Any thoughts or suggestions appreciated. 

956
S
Data Pioneer
1 year ago
J
Data Voyager
JerinData Voyager
asked in Data Quality & Governance ⚙️
Results message not showing as message.

Hello, I’ve got a question while using DQ evaluation rules.I have been creating rules with valid and invalid messages but just the invalid message at the end does not display as intended. But when I view this in the results tab it just says OTHER as the explanation.    Is there a workaround to this problem? Should I edit my rule logic to include some specific condition and leave the rest to fall under the ‘IF none of the conditions above apply THEN’ TIA for the replies.  

672
J
Data Voyager
1 year ago
A
Ataccamer
alinAtaccamer
published in Data Quality & Governance ⚙️
Data Quality and Pipeline ExecutionBest Practice

Hello, here is a short video to illustrate how an organization might use Ataccama Data Quality to run checks as part of an existing data pipeline, and verify data integrity before further processing. With Ataccama, you can use various data warehouses or ETL tools, and GraphQL API integration allows conditional scheduling and more complex orchestration.In this demonstration, we use DataBricks Notebook to trigger DQ execution and retrieve the results. These determine whether to halt or commence the pipeline, and in-app dashboards display the metrics. All results and metadata can also be exported to a data mart table for further analysis and visualization.The platform is designed to provide visible and usable data quality insights, and we offer flexible integration capabilities to suit any data stack. Thanks for watching. 

2420
A
Ataccamer
1 year ago
S
Ataccamer
SathyaAtaccamer
published in Data Quality & Governance ⚙️
Adding a New Entity in MMM ModelBest Practice

METADATA MODELThe metadata model is used to configure the application entities and their properties. The Metadata Model describes the entities and their properties in the application.  Web Application) use the Metadata Model to create the frontend user interface. The metadata model is also responsible for the configuration of the left navigation menu of the application.ONE METADATA PROPERTY REFERENCESEntities consist of properties of Simple types, such as strings and integers, and Object types, which link the entity to other entities.ADDING NEW ENTITY TO MMM MODELIf you are creating an entity using the user interface, you must then add it as a property of the root entity metadata and then publishFor example, if we need to add a new entity under Business Glossary named (TestEntity) to the MMM modelClick on the Global settings on the left panelChoose Metadata Model under Maintenance CenterWe must create a new entity under Term so click on create entityFill in the name of the Entity, and

741
S
Data Enthusiast
1 year ago
E
Data Voyager
EllieData Voyager
asked in Data Quality & Governance ⚙️
Export profiling result

Hi Team, Is there anyway that we can export the result of profiling in ataccama? Thanks, Ellie

1141
O
Star Blazer L1
1 year ago
K
Universe Traveller
KP_DQUniverse Traveller
asked in Data Quality & Governance ⚙️
Extracting data quality results- deatiled failed records

Hi,We are using Ataccama one 12.6 version. We have a project containing 40+ models with 90 + rules. I am looking for an options to get the detailed invalid/failed records for each model. Like all the attributes from the model along with invalid rule instance names.The idea is to get the whole failed dataset to automate the remediation process. We already have automated process to remediate the data using other technologies.I don’t want to use the download results in csv option, because we are transitioning to latest version of Ataccama.I am looking for couple of optionsExtract the detailed failed records from backend tables using sqls (where Ataccama stores the results) or would like to understand how those csv files are generated in backend and use the same way to get the data and store in  db. Reading data through Ataccama inbuilt API’s and loading into a db. I am  going through the GraphQL options. Could you please let me know, If there is way to get the similar data through GraphQL

2655
Cansu
Community Manager
1 year ago
S
Data Pioneer
sgillaData Pioneer
asked in Data Quality & Governance ⚙️
subject_matching.online: Error message: Online configuration is not valid!

During bundle deployment I see this error. I have tried different combination ofUpdate to old bundle. Manually update the component again on the existing old bundle and upload Copying extended unification Updating OmnigenServer folders (bundle,cmd,dbms) with the backup. restarting services as needed.Any thoughts or suggestions? Thank you.

363
S
Data Pioneer
1 year ago
S
Data Pioneer
sgillaData Pioneer
asked in Data Quality & Governance ⚙️
DQC match - slow incremental load

The matching component is slow at the extended unifications step for a incremental load. When a record like ‘WALGREEN%’ is loaded and there are 20K records with similar name on the instance, the records are all considered through the process of matching. Hence the process is slow when processing the increment and even if the incremental load has 11 records to process 30K records have the transaction Id and omni_modified date updated with the work order related on the instance table.  When I check history, nothing changes on these records except the omni_modified_date and transaction_id to that of the work order. Tweaking the union keys to create combination of name and key factors or adding separate groups for the high count records is not helping. Any thoughts or suggestions?

876
S
Data Pioneer
1 year ago
C
Universe Traveller
C_RUniverse Traveller
asked in Data Quality & Governance ⚙️
What is the best mechanism or approach to share the desktop html profile results?

Hello everyone,I created a profile of my catalog item using ONE desktop and I would like the output(.html) to be accessible to business analyst with read access. The html is only accessible on my local computer. Where’s the best / secure location to store output reports of profiling? Please chime in with any thoughts. Thank you! 😀     

451
Maxim Kim
Ataccamer
1 year ago
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Governance ⚙️
Comparison of two columns for matches in One Desktop

Hello everyone!I have an interesting task related to the desktop, we have a table combined with certain attributes and with full names, full names are divided into columns (firstname, lastname, midname) and they need to be compared with other columns (firstname1, lastname1, midname1) it is necessary to output non-matching full names. I can't compare one table between columns, I need your help. Do you know how to implement this?I will be very grateful for your help

911
A
Ataccamer
1 year ago
T
Data Voyager
TamilData Voyager
asked in Data Quality & Governance ⚙️
Identification of new Dataset using Detection Rule

Is it possible to Set up the detection rule, which would allow to detection of new table(s) within the Snowflake schema and trigger the required potential alerts

611
Maxim Kim
Ataccamer
1 year ago
M
Data Pioneer L2
MayKwokData Pioneer L2
published in Data Quality & Governance ⚙️
Integrating Ataccama with JiraBest Practice

Imagine the scenario where you have Ataccama ONE Data Quality Monitoring Projects running regularly. Any Data Quality (DQ) issues are already exported to our DQ Issue Tracker (DQIT) module. You can also see trends of DQ over time within the Ataccama ONE Web application.You start noticing that the score for some DQ rules has had a sharp drop in quality, and you want someone to investigate this.One of our customers decided that they want to integrate Ataccama with Jira at this point. Have Ataccama raise a ticket into Jira when this happens automatically.In this article, we will explore:Preparing to integrate with Jira The Json Call stepPreparationsBusinessUnderstand your end users. What do they expect to see in their Jira ticket? Know which fields in the Jira ticket are mandatory Have an idea of what extra information you should supply in the detailed description of the ticket. Think about formatting and presentation. Good to understand Jira markdown notation (link)JiraKnow which version

4180
M
Data Pioneer L2
1 year ago
T
Data Voyager
TamilData Voyager
posted in Data Quality & Governance ⚙️
Is it possible to do Duplication check (Uniqueness) on Table level using Data Quality

Hi Team,Our Aim is to Avoid duplication of the same data coming from different requirements (project trains) - in case of identified duplication DQ Brach alert should be reported.      1.How to identify if Somone ingested the same table in a different Snowflake schema ?Is it possible to detect duplicate table through ONE web application or ONE Desktop by creating a rule?Thanks in Advance!

970
T
Data Voyager
1 year ago
S
Data Voyager
samyData Voyager
asked in Data Quality & Governance ⚙️
How to parameterize data source name or how make value in data source name as dynamic in components?

I have a sql select where in I wanted to pass multiple data source names. how to make it dynamic or parameterized so that It can read all the names one by one.

841
A
Ataccamer
1 year ago
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
posted in Data Quality & Governance ⚙️
Re-usable published components

Hi,For our monitoring projects  post processing we use fairly standardized (sub-) components. Now for every MP / CI combination these sub-components are all the same. If an MP Postprocessing is published these components get published also, right?Is there a way to have a single version of a component stored?This would make maintenance and life-cycle-management on these components a lot easier. In stead of having to go in the PostProcessing main component, we'd just have to change one.A plausible solution would be to split things and do things asynchronously, maybe. But that is a more of a workaround.Any ideas?

430
M
Star Blazer L2
1 year ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings