Skip to main content

Data Quality & Governance ⚙️

Discuss, learn, ask, suggest, share all things DQG 🔍

  • 552 Topics
  • 1,329 Replies
552 Topics
U
Data Voyager
UmaData Voyager
posted in Data Quality & Governance ⚙️
Publish Atacamma profiling results runtime via PubSub on Azure?

Hi We would like to publish the profiling results of a scan on to Azure service so that we can pick up those results and update it back to another reporting systems via Kafka or Azure Pubsub.What we are mainly interested is in how to achieve sending publish results to Azure post scanning in runtime without impacting the performance on PostgreSQL DB where profiling results are getting logged. Did any one encounter similar scenario and how did you achieve it.Regards,Uma

511
S
Ataccamer
4 months ago
V
Data Voyager
vdatlaData Voyager
asked in Data Quality & Governance ⚙️
Ataccama Naming Best Practice

Hi everyone, Just wondering if Ataccama provides any best practice guides on naming conventions when working with the platform. This includes best practice for rule names, error names, sql catalogue item names etc.  Kind Regards,  Vishy

663
Cansu
Community Manager
4 months ago
P
Data Voyager
PrithikaData Voyager
asked in Data Quality & Governance ⚙️
Masking Sensitive Attributes

Hello, Is it possible to mask attributes that are classified as restricted/confidential using Ataccama One Web? I was able to assign rules and terms to attributes, but I was not able to create/apply any masking methods. Please let me know if anyone has tried this before. Thanks in advance!

734
Cansu
Community Manager
4 months ago
S
Data Voyager
SoniyaData Voyager
asked in Data Quality & Governance ⚙️
Regarding Completeness DQ control

Hi Team,I am trying to find how we can achieve completeness DQ control by comparing data in the file and the table irrespective of the position of primary key in either of them. Also, need to know how we can achieve the same by using the composite keys.

342
Cansu
Community Manager
4 months ago
V
Universe Traveller
Vanja GilbertUniverse Traveller
asked in Data Quality & Governance ⚙️
Invalid samples for sections

Hi,I have a question about the display of invalid samples when a specific section is selected. It's helpful that the invalid samples are based on that section's checks, but I'm wondering why all attributes are shown by default.Wouldn't it make sense to show only the attributes relevant to the failed rules, with an option to add more if needed? The current setup is a bit confusing.Thank you!

304
Cansu
Community Manager
4 months ago
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
asked in Data Quality & Governance ⚙️
Relationship type visualization / Graph visualization

Hi community,Does anyone know about documentation about visualizing relationship types? On the documantation portal I found this, but that doesn't seem to cover the setup in the Graph Visualization section in the Application Settings. The documentation mentions “For more information, see Configure Graph Visualization.”, but that is a link to the page you are already on.Kind regards,Albert

554
Albert de Ruiter
Star Blazer L2
4 months ago
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
asked in Data Quality & Governance ⚙️
Case statement in computed content

Hi community,We have a usecase to set a property's value automatically: based on several conditions that property gets a value. First we considered creating a desktop plan for this. Then we figured it would be preferrable to have this value set automatically, something that can possibly be accomplished by applying computed content.We have used computed content before, mainly for concatenating values (like described in Computed Content: Show attribute values based on existing attributes (ataccama.com)). But in this case we would need to define a case statement to define the conditions and determine the value.So the question is, does someone know if sql-like case stamenent can be implemented as computed content? And if so, can an example be provided?Thanks for your attention and kind regards,Albert

775
Albert de Ruiter
Star Blazer L2
4 months ago
M
Data Pioneer
Maria GolikovaData Pioneer
asked in Data Quality & Governance ⚙️
Renaming Monitoring projects and DQ rules

Hello Dears! Would someone advise if there any watch outs once I want to rename Monitoring projects and DQ rules on ONE Web?I have running projects and DQ rules naming convention of which I would like to improve. If I just simply change names, won’t it have negative impact anywhere?Thank you. Kind regards,Maria

593
M
Data Pioneer
4 months ago
A
Ataccamer
Antonio DeChausayAtaccamer
published in Data Quality & Governance ⚙️
Creating Rule Folders in ONE Web

Hi Community I’m Antonio, a Professional Services Consultant here at Ataccama. Assets created in ONE do not come with the option of being sorted into folders out of the box. This guide will demonstrate how to set up folders for rules, but can be used for other asset types and isn’t specific to rules. In this guide we will describe how to set up a custom attribute ‘ruleFolderValueList’ on rules which will hold data from a list of values, then create new navigation sections in the tool where we can apply AQL expressions to filter rules based on our set criteria. This will allow us to split rules into folders based on the value that has been entered into the custom attribute. The process and results outlined have only been tested in v13.9.3 and v14.5.0 of the ONE Web application. Create Value ListFirst we will create the list of values. To do this follow the below stepsGo to Global Settings > Metadata Model. Select Create entity.Provide the following information:Name of the value that

1723
M
Data Pioneer
4 months ago
A
Data Voyager
AdilhussainData Voyager
asked in Data Quality & Governance ⚙️
How to Parameterize input to DQ rule?

I would like to setup a generic data quality rule to check string length. Requirements for length checks differs per attribute. Say column1 needs to have max length of 10 column2 needs to have max length of 15 and so onIs it possible to create a generic rule to check string length, passing value of length as parameter?I am aware of attribute length insights available as part of DQ profiling results but I would like to make the check as part of monitoring project. Thanks!

1012
greglvaughan
Data Enthusiast
4 months ago
R
Data Voyager
RadziahData Voyager
asked in Data Quality & Governance ⚙️
Summary of DQ Checks and Results in Ataccama ONE Desktop

HiI would like to get a summary of DQ checks and results like the one we have in ONE web like how many in % and count of broken rules, how many are passing and how many records do we have in total etc. in Ataccama ONE desktop. I tried looking out any related topic in the community but I could not find one. Is it possible to achieve this via Ataccama ONE desktop as we have an existing plan comparing data fields in two different sources. Do you mind sharing the suggested steps to achieve this? Thanks in advance!

14210
S
Ataccamer
4 months ago
K
Ataccamer
KaterinaAtaccamer
published in Data Quality & Governance ⚙️
Customizing DQ Monitoring Projects Results Export

When it comes to data quality (DQ) management, monitoring projects provide us with ability to export DQ results and maintain a close eye on data quality. However, the initial export contains attributes organized in columns, alongside additional columns housing information regarding the DQ results for these attributes. Analysing this data can be challenging because DQ rule results and explanations are presented as a single string.Despite the fact, that extracting DQ results in a customized format might require a more detailed approach, it might be better aligning with the client's specific needs. In this guide, you will find information how to export DQ results tailored to specific requirements.The Task at Hand:Our objective is to ensure a separate line for each attribute in an export file, with the following columns related to DQ results:the name of the DQ rule; an explanation for the check; a list of terms assigned to that attribute; the attribute value.We have three main steps:Attrib

2681
greglvaughan
Data Enthusiast
4 months ago
S
Ataccamer
SathyaAtaccamer
published in Data Quality & Governance ⚙️
Creating New Entity Data Classification And Adding New Relationship Types To MMM Model.

Hi everyone, I'm Sathya, Solutions Consultant at Ataccama. In this post I am going to discuss about adding Data Classification and relationship types. Step 1: Go to Global setting and you will see Metadata Model, click on Create Entity   Step 2: Once Create Entity page opens, fill the Name of the Entity: Data Classification Step 3: Click Add Property and fill in the property needed for the Entity: Name and Description with its data type and hit save Step 4: Goto Metadata Model under Global settings and search for metadata Step 5: Click Add property: Fill in name, type: Embedded Object Array and Object type:dataClassification Step 6: Goto Metadata Model under Global settings and search for term Step 7: Click add property and Fill in data Classification type as Referenced Object, PII and CDE as Boolean   Step 8: Hit Apply changes and the System will be Switching to Maintenance mode Step 9: Click DB backup and the application restarts.  Step 10: Click on Run changes again and all the chan

290
S
Ataccamer
4 months ago
O
Data Voyager
olayinkadaramolaData Voyager
asked in Data Quality & Governance ⚙️
Rules- with many conditions

Hi, is it possible to have a  single rule that checks many condition. example is below   i want a single rule named--Name Validationi want this rule to check  and validate name that are in -Title Case --first letter upper case follow by all lower case  Accented--- Accepts accented character  Accepts special characters  Could have Hyphen - the rule should should invalidate any other names that does not meet these conditions . or is there a better way to do this ? preferred to break it down so its will be easy to know the reason why the names are not valid whenever its does not matches 

352
Cansu
Community Manager
4 months ago
M
Data Voyager
michael.willisData Voyager
asked in Data Quality & Governance ⚙️
Call Runtime-Server OnlineService using Azure Synapse

Hello,I have a runtime-server online service that I am trying to call using Azure Synapse using OAuth authentication. I created a REST Linked Service in Azure that connects successfully to the runtime-server online service. I am using a copy activity in Azure to do a POST call on the online service. When running the copy activity I get the following error:{ "errorCode": "2200","message": "Failure happened on 'Source' side. ErrorCode=UserErrorCharSetInvalidOrNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The CharSet '\"UTF-8\"' is invalid or not supported from Response.Headers.Content-Type.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentException,Message='\"UTF-8\"' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method.\r\nParameter name: name,Source=mscorlib,'", "failureType": "UserError", "target": "CopyDataCatalogToRaw","details": [] }T

252
Cansu
Community Manager
4 months ago
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
asked in Data Quality & Governance ⚙️
Filtering in metadatareader on mutation datetime

Hi Community,Would there be a way in Ataccama Desktop to select records from for instance entity term that have been inserted/updated/deleted from a certain datetime, like the Published on datetime in the History tab? My intention is to use the metadatareader to read the details from term and then use this mutation datetime in the filter in the metadatareader.Kind regards,Albert

13615
Cansu
Community Manager
4 months ago
P
Data Voyager
PrithikaData Voyager
asked in Data Quality & Governance ⚙️
Masking Attributes and Exporting

Hello, I’m trying to mask certain attributes in a table and be able to export them while exporting the table as a file. Is it possible to mask them as part of the JSON code while creating an export plan or create masking rules and apply it to the attributes and then export them. (using ataccama ONE Web). Has anyone tried doing this before or has any idea about this? Please let me know!Thank you,Prithika  

932
Cansu
Community Manager
4 months ago
B
Data Voyager
boopeshmData Voyager
asked in Data Quality & Governance ⚙️
Preventive Data Controls using Ataccama

Hi Everyone,We are in the process of transitioning to Ataccama as our DQ solution. In our current setup, we perform various data controls such as:Empty File Check Duplicate File Check Header or Trailer Validation (e.g., validating trailer record count or checksum)We are keen to understand if these types of data controls can also be implemented using Ataccama. Our goal is to consolidate all data quality processes into a single tool for better efficiency and management.Thank you!

382
Cansu
Community Manager
4 months ago
S
Ataccamer
SathyaAtaccamer
published in Data Quality & Governance ⚙️
How to add relationship types in Ataccama

Hi everyone, I am Sathya, Solutions Consultant at Ataccama. In this post I am going to discuss about how to create Relationship types(Source and Target reference). Step 1: To add new relationship type, go to Global settings on the left navigation bar. Step 2: Under Application Settings, choose Relationship types  Step 3: Available Relationship Types will be listed.  Step 4: Click on create button on the top right corner of the screen layout.  Step 5: Fill in the name, Source rel name and Target rel name of the Relationship Type  Step 6: DomainOf relationship type is created, with its source rel name and Target rel name with its ID.  Step 7: All the relationship Type is listed along with the newly created DomainOf.  I hope you found this article useful, please let us know if you have any questions or thoughts in the comments below.  

410
S
Ataccamer
5 months ago
O
Data Voyager
olayinkadaramolaData Voyager
asked in Data Quality & Governance ⚙️
Rules on Names

Hello, I am trying to create a rule for validity on Names. The logic and condition is that the rule matches names that are Title Case --first letter upper case follow by all lower case  Accented Accepts special characters  Mandatory Could have Hyphen - I have this regex ^[a-zA-Z '-]+$ ^[A-Z][a-z]*(?:[ '-][A-Z][a-z]*)*$(?<=^|\s)[a-z]*[A-Z][a-z]*[A-Z]*[a-z]* above regex capture and validate names like John DoeMary Anne which are okay.  but also validate name in all lower case for example john doe which is not okay. i want to know what i need to do to allow the regex not validate names in all lower case . 

8613
S
Ataccamer
5 months ago
S
Ataccamer
srija piratlaAtaccamer
published in Data Quality & Governance ⚙️
Profiling Triggered While Running Monitoring Project

Hello everyone !  I’m Srija ,Solutions Consultant here at Ataccama. Today, I’d like to talk about how to trigger profiling while running a monitoring project.While running the monitoring project, the profiling process is getting triggered without any schedule or event created on the catalog item to initiate profiling. This unexpected profiling of over 1000+ records significantly slows down the system. The likely cause for this issue is the anomaly detection feature being enabled on the catalog item.To address this issue, you need to check if anomaly detection is enabled on the catalog item.    Anomaly Detection in Ataccama DQWhat Anomaly Detection Does:Anomaly detection in Ataccama Data Quality (DQ) automatically identifies outliers and unusual patterns in your data. This feature is designed to ensure data quality by detecting anomalies such as unexpected values, missing data, or patterns that do not conform to the expected behavior.Implications of Enabling Anomaly Detection:Automatic

390
S
Ataccamer
5 months ago
S
Ataccamer
SathyaAtaccamer
published in Data Quality & Governance ⚙️
Adding Data Domain Entity to the MMM Model

Hi everyone, I'm Sathya, Solutions Consultant at Ataccama. In this post I am going to talk about creating Data Domain , relationship using AQL expression. Step 1: On the organization tab, go to Global settings Step 2: Click Navigation Icon Step 3: List of entities will be listed, under Business glossary we will have Data Domains Step 4: Click on the 3 dots and click Add Child option to add a child to the list. Step 5: Pop up window appears, fill in the details as follows:Navigation item type: Listing Listing Node: Term Icon: Folders Text Medium Name: Fill the name of the Data Domain Save and publish.    Step 6: All the terms will be added to the Data domain. To filter out we have to update the AQL expression. Step 7:Click on the 3 dots on the right corner of the terms page where the data domain is displayed and choose Edit page. Step 8: In the Filter part we must fill in the AQL Expression Step 9: Syntax for AQL expression:name is "(name of the dataDomain)" OR @termRelationship(target)

760
S
Ataccamer
5 months ago
C
Universe Traveller
C_RUniverse Traveller
asked in Data Quality & Governance ⚙️
How to rename Post Processing Plans

Hi all, I have a group of PPP that need to be renamed. Is it possible to update name once published? How? Thanks

302
Cansu
Community Manager
5 months ago
N
Data Voyager
NazrinDData Voyager
asked in Data Quality & Governance ⚙️
combine email with ataccama MP results

I want to configure one general email notification that will be used for all of my monitoring programs. I would prefer to receive a single email notification that compiles the findings from various monitoring projects rather than the numerous emails I usually receive for each monitoring project. I'm aware that I should use One Desktop, but I'm not sure how to proceed precisely. Well, I know that I have to call monitoringProjectProcessing which allows me to  retreive the finish time with processing ID. And then I have to complie to get the ID link of notification but the problem is here that I dont know the step exactly. It would be great if you could help.Thanks in advance.

632
Cansu
Community Manager
5 months ago
S
Data Pioneer
sumeyraData Pioneer
asked in Data Quality & Governance ⚙️
Writing data to One Data from One Desktop

Hello all 🤓I would like to write my post processing data (export file of my DQ results) to One-Data (database) from One Desktop. Since one-data is not jdbc, I couldn’t use JDBC writer. It shows red icon when I want to try to use this step. Can anyone recommend what can I do here? Thanks ! 

533
Cansu
Community Manager
5 months ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings