Skip to main content

Learn & Share

Discuss data, share best practices, learn, and grow with Ataccama's Community.

888 Topics
alexAguilarMx
Data Pioneer
alexAguilarMxData Pioneer
asked in Master Data Management & Reference Data Management 🛰️
how to apply new changes into clean operation for records already in the db?

We made changes into the clean operation of an entity. it will take effects for new transactions.How can we apply this to the current records that we already have into the database? We are exploring use a full reprocess of this entity:but we have some doubts about the setup. For example:What is the difference between a reprocess vs rematch checkbox? We tried to read the documentation but there is not detail explanation about it. Thanks in advance!

785
Ales
Ataccamer
O
Data Enthusiast
onufryData Enthusiast
asked in Data Quality & Catalog ⚙️
Best Practices for Searching JSON/XML Values in Database Columns Using Regular Expressions

I'm working with a database table that contains a column storing complicated JSON/XML data structures. I need to implement data quality rules that will search for and validate specific values within these structures and compare them with values in other columns.My specific requirements: Need to search for particular patterns and values nested within JSON/XML structures Considering using regular expressions for this search capability Concerned about the performance impact on database resources Want to ensure this approach is sustainable as data volume grows Has anyone implemented data quality rules for searching/validating values within JSON or XML columns? Are regular expressions an appropriate approach, or are there better alternatives within Ataccama (Desktop?) for handling hierarchical data structures?I'm particularly concerned about performance impacts and would appreciate insights from those who have solved similar challenges.

30
O
Data Enthusiast
V
Data Voyager
vishnuData Voyager
asked in Data Quality & Catalog ⚙️
How to insert Columns from a Plan to Send Mail email body?

I have created a plan for data quality checks using Data Quality Indicator, and it will produce an excel file which flags each records with error_messages corresponding to each checks, and a summary txt file. Now, I have created  a workflow using Run DQC and Send Mail step. The requirement is to soft code the below values in the email template.Summary excel fileBelow is the email body:Dear Business Users,Please find the attached Error Result (Error_Report.csv), which details the records that failed validation and their respective error types.Summary:Total Processed Records(Total): Total Error Records(Success_Count): Error Types(Code): We recommend reviewing the attached file and taking the necessary corrective actions. If you need further clarification or assistance, please feel free to reach out.Best regards, 

20
V
Data Voyager
B
Data Voyager
bobparryData Voyager
asked in Data Quality & Catalog ⚙️
Ataccama Desktop Iterate Tool Erros

Hello!Does anyone have experience using the Iterate tool within the Ataccama Desktop, when creating a workflow? I’ve found a couple of old community posts referring to it, along with the files they’ve used, but I can’t get it to work in my environment. see the following:Iterating rows of a table (Record-by-record - SQL Iterator) in a workflowBased on that post and my testing, it seems like the Iterate tool needs to call another workflow, so I’m calling another .ewf file that just has the Run DQC tool, which runs the .plan file. I’m able to succeed in running the .plan file from the Run DQC .ewf, however i can’t seem to get the Iterate tool to run, without getting Configuration errors, like the following:12.03.2025 15:30:51 [INFO] Task 'Iterate' : failed with exception: com.ataccama.adt.internal.core.runtime.EwfConfigException: Configuration contains errors, exitingjava.lang.RuntimeException: com.ataccama.adt.internal.core.runtime.EwfConfigException: Configuration contains errors, e

40
B
Data Voyager
A
Data Voyager
akshayl09Data Voyager
asked in Data Quality & Catalog ⚙️
Data Profiling schedule with failure notification

Is there any way to get notification for data Profiling schedule failure with log details or successful.

262
A
Data Voyager
T
Universe Traveller
TimBrown74Universe Traveller
asked in Data Quality & Catalog ⚙️
Confusing date data

The Business wants this date attribute validated.  I don’t know why, it’s a key field and I assume managed by the DB.  Nonetheless, I need to build a rule.But I think it’s strange that the Data Type is DATE while the date values are YYYY-MM-DD. I would expect to see DATE format data in the MM-DD-YYY HH:MM:SS AM format.I don’t know how to code it in the date functions since it doesn’t have the standard date format and it doesn’t work in any string functions.Any ideas?  

626
T
Universe Traveller
anastasiia.popova
Ataccamer
anastasiia.popovaAtaccamer
published in Data Quality & Catalog ⚙️
Understanding general vs. specific notifications and how to enable email notifications

Hi everyone, Managing email notifications effectively is essential for monitoring Data Quality and ensuring timely alerts. In this post, we will explore the differences between general and specific notifications and how to enable specific email notifications, as they are turned off by default in self-managed installations.What is the difference between general and specific notifications?General notifications are triggered based on predefined system-wide scenarios, such as monitoring failures or anomalies. These notifications alert users to general events occurring in the system, ensuring broad visibility into key operational issues. However, general notifications do not include detailed Data Quality results.On the other hand, specific notifications provide more granular alerts, tailored to particular data quality conditions, thresholds, or projects. These notifications are linked to specific configurations, allowing users to receive alerts based on structured errors, Data Quality issue

70
anastasiia.popova
Ataccamer
R
Data Voyager
RadziahData Voyager
asked in Data Quality & Catalog ⚙️
Replicate LISTAGG(DISTINCT columnA, ' | ') OVER (PARTITION BY columnB) in ONE Desktop

HiI have a sub-query in Snowflake that is using the function of LISTAGG and PARTITION BY in multiple SELECT DISTINCT statements for example LISTAGG(DISTINCT TRIM(SUBJECT_REF), ' | ') OVER (PARTITION BY IDENTIFIER) AS EN_NUM_AGG, kindly see attachment to see the full SQL statement. Example of one of the subqueries: …en AS(SELECT DISTINCT     IDENTIFIER,    ID,    LISTAGG(DISTINCT ID, ' | ') OVER (PARTITION BY IDENTIFIER) AS ID_AGG,    TRIM(SUBJECT_REF) AS EN_NUM,    LISTAGG(DISTINCT TRIM(SUBJECT_REF), ' | ') OVER (PARTITION BY IDENTIFIER) AS EN_NUM_AGG,    DQ_EN_CARD,    LISTAGG(DISTINCT DQ_EN_CARD, ' | ') OVER (PARTITION BY IDENTIFIER) AS DQ_EN_CARD_AGG,    DATA_SOURCE,    ORIGINAL_SOURCEFROM    db.schema.view1    WHERE     subj_type = 'English') ...I wanted to replicate the same in Ataccama ONE Desktop using the Group Aggregator step, whereI allocate each sub-query in one aggregation set with a condition of subj_type = '<value>' in the When condition. For each column in the SELE

231
Phil Holbrook
Ataccamer
E
Data Voyager
EllieData Voyager
asked in Data Quality & Catalog ⚙️
Export profiling result

Hi Team, Is there anyway that we can export the result of profiling in ataccama? Thanks, Ellie

1392
S
Data Voyager
A
Space Explorer
Ayush kumarSpace Explorer
asked in Data Quality & Catalog ⚙️
Arithmetic operations on Lookup using DQ evaluation rule in Ataccama One ui

I am making a DQ evaluation rule in ataccama one ui where I have to subtract each long value of catalogue item from its corresponding long value in lookup and if the difference is greater than 100, I want to raise that particular value as invalid sample, so far I am not able to achieve this, it would be great if anyone can help me with this.

654
A
Space Explorer
A
Data Voyager
akshayl09Data Voyager
asked in Data Quality & Catalog ⚙️
Duplicate Record in the table based on all the columns

Is there a method to create a single rule for identifying duplicate records based on all columns in a table? This rule should be applicable to all tables, regardless of the number of columns they contain.Current situation we need to add the column to use aggregated rule. It is very difficult if the columns are more than 10 or 15 in table.  

50
A
Data Voyager
T
Universe Traveller
TimBrown74Universe Traveller
asked in Data Quality & Catalog ⚙️
Joining SQL Views and Data Slices to DQ Rules

We’re using version 15.4, mostly in the web tool.We’re trying to a subset of our input tables for testing before implementation the full table(s) in Production.We looked at using Data Slices, but we found that the ROWNUM feature of SQL met our purpose better: select * from (select * from dbname.tblname) where ROWNUM <=10000Our primary question is how do we link/connect the SQL View and/or the Data Slice to the DQ Rule.  We believe that it occurs in the Monitoring Project. I looked thorough the Ataccama online documentation, and the Community, but I can’t find the options to make these either of these connections.Any ideas!Thanks!Tim 

282
T
Universe Traveller
S
Data Pioneer
sumishaData Pioneer
asked in Data Quality & Catalog ⚙️
AQL to filter catalog by Term name

What is the AQL to filter all catalog items’ attributes tagged with a particular term?

61
S
Data Pioneer
G
Universe Traveller
GloriaUniverse Traveller
asked in Data Quality & Catalog ⚙️
DQ Firewall API Integration

Can the API response generated from the Data Quality Firewall feature be edited?

231
E
Ataccamer
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Catalog ⚙️
importing catalog item descriptions and attribute descriptions from dbT docs

We are using dBT for our data pipelines, and for the tables and fields, there is description maintained in dBT that we would like to export (json) and update Ataccama Data Catalog.Is there a known, trusted, supported way to automate this?I already do have a desktop plan i created that can update catalog objects and attributes based on a excel input file. I may be able to modify that if thats the only option. Iam reaching out to the community to see if there is a preferred way to get descriptions and documentation from dBT to Ataccama.

1153
B
Data Voyager
A
Data Voyager
anikData Voyager
asked in Data Quality & Catalog ⚙️
Get all the workflow names from runtime server

Is there a way to retrieve all the workflows deployed on the Ataccama runtime server? For instance, we can obtain all the monitoring projects and post-processing details from the Ataccama metadata model. Additionally, I would like to get the workflow names along with their associated plans and component files from the runtime server.

252
A
Data Voyager
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
posted in Data Quality & Catalog ⚙️
Relationship type, or not?

Hi all,I would like to share the guidance of Ataccama Support regarding a question that I had with respect to using relationship types. I suppose the question will become relevant for us all at some point. So hereby the question and Ataccama's response. Question“In the metadata model we can define relations between entities by means of references, or we can embed properties from other entities etc.Another way to define relations between entities is by means of relationship types. The latter option allows us to visualize the relation in for instance a relationship tab, like there is for terms.Suppose you are to model new entities. There is no requirement to have this visualized in a relationship tab. So you define a reference from one entity to the other. Later a requirement pops up from a new project, that the relation should be depicted in the relationship tab. So, unfortunately, not possible. Changing the reference into a relationship type is a major system change. Then always choose

111
Cansu
Community Manager
V
Data Voyager
Varsha KumariData Voyager
asked in Data Quality & Catalog ⚙️
Add location feature

Hi all, Can anyone help me with what this Add location feature do which is available in overview tab of Source connection? 

131
Cansu
Community Manager
anastasiia.popova
Ataccamer
anastasiia.popovaAtaccamer
published in Data Quality & Catalog ⚙️
A guide to Data Quality validation: Understanding % threshold in regex-based detection checks

Hi everyone, In this post, I’ll walk you through regex-based detection checks. Understanding regex-based detection checks is crucial for ensuring data quality, validation, and anomaly detection. One common question that arises is about the % threshold when specifying such a check.What does % threshold mean in regex-based detection checks?When applying a regular expression (regex) check, the % threshold represents the proportion of data entries that must match the regex pattern in order for the check to pass.For example, if you set the threshold to 70%, it means that at least 70% of the data elements must match the regex pattern for the detection check to be considered successful.Common misconception: Matching data vs. matching regexA frequent point of confusion is whether the threshold means:70% of the data should match the regex (Correct) 70% of the regex should match each data element (Incorrect)A regex pattern either matches or doesn’t for each data element - it does not partially m

470
anastasiia.popova
Ataccamer
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
Ataccama One Desktop - Select only distinct rows

Is there a way to quickly select only distinct rows in Ataccama One Desktop? (I now used a record descriptor and filter with Regex find(“.:.:1”, rd_column)

20610
DannyRyan
Ataccamer
S
Data Voyager
Susan24usData Voyager
asked in Data Quality & Catalog ⚙️
Post deployment plan filter

I applied two rules to a monitoring project, but I only want to export the results for one of the rules. In the post-deployment plan, I initially used a filter where the column is not null, and it worked fine. However, it exported all results where the column was not null, meaning the results of both rules were included. I would like to update the filter to only export specific invalid samples, but it's not working, as no data is being exported. Is this filter correct? Previously, it was "Column Name IS NOT NULL," and I updated it to "Column Name IS NOT NULL AND Column Name = 'XYZ specified rule’How can i resolve this ?

91
Albert de Ruiter
Star Blazer L2
S
Data Pioneer
sumishaData Pioneer
asked in Data Quality & Catalog ⚙️
Business Domain Analysis

How do we activate Business Domain Analysis option in Profiling section. 

282
S
Data Pioneer
M
Data Voyager
manidharData Voyager
asked in Data Quality & Catalog ⚙️
Export of DQ monitoring project results along with filters

Hi Ataccama communityI am trying to do the export of DQ monitoring project the results and use the filters in the export. I a post on this one ONE Desktop: Data Quality Results 🧑‍ on using the filters in the export data. I am able to get the filters extracted using the DQ monitoring project filter values but couldn't get the corresponding monitoring project results using the DQ Monitoring project check results. I am getting an empty file with just the column names in the file in the output file which I have checked the output into a CSV file just after the DQ monitoring project check results step. And I have enabled only one filter in the DQ monitoring project as I read in the community forum that two filter wouldn’t work with this setup.Are there any changes I need to make the way I am using the filter in DQ monitoring project check results step ? I am attaching the screenshots below for reference. 

191
C
Data Enthusiast
M
Data Voyager
manidharData Voyager
asked in Data Quality & Catalog ⚙️
Unable to find the SFTP upload file step in OneDesktop

Hi Ataccama Community, I want to upload a CSV file which is an export of DQ monitoring project results to SharePoint folder. I couldn’t see the Task SFTP upload file step in my OneDesktop (version 15.4). Is this step not available in the current version of Ataccama OneDesktop ? Or is there any work around to upload the CSV files to the Share Point server. CheersManidhar

82
Cansu
Community Manager
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Catalog ⚙️
Connect with Collibra

Hi,I am attempting to connect to Collibra using the Collibra reader (refer to the page: Collibra Integration in Ataccama ONE Desktop 🏹).However, I am encountering the following error message when I execute the plan. Could someone please assist me?  I have set up a server without a username/password because I don't need to enter a username/password when accessing Collibra.  

172
Cansu
Community Manager

Badge winners

  • First idea 💡
    Gloriahas earned the badge First idea 💡
  • First idea 💡
    farshid_mashaghhas earned the badge First idea 💡
  • First idea 💡
    sumishahas earned the badge First idea 💡
  • First idea 💡
    BarbaraBhas earned the badge First idea 💡
  • First idea 💡
    jbinghas earned the badge First idea 💡
Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings