Skip to main content

Data Quality & Catalog ⚙️

Discuss, learn, ask, suggest, share all things DQ&C 🔍

  • 788 Topics
  • 2,012 Replies
788 Topics
anastasiia.popova
Ataccamer
anastasiia.popovaAtaccamer
published in Data Quality & Catalog ⚙️
A guide to Data Quality validation: Understanding % threshold in regex-based detection checks
A guide to Data Quality validation: Understanding % threshold in regex-based detection checks

Hi everyone, In this post, I’ll walk you through regex-based detection checks. Understanding regex-based detection checks is crucial for ensuring data quality, validation, and anomaly detection. One common question that arises is about the % threshold when specifying such a check.What does % threshold mean in regex-based detection checks?When applying a regular expression (regex) check, the % threshold represents the proportion of data entries that must match the regex pattern in order for the check to pass.For example, if you set the threshold to 70%, it means that at least 70% of the data elements must match the regex pattern for the detection check to be considered successful.Common misconception: Matching data vs. matching regexA frequent point of confusion is whether the threshold means:70% of the data should match the regex (Correct) 70% of the regex should match each data element (Incorrect)A regex pattern either matches or doesn’t for each data element - it does not partially m

550
anastasiia.popova
Ataccamer
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
Ataccama One Desktop - Select only distinct rows

Is there a way to quickly select only distinct rows in Ataccama One Desktop? (I now used a record descriptor and filter with Regex find(“.:.:1”, rd_column)

28910
DannyRyan
Ataccamer
S
Universe Traveller
Susan24usUniverse Traveller
asked in Data Quality & Catalog ⚙️
Post deployment plan filter

I applied two rules to a monitoring project, but I only want to export the results for one of the rules. In the post-deployment plan, I initially used a filter where the column is not null, and it worked fine. However, it exported all results where the column was not null, meaning the results of both rules were included. I would like to update the filter to only export specific invalid samples, but it's not working, as no data is being exported. Is this filter correct? Previously, it was "Column Name IS NOT NULL," and I updated it to "Column Name IS NOT NULL AND Column Name = 'XYZ specified rule’How can i resolve this ?

131
Albert de Ruiter
Rocket Pioneer L1
S
Data Pioneer
sumishaData Pioneer
asked in Data Quality & Catalog ⚙️
Business Domain Analysis

How do we activate Business Domain Analysis option in Profiling section. 

292
S
Data Pioneer
M
Data Voyager
manidharData Voyager
asked in Data Quality & Catalog ⚙️
Export of DQ monitoring project results along with filters

Hi Ataccama communityI am trying to do the export of DQ monitoring project the results and use the filters in the export. I a post on this one ONE Desktop: Data Quality Results 🧑‍ on using the filters in the export data. I am able to get the filters extracted using the DQ monitoring project filter values but couldn't get the corresponding monitoring project results using the DQ Monitoring project check results. I am getting an empty file with just the column names in the file in the output file which I have checked the output into a CSV file just after the DQ monitoring project check results step. And I have enabled only one filter in the DQ monitoring project as I read in the community forum that two filter wouldn’t work with this setup.Are there any changes I need to make the way I am using the filter in DQ monitoring project check results step ? I am attaching the screenshots below for reference. 

321
C
Data Enthusiast
M
Data Voyager
manidharData Voyager
asked in Data Quality & Catalog ⚙️
Unable to find the SFTP upload file step in OneDesktop

Hi Ataccama Community, I want to upload a CSV file which is an export of DQ monitoring project results to SharePoint folder. I couldn’t see the Task SFTP upload file step in my OneDesktop (version 15.4). Is this step not available in the current version of Ataccama OneDesktop ? Or is there any work around to upload the CSV files to the Share Point server. CheersManidhar

152
A
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Catalog ⚙️
Connect with Collibra

Hi,I am attempting to connect to Collibra using the Collibra reader (refer to the page: Collibra Integration in Ataccama ONE Desktop 🏹).However, I am encountering the following error message when I execute the plan. Could someone please assist me?  I have set up a server without a username/password because I don't need to enter a username/password when accessing Collibra.  

722
A
K
Universe Traveller
kellymremzilybUniverse Traveller
asked in Data Quality & Catalog ⚙️
Nested SQL Catalog Items

Hello, Can we create nested sql catalog items.  Meaning, can I use a SQL catalog item in the query as a source to another sql catalog item?  

181
Albert de Ruiter
Rocket Pioneer L1
Albert de Ruiter
Rocket Pioneer L1
Albert de RuiterRocket Pioneer L1
asked in Data Quality & Catalog ⚙️
Creating array of values from multiple rows

Hi all,Suppose you have 2 tables that relate as 1:n. When joining the tables in a query without filtering, the result will be n rows. My requirement is to have just one row returned, with the values of the n-table shown as an array. In SQL Server you could use a query likeSELECT     p.PK_ONETEST,    p.Name,    p.Description,    (SELECT         c.Name     FROM ODS.NTEST c     WHERE c.FK_ONETEST = p.PK_ONETEST     FOR JSON PATH) AS ChildrenFROM ODS.ONETEST p;resulting in My question is if we can also achieve this in one or more steps in One Desktop.Kind regards,Albert

672
Albert de Ruiter
Rocket Pioneer L1
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Data Quality Results 🧑‍🔬
ONE Desktop: Data Quality Results 🧑‍🔬

Hi Community! Are you looking to retrieve Data Quality (DQ) results from monitoring projects or catalogs using Ataccama's ONE Desktop? We have created a set of easy-to-follow steps to help you achieve this efficiently. These steps will mimic the API calls created for the same purpose and return DQ results. Let’s get into it 🔮 PrerequisitesBefore diving into the process, make sure you have the following prerequisites in place: Active Server Connection: Ensure you are connected to the ONE Web Application. Input Data: You need the data for which you want to retrieve results. You can use either the appropriate Input step or the MMM Reader step. You can add input data by dragging and dropping files onto the canvas or by configuring the steps manually. If multiple input steps are required, use the Join step and create connections accordingly.   Note: Depending on the step you wish to use to retrieve results, different parameters will be required in the input. Refer to the individual step

6007
M
Data Voyager
M
Data Enthusiast
manikanta vData Enthusiast
asked in Data Quality & Catalog ⚙️
How to Implement Logic in Ataccama to Process One Record and Flag Another as Error for Duplicate Correlation ID and Version Number

Two records as same correation id and same version number one should process another one should error out how to implement this logic in ataccama 

24413
S
Universe Traveller
S
Universe Traveller
Susan24usUniverse Traveller
asked in Data Quality & Catalog ⚙️
DATA STORIES

Hello, i am using a free version of the Data Stories , I cant seems to find where to upload or import dataset as seen from the demo --- Data Stories Demo does this free version not support creating a story , Dashboard or visualization ? if its does, is there something i’m not doing right 

442
A
D
Data Voyager
Data_VaultData Voyager
asked in Data Quality & Catalog ⚙️
Nested JSON with DAT file extension processing in Ataccama ONE

Hello,We recently started getting nested JSON event files with DAT file extension. We are getting an ‘Unknow Error’, when we try to import it.Can you help me in understanding, how nested JSON with .DAT extension can be processed in Ataccama.Thank you.   

192
A
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Catalog ⚙️
Multiple workflows in one scheduler

Hi Can we have multiple workflows in single scheduler? I tried to add multiple workflows but I am getting error. is there any way to do this? Thanks in advance

141
A
Ataccamer
S
Universe Traveller
Susan24usUniverse Traveller
asked in Data Quality & Catalog ⚙️
Connections Update

We recently updated the Data Sources connection, and all Catalog items that were using the old connection are no longer working in the monitoring projects. When I expand the catalog items, an error message appears: "You are looking at a deleted item."However, if I create a new catalog item and add it as an attribute in the monitoring project, it works perfectly fine.My question is: How can I update the existing catalog items to use the new connection? Do I need to manually recreate all these catalog items with the new connection, or is there a way to automatically switch all catalog items from the deleted connection to the new one

252
A
M
Data Voyager
manidharData Voyager
asked in Data Quality & Catalog ⚙️
How to export a Summary of DQ Monitoring Project results?

Hello Ataccama Community,I am trying to export the summary statistics of the DQ monitoring project?   I wanted to have a csv file with the columns like Date of the test results execution, Data_Catalog_Name, Attribute Name, DQ rule name, Data Source, DQ dimension, Total Number of Records, Number of records passed, and Number of records failed.To begin with I have tried using DQ monitoring projects step in One Desktop and the DQ Monitoring project is failing if I enable this post processing plan. Not sure why it was failing after enabling the below post processing plan. Any help is much appreciated. Thanks in Advance

1032
A
N
Universe Traveller
NataliiaUniverse Traveller
asked in Data Quality & Catalog ⚙️
Boolean DQ

Hi guys, how did you check boolean date type in DQ rules? When I apply this rules  it  also make numbers, string, others value as correct.  

282
A
J
Data Voyager
jaygskData Voyager
asked in Data Quality & Catalog ⚙️
Creating Source in Ataccama One for Snowflake

I am creating new source connection from Ataccama ONE and I need help on how to connect Snowflake.From below screenshot I am trying to understand JDBC connection looks like.For e.g. I am using jdbc:snowflake://myorganization-myaccount.snowflakecomputing.com/, but it is not working.Any help is greatly appreciated.  

313
J
Data Voyager
A
Data Voyager
Ankur SinhaData Voyager
asked in Data Quality & Catalog ⚙️
inserting data to Bigquery table using Ataccama ONE Desktop

Hi,I am able tot connect Bigquery from Ataccama ONE Desktop application but getting below error while inserting data in table - com.ataccama.dqc.commons.sql.atc.WrappedSqlException: [Simba][BigQueryJDBCDriver](100032) Error executing query job. Message: Transaction control statements are supported only in scripts or sessionswhile executing INSERT INTO ……..  

240
A
Data Voyager
S
Data Pioneer
sumishaData Pioneer
asked in Data Quality & Catalog ⚙️
How to get terms tagged to attributes in Catalog in One desktop

How do I get Terms tagged to attributes to various catalog items in One Desktop?I need to get all catalog items which are tagged with a specific Glossary Term.

662
S
Data Pioneer
T
Data Voyager
ThanujaData Voyager
asked in Data Quality & Catalog ⚙️
One desktop trans comoponent issue

When I try to open a Trans component on Ataccama One Desktop, I receive the error 'Index -1078132736 out of bounds for length 10.' Do you have any ideas or suggestions to resolve this issue?

152
T
Data Voyager
T
Data Voyager
ThanujaData Voyager
asked in Data Quality & Catalog ⚙️
Audit log configuration in ataccama one desktop for both backend/front end

Hi everyone, I was wondering if anyone could share how the i_event_log table is configured, as well as its details for the front-end (UI)? Also, could anyone provide guidance on how to configure the back-end audit log and retrieve it using pgAdmin? I’d really appreciate any insights. Thank you!

430
T
Data Voyager
A
Data Voyager
AbbyData Voyager
asked in Data Quality & Catalog ⚙️
Flow of columns after using group Aggregator

Hello All,I am using group aggregator in our plan but I am not sure how to make all the columns flow from input till the end/output through group aggregator. 

1412
R
Data Voyager
S
Universe Traveller
shikoh.zUniverse Traveller
asked in Data Quality & Catalog ⚙️
Creating rule for productline

Hi,Could someone guide me on creating rules for the Productline field? I am following the lab exercise but am unable to find the option for selecting "is not from reference data."Any guidance would be appreciated.Thanks,Shikoh Zaidi   

272
DannyRyan
Ataccamer
S
Data Voyager
SoniyaNData Voyager
asked in Data Quality & Catalog ⚙️
Combining Excel file

I have 10 excel sheets, which need to be combined in one single excel sheet. can this be done in Ataccama?

251
Lisa Kovalskaia
Ataccamer

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings