Skip to main content

Data Quality & Catalog ⚙️

Discuss, learn, ask, suggest, share all things DQ&C 🔍

  • 791 Topics
  • 2,012 Replies
791 Topics
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Environments 🌍Best Practice

Hi everyone! I hope you are all excited to take one more step into ONE Desktop today 👣Today on the menu we have…..Environments!Imagine effortlessly crafting and testing your solutions across different operational landscapes like DEV, TEST, or PROD. That's what Environments are all about! These versatile tools offer a seamless platform for:Running plans on diverse databases without building separate plans. Testing plans, workflows, and web services across testing and production servers. Crafting and exporting runtime configurations for various environments (perfect for those snazzy folder shortcuts).How Environments Work: A Guided TourEnvironments are your game-changers. Here's a sneak peek at how they operate:Local Launch EnvironmentWhen you're running your plans or workflows on your local machine through ONE Desktop, this environment kicks in. It's your go-to for efficient local testing and development.ONE Platform Launch EnvironmentFor those times when your plans need to be executed

1170
A
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
#1 Adding Data SourcesGet certified with Ataccama

👋 Welcome to our new article series, where we will go through the individual topics that are part of the Ataccama Data Quality & Governance Certified Professional: Foundation Level Ataccama ONE v13 certification exam.We will focus on practicing and pointing out the most common mistakes. Our first topic is Adding Data Sources. 🔎 LET’S TEST YOUR SKILLSUsing ONE Desktop Application, create a database connection to your database. ​​Using the ONE Desktop Application, upload the 'bloodtypes.xlsx' into your database. Create the database table <your_name>_bloodtypes, and upload the contents of the 'bloodtypes.xlsx' file. 💡 How to do it?Open Ataccama ONE Desktop Application and create a new projectIn your newly created project, create a plan and name it loaddata.plan Add (drag and drop from the palette on the left side) steps Excel File Reader and Jdbc WriterUpload bloodtypes.xlsx (attached to this article) into your project Open Excel File Reader and fill in File Name, Sheet Name

3881
F
Data Pioneer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Tasks 📃✔️Best Practice

Happy Wednesday everyone!Today’s topic is Tasks! We will cover how to optimize your task management on Ataccama to effectively organize and track your work. Let's dive in! A Glimpse into Task ManagementAt the heart of the Ataccama ONE Platform💜 lies the Task management system. This tool brings together your tasks into a Kanban board, offering an intuitive visual representation of your workload and its progress. From prioritizing tasks to altering their status and engaging in collaborative discussions, this system is your gateway to productivity. Unlocking the Potential of TasksWithin the Task management system, you can accomplish a multitude of actions: Task Creation and Management: Create tasks and their subtasks, assign them to team members, and even solicit assistance from colleagues. Change Management: Request changes like entity creation or access adjustments seamlessly. Workflow Task Management: Efficiently handle tasks related to access requests, approval workflows, and mor

910
A
A
Data Voyager
anagha.kData Voyager
asked in Data Quality & Catalog ⚙️
Where do we get Monitoring project ID?

Hi Team,I want to use the DQ monitoring project results component. From where do we get the  Monitoring project Id for the component?Thanks in AdvanceAnagha 

985
anna.spakova
Ataccamer
B
Data Voyager
babamarData Voyager
asked in Data Quality & Catalog ⚙️
Snowflake key pair authentication

Hello,I’m setting a new connection for Snowflake datasource, and I need to use key pair authentication for this.In the credentials drop-down menu it doesn’t look like this option to be present:  What do you advise?

5761
A
Marcel-Jan
Star Blazer L2
Marcel-JanStar Blazer L2
posted in Data Quality & Catalog ⚙️
Ataccama source monitoring

At our site we have Ataccama on premise and several sources from different data systems. It happened a couple of times that profiling on one of the sources was no longer possible and that we found that out more than a week later. After which it was difficult to track back why this happened.We do have monitoring of all kinds of Ataccama components with Prometheus. But that didn't seem to detect these issues. I was looking for a way to monitor profiling functionality.So what I came up with was a monitoring project that profiles a small table from each source, every hour.Since you can schedule a monitoring project every hour at least, and since you can add notification in case of failure, this gives us warning at most near 1 hours after something breaks.Just adding a catalog item of a table to the MP turned out not to be sufficient though. Because when the monitoring project ran, it did only a structure check, not a profile. To get it profiling, it needed an anomaly check.When you do an a

682
Marcel-Jan
Star Blazer L2
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
Hiding the Global Settings or at least users for all viewers

We have a question. The basic viewer can view the global settings and with that also a list of users. We'd rather not share this info with the basic viewer. How can we change this?

989
anna.spakova
Ataccamer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Mastering Anomaly Detection in Your Data Catalog📔
Mastering Anomaly Detection in Your Data Catalog📔Best Practice

Hi Community!  Yesterday we took a step into Anomaly Detection, what is it, and how to use it. If you’ve missed that post, go check it out here:Today, we will have a short summary of what we covered yesterday with a focus on the Data Catalog.Spotting the Unseen: Anomaly Detection DemystifiedAnomaly Detection is like having a data detective, tirelessly examining your catalog items for any unusual activities. It operates on two levels:Catalog Item Level: Analyzing key metrics like the number of records and summary statistics. Attribute Level: Delving into value distribution, standard deviation, numeric sum, and more.Whenever a metric's value behaves abnormally, it raises a flag. But wait, there's more! Our AI-driven Anomaly Detection system, fondly known as "ONE," is a quick learner. With every confirmed or dismissed alert, it refines its understanding of your data's unique quirks. This means more precise anomaly detection with each profiling session.Customizing Anomaly Detection to Fit

2410
A
L
Data Enthusiast
lsmall028Data Enthusiast
asked in Data Quality & Catalog ⚙️
AQL for Purpose Tags

Easy question: I used to be able to use this AQL search to look up tables with specific purpose tags, but it’s no longer working with the upgrade to 14.3. Does anyone have a suggestion? Here is the AQL search I used: dataInstances.any(displayName = '[Purpose Tag]')

482
L
Data Enthusiast
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Mastering Anomaly Detection: Unveiling Data Inconsistencies 🔎

Hi everyone!A new week means new best practices! This week we will focus on Anomaly Detection to our new Data Dictionary so if you haven’t started to follow our forums - click on Subscribe now to not miss out on any news 🔔 Easiest way to follow us is by selecting all forums you are interested to follow from our homepage and hit subscribe. Now let’s dive in! Anomaly detection is a pivotal tool for ensuring data accuracy and quality by identifying irregularities and inconsistencies within datasets.Why Anomaly Detection MattersAnomaly detection is your safeguard against data discrepancies and plays a crucial role in maintaining data quality and relevance. With this service, you gain the following advantages:Swift Issue Resolution: Users are promptly notified of potential data discrepancies or pipeline variations, enabling timely problem resolution. Enhanced Data Quality Monitoring: Monitor your dataset's quality to ensure its alignment with actual data conditions. This, in turn, improves

1080
A
K
Data Pioneer L2
KundanData Pioneer L2
asked in Data Quality & Catalog ⚙️
JSON Parsing

How to extract the root of JSON result which I got from GET api call from JSON CALL step .?I am able to fetch the attributes of root but I am unable to fetch the root itself .

1493
A
Ataccamer
L
Data Voyager
LianeData Voyager
asked in Data Quality & Catalog ⚙️
Which steps do you take to add a DQ rule?

I’d like to check if I am listing the correct steps to add a DQ rule in an existing Monitoring Project, and the order in which to take them. I am quite new to Ataccama so please excuse my ignorance. Can anyone please confirm or adjust them? Many thanks in advance! How to add a DQ ruleCheck if the rule you want to apply, already exists out of the box. For example, if you want to say a certain field cannot be empty, you can apply the predefined rule: String completeness. Other standard checks include validation of email addresses, and URLs. Go to Monitoring Project, tab Configuration & Results Choose the file you want to check (either a VCI or a Parquet File – the latter starts with PF). Choose which attribute you want to check In the 5th column, add: applied DQ checks Doublecheck whether the scoring of the rule is correct (some of the standard rules have a scoring of 0 – you might want to change that) In the upper right hand corner, choose “Publish” Run monitoring. This will take so

1052
L
Data Voyager
A
Data Voyager
anagha.kData Voyager
posted in Data Quality & Catalog ⚙️
DQ Timeliness rule

How to implement Timeliness rule in web application 

932
A
Data Voyager
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Monitoring Project - Results Retention 🦾Best Practice

Hi everyone! This week’s last best practice post is on Monitoring Project Results Retention! Managing data retention is crucial for maintenance and security purposes. Whether you want to keep results for a specific duration or limit data storage based on the number of runs, Ataccama's Monitoring Project Results Retention feature has got you covered. Flexible Retention SettingsWith Monitoring Project Results Retention, you have the flexibility to choose how long you want to store the results of your monitoring projects. This can be defined in two ways: Per project configuration: Keep the last X runs. Remove runs after X days. Global configuration (available to power-users only) that takes precedence over project configuration: Keep the last X runs. Remove runs after X days. When global settings are configured, you'll receive notifications in the web application.For more details on configuring global settings, check out Monitoring Projects Global Retention.When Data Will Be DeletedMo

1110
A
A
Data Voyager
anagha.kData Voyager
asked in Data Quality & Catalog ⚙️
DQ Accuracy Rule for Date datatype

Hi Team,I have tried to create Accuracy rule for Date datatype to check whether Date is in the standard format. Accurate rule should be as follows:Standard date format is ‘MM-DD-YYYY’. If date is entered in any format other than the above mentioned (‘DD-MM-YYYY’ or ‘YYYY-MM-DD’ or ‘YYYY-DD-MM’) then it should be considered as inaccurate.When I tried using regex expression, I am getting error. Any ideas?

4021
A
Ataccamer
T
Data Voyager
ThangarajData Voyager
asked in Data Quality & Catalog ⚙️
Thin Client - Catalogue item based access Request

Is there any way that we restrict accessing the data that are imported to Ataccama Thin client ?, as we do role based access , we are able to see all others catalogue items imported into Thin Client.Such catalogue item view breach the data classification compliance which is managed in main data factory where we store the data.Anyone suggest here your thoughts to how restrict the data by giving right access / restriction while we importing the catalogue item or any measure we can take to restrict in viewing the catalogue items within Thin client w.r.t catalogue item import owner?  

451
Lisa Kovalskaia
Ataccamer
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
How can we implement concatenation of the last 10 rows in a column?

How can we implement concatenation of the last 10 rows in a column?I am looking for a way of storing previous values in a variable and adding the current row value to the variable like that.

341
A
R
Data Enthusiast
rupakdeData Enthusiast
asked in Data Quality & Catalog ⚙️
Pass parameter as filter condition in JDBC reader sql

As per requirement I want to read a marker value from a file and pass that value as parameter in select SQL of a JDBC reader to pull only delta records. Is this possible in 13.9.1 version? Please advise with documentation link if available.

1581
A
Ataccamer
B
Data Voyager
BhumiData Voyager
asked in Data Quality & Catalog ⚙️
How to implement a DQ rule to check if an attribute's value is present in another attribute (for eg. Attribute1 'Product' value is present in Attribute2 'Quantity' field) using Thin client

How to implement a DQ rule to check if an attribute's value is present in another attribute (for eg. Attribute1 'Product' value is present in Attribute2 'Quantity' field) using Thin client

351
B
Data Voyager
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Import Monitoring Project Configurations Part 3️⃣Best Practice

Hi everyone! In the last part of Monitoring Projects Configurations, we will talk about how to easily import configurations from one monitoring project to another, making your data quality monitoring more efficient. If you’ve missed Part 1&2 you can find them below:Import the Configuration Instead of creating a new project from scratch, you can copy the configuration of one monitoring project and apply it to another. This is particularly useful when you want to replicate settings from a testing environment to a production environment. The copied configuration includes all checks, invalid samples, reports structure, and DQ filters.To import the configuration:Navigate to the project where you want to apply the configuration.   Expand the more options menu and select Import configuration. Choose the monitoring project from which you want to copy the configuration, from the list provided. Check if the preselected catalog items are correct. You can modify the list by deleting items or

871
ivan.kozlov
Ataccamer
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
Pivot every 13 rows into one row in target.

The source csv file has splitted into 13 rows for each actual row in target.By using Ataccama one desktop we would like to pivot the data by sending source 13 rows into one row in target. I am trying to implement the logic below in the Text file reader step but it's not working . v_RowNum =  iif(v_RowNum IS  NULL,0,v_RowNum+1)                    -------- Initialize v_RowNum to 0 if NULL; otherwise, increment it by 1 for each row.v_CurrentNumber =  iif(v_CurrentNumber%13= 0, v_CurrentNumber+1,  v_CurrentNumber  ).    -- Increment v_CurrentNumber by 1 for every 10 rows.GENERATED_NUMBER= v_CurrentNumber GENERATED_NUMBER INTEGER   Please  help us  to make it work in Ataccama one desktop .

772
M
Data Pioneer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Monitoring Project Configuration Part 2️⃣Best Practice

Hi everyone! In this continuation of our best practices post on Monitoring Project Configuration, we will explore additional settings and actions to ensure seamless data quality monitoring.Check out the part one of the series if you’ve missed it:Disable Rule SuggestionsPower users can disable rule suggestions in monitoring projects. For more details, refer to the link provided in the text. Set Partition DefaultsFor partitioned tables in monitoring projects, identify them with the provided icon. Define default analysis for partitioned tables using the more options icon and selecting either Last partition analysis as default or Full analysis as default. These settings will be applied when running the monitoring project. See the link provided for more information. Anomaly DetectionWhen running anomaly detection during monitoring projects, choose between the time-dependent model or time-independent model. Configure the sensitivity of the detection as well. For further details, check out th

1240
A
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
Pivoting all rows into columns in target.

My input file is in the below structure and looking to pivot all rows into columns. No name 1 value1 2 value2 3 value3 4 value4 5 value5    Expecting output should be like  Col1 Col2 Col3 Col4 Col5 value1 value2 value3 value4 value5   What will be the best approach in Ataccama One desktop? Looking for support and help..

2074
A
Ataccamer
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
I have to write below Multi array list in to a file through JSON call.How can I perform this?

Json : [   {      "Headers":[         "Empid",         "Name",         "dept",         "EMAIL"      ],      "ID":"Table_112",      "Rows":[         [            "100",            "John",            "Finance",            "John@gmail.com"         ],         [            "101",            "Smith",            "HR",            "Smith@gmail.com"         ],         [            "102",            "Derek",            "Marketing",            "Derek@gmail.com"         ]      ]   }] Expecting the output in below format:Empid Name dept EMAIL ID 100 John Finance John@gmail.com Table_112 101 Smith HR Smith@gmail.com   102 Derek Marketing Derek@gmail.com      I have created a plan with below details:After running a plan with JSON CALL step below response file got created.   Based on response file i have given below columns in Data Streams    I am not getting the out put what i expected. Appreciate your help and support . 

2441
Lisa Kovalskaia
Ataccamer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Getting Started with Monitoring Projects 🎢Best Practice

Hi Community!In the next few days, we will discuss Monitoring Projects, a powerful tool to evaluate and monitor data quality over time for selected catalog items. Monitoring projects use DQ checks, automatic anomaly detection, and structure checks to maintain data quality. We also have a short video introduction to Monitoring Projects, go check it out here 👇 If you are ready, let's dive in 🤿 Accessing Monitoring ProjectsTo access Monitoring Projects:Go to the left-hand menu and select Data Quality. Choose Monitoring Projects.     In this section, you'll find all configured projects. If you want to start a new project, click Create.Each project listed will show the following information:Name: The project's name. Terms: The glossary terms associated with the project. Items: The number of items monitored within the project. Latest run: The date of the project's last evaluation.   The current state of a project is indicated through row highlighting, showing if it's published, recently ad

1840
A

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings