Skip to main content

Data Quality & Catalog ⚙️

Discuss, learn, ask, suggest, share all things DQ&C 🔍

  • 791 Topics
  • 2,012 Replies
791 Topics
S
Data Pioneer
sgillaData Pioneer
asked in Data Quality & Catalog ⚙️
DQS - camel case

Is there a function or custom ways to convert strings into camel case on DQS - Data Quality services?

374
A
Ataccamer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Monitoring Project Configuration Part 1️⃣Best Practice

Happy Tuesday Community! We started this week by sharing best practices on Monitoring Projects, and today we’ll be continuing with how to configure Monitoring Projects. If you have missed the first post you can find it here:Let's explore the best practices for configuring Monitoring Projects to ensure robust data quality checks and anomaly detection.Select Catalog ItemsGo to the Configuration & Results tab. To add catalog items: For a new project, select Add catalog items.   To add more items to an existing project, select +Add under Items to Monitor.   To remove an item, use the more options icon and select Delete. The listed catalog items will display a summary of: Structure: Aggregated results for structure checks on attributes. Anomalies: Detected anomalies on attributes. DQ Checks: Results of data quality checks assigned to attributes. Overall: Aggregated quality based on assigned DQ checks contributing to overall quality. Individual DQ dimensions, listed with the

2151
R
Universe Traveller
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Remote Plan Execution 🦾
ONE Desktop: Remote Plan Execution 🦾

Happy Monday community! Today we are going to talk about Remote Plans. Let’s start with the basics, what are remote plan executions? The remote plan execution lets you run any ONE plan on the Ataccama ONE Platform. Plans are typically developed and tested on small data sets or on dedicated environments. This helps you validate the plans and assess how well they perform so that you can make the necessary adjustments before applying them on your actual data. Once the plan works as expected, you can run it remotely on production data in ONE.Jobs are first sent to the Data Processing Manager (DPM) module, which forwards the job information to a suitable Data Processing Engine (DPE). Actual data processing takes place only in DPE instances. How to configure Remote Plan ExecutionRemote plan execution leverages existing controls and concepts of ONE Desktop. Once you connect to the Ataccama ONE Platform, you can choose between the following launch options:Local: A suitable DPE is selected auto

1280
A
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
How do we connect Workday objects  from Onedesktop by using CData drivers

We successfully connected Workday from Atacccama one web by using CData drivers.Now we are trying to do the same from One Desktop.How do we connect Workday objects  from Onedesktop by using CData drivers?We were able to connect Workday  through  the API's  but our requirement demands CDATA drivers connectivity.Thanks,Mahesh Arigela

351
A
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Component Steps Part 2️⃣
ONE Desktop: Component Steps Part 2️⃣Best Practice

Happy Wednesday Community! We are halfway into the week and ready with Part 2 of Component Steps. If you have missed Part 1 don’t worry, check it out on the link below 🔽 In the second part of our journey into Component Steps, we will delve deeper into their creation, configuration, parameter mapping, running from the command line, and working with nested components. Creating a ComponentBuilding a component step is akin to constructing a plan, but certain rules dictate its placement within a plan: At the Beginning: A component at the start of a plan should commence with an input step connected to actual data, not Integration Input. In the Middle: For components in the middle of a plan, they must have at least one input endpoint with Integration Input and one output endpoint with Integration Output. At the End: Components concluding a plan must start with an input endpoint with Integration Input and conclude with an output step that isn't Integration Output (ensuring data is written

2240
A
S
Data Enthusiast
SaranyaTData Enthusiast
asked in Data Quality & Catalog ⚙️
Time series analysis in V13.9

Any one who has worked on Time series analysis option in V13.9 and successfully got the required results

392
S
Data Enthusiast
C
Data Voyager
CharishmaData Voyager
asked in Data Quality & Catalog ⚙️
Json writer

Hi ,I want to know what was the input to be given in root object for JSON writer  

531
A
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Frequently Used Steps and Expressions 🪜
ONE Desktop: Frequently Used Steps and Expressions 🪜

Hi community, happy Tuesday! Today, we are covering one of the essential functions of ONE Desktop - Expressions & Steps ✨Steps and functions enable you to craft plan files tailored to your project's unique needs. Here, we provide a comprehensive overview of the frequently used steps and functions, guiding you towards best practices.Frequently Used Steps:Flow Control StepsData Parsing StepsAnalysis StepsMatch and Merge StepsUsing FunctionsWithin these steps, you can leverage various functions to further enhance your data processing capabilities. Here are some commonly used functions:matches: Performs a full match of input data with a regular expression, returning true or false. find: Conducts a partial match of a regular expression within the input string, returning true or false. substr: Extracts a substring from the input string, starting from zero.Using Regular ExpressionsAtaccama's products support regular expressions for powerful pattern matching. Here are some fundamental regu

3340
A
Maxim Kim
Ataccamer
Maxim KimAtaccamer
posted in Data Quality & Catalog ⚙️
Export Data Quality results

Eventually, you might need all your data quality results evaluated in One monitoring projects or within regular catalog items (via business terms and related DQ rules) exported to some reporting system or into excel. And it would be nice if it looked like this:  TLDR; There are 2 components attached you can use to export DQ results, one for monitoring projects, another one for regular catalog items. Monitoring ProjectsThe idea is to read all monitoring projects metadata, join it with their check results as well as rules names, catalog item names and voilà, we can save it to excel/csv or to some database table.It sounds like something very simple, and indeed there is no black magic, but nonetheless component looks a bit more elaborated: Catalog Items DQ (based on Term evaluations)The idea is to find all catalog items that have aggregated overall data quality and join them with the DQ rules results:This component has a bit simpler implementation:  Of course, the output of the components

7266
M
Star Blazer L2
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Data Quality Scoring 🧮
ONE Desktop: Data Quality Scoring 🧮

Happy Monday Community! We are continuing our ONE Desktop best practices series this week as well! Today on the menu is - DQ Scoring!Data quality scoring is a vital process in Ataccama Desktop, enabling you to assess data quality based on specific rules and assign a numerical score. This score helps differentiate between high-quality and low-quality records, making it an essential step in data processing.Here are the best practices for data quality scoring in Ataccama Desktop:1. Categorize Scoring into Four LevelsTo effectively score data quality, categorize your cleansing scores into four quantification levels:Scoring Level Scoring Description Scoring Result 0 No modification was done to the input value. VALID < 10,000 Small modification was done to the input value. VALID/CORRECTED < 10,000,000 Huge modification was done to the input value. UNSAFE/UNKNOWN > 10,000,000 Input value or pre-cleansed value is null. NOT VALID Each cleansing operation, as de

2190
A
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop Plans: Column Naming & Debugging 🪲
ONE Desktop Plans: Column Naming & Debugging 🪲

Hi everyone!Today, we have a quick best practice on covering the basics of column naming and debugging in ONE Desktop plans.When working with ONE Desktop, maintaining a consistent and meaningful naming convention for columns in your output files is crucial. This practice ensures clarity, organization, and seamless data processing. In this guide, we'll provide recommendations for naming conventions and debugging your expressions and steps for efficient plan execution. Column Naming ConventionsOutput files in ONE Desktop can contain numerous columns, especially after processing input data in DQC. To keep your data well-organized, we recommend grouping columns by content. Below are the key prefixes and suffixes used in our naming convention: PrefixesAttribute Prefix Explanation src_xxx Source input values (read-only attribute) meta_xxx Source input metadata dec_xxx Decoded source input values dic_xxx Translated master value storage (src_xxx is the source) pur_xxx Pre-clea

2070
A
Albert de Ruiter
Rocket Pioneer L1
Albert de RuiterRocket Pioneer L1
asked in Data Quality & Catalog ⚙️
Hiding empty metadata fields in business glossary

Hi,I am wondering if the following is possible. On the glossary's term screen not all properties/metadata will always be filled. For instance in most occasions there will not be an abbreviation for a term.Is there a way, on the view screen only, to hide this abbreviation field when it contains no value?Thanks for your time!Kind regards,Albert de Ruiter

674
Albert de Ruiter
Rocket Pioneer L1
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Running Plans 🏃
ONE Desktop: Running Plans 🏃

Hi Community!We are back with part 2 of One Desktop Plans! If you have missed our first post on how to build plans you can find it by clicking on the link below 👇 When it comes to executing plans within ONE Desktop, understanding the various methods and choosing the right approach is essential for a seamless experience. In this guide, we'll explore the best practices for running plans effectively. Understanding Execution OptionsONE users have multiple ways to run a plan: Local Launch: This is done using Ataccama ONE runtime (formerly known as DQC) from the ONE Desktop interface. ONE Platform Launch: You can employ DPE or Spark DPE for remote execution via the ONE Platform. Command Line Execution: Utilize the runcif.sh|.bat script via command line when employing Ataccama ONE runtime. DPM CLI Tool: Employ the DPM CLI tool for remote execution using DPE or Spark DPE. This article primarily focuses on executing plans. For information on running plans within workflows, please refer t

2200
A
I
Data Voyager
Ilia.StepanichevData Voyager
asked in Data Quality & Catalog ⚙️
Simple maker-checker process (approval workflow)

Hi team!I'm trying to implement a simple maker-checker process for Business Glossary Management: there are different Data Domains, each with its own responsible: Data Steward and Data Owner. So, for each Data Domain, the corresponding Data Steward creates the term --> the corresponding Data Owner approves it. Therefore, it's essential to consider the separation of permissions by data domains.Please give me some ideas on how I can implement it in the current product version. As far as I could investigate, grouping terms by different folders/categories in the Glossary doesn't help since I can't assign any roles to a particular folder/category: I can't assign Steward1 and Owner1 to the BusinessTerms group and Steward2 and Owner2 to the SecurityTerms, for example.How should it be implemented, then? Many thanks for considering my request!

2012
M
Star Blazer L2
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
Term suggestions overwhelming - How to tweak and make it stop.

Hi,We've loaded 2000+ terms and assigned those with desktop as a migration from a data dictionary. When we now have new catalog items, Ataccama suggests 20+ terms for a CI Attribute. This is not usable.How we tweak it to:Limit to only top 5 Stop when a term is already assigned.And here's an idea: If the column header is the same or can be matched using a simple fuzzy match, that is a better indicator than most AI.Another: Please skip some of the datatypes, a boolean … probably not a good idea.Any idea's?We are also wondering if we can turn it off.Thank you in advance!Marnix

1278
M
Star Blazer L2
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Building Plans 🏗️
ONE Desktop: Building Plans 🏗️Best Practice

Hi Community! This week is all about ONE Desktop Plans 🏗️We will start with how to build plans on ONE Desktop and work our way to DQ Scoring.Next week, we will continue to cover different features of plans and ONE Desktop so don’t forget to subscribe to our forums to receive a daily notification when a new article is published ✨Let’s kick it off! Understanding Plan EditorThe Plan Editor is your workspace for crafting plans. It comprises two core components:Canvas (1): This is where you define the logic of your plan by connecting different steps together. Palette (2): Here, you'll find a wide array of steps and actions that can be added to your plan. Creating a New Plan FileStarting a new plan file is a breeze:Right-click on a project or folder in the explorer panel.   Select New > Plan. Alternatively, use the toolbar and select Plan.   Specify a Name for your plan and designate its storage location (Container).   Adding Steps to the CanvasBringing functionality to your plan invol

1030
A
A
Data Voyager
anagha.kData Voyager
asked in Data Quality & Catalog ⚙️
Need only Valid rule names in Monitoring Project Report Export

Hi I want only the valid rule names in valid_rules column for Monitoring project export. How to implement that? 

621
P
Space Explorer
C
Data Pioneer
C_RData Pioneer
asked in Data Quality & Catalog ⚙️
I'm trying to retrieve only 10 or so rows from source - SQL Catalog Item

Hi, I have created a SQL Catalog item in UI however I'm trying to retrieve only 10 or so rows from the database. I am using the SELECT TOP/LIMIT clause to choose how many rows I want to retrieve both queries are having errors, I'm not sure what the problem is, does anyone have any ideas? Thank you. SELECT TOP 10 FROM x  SELECT * FROM x LIMIT 10    

601
Albert de Ruiter
Rocket Pioneer L1
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Overwrite the data using Jdbc Writer

Hello,everyone. When you load data into the database using Jdbc Writer and execute the plan every time, it keeps adding the same data from above. How can I overwrite the data? For instance, I have a table like this.I saved it in the database, made some changes to the table, and when I re-executed the plan, it added the changes on top. Duplicates appeared, to put it bluntly.I will be very grateful for your help Thank you!

874
F
Data Pioneer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Projects Part 2️⃣
ONE Desktop: Projects Part 2️⃣Best Practice

Hi community! We’re happy to share the second part of our ONE Desktop Projects guide. In this section, we'll delve into essential techniques for project creation, workspace management, and resource configuration. If you have missed the first part you can find it here: Creating a New ProjectBefore you embark on data transformations and analyses, a structured project is your starting point. A project is a collection of plans, components, and data files, organized logically into folders. To initiate a new project:Navigate to the File Explorer. Click on New > Project.   In the ensuing dialog, specify the project's Name and type. Regular projects encompass various categories: Empty project: A blank slate where you define your structure. Project from template: Predefined layout, useful for component projects. Tutorials project: Generates predefined tutorial projects.   Click Finish, and your project will appear in the File Explorer. See the figure below for a template project based

1820
A
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Monitoring Projects - Lifecycles 🔁Best Practice

Hi community!We are almost halfway into our Monitoring Project Best Practice series, today we are going to talk about Lifecycles 🎢Let’s dive in! Introduction to Monitoring Project LifecyclesThe introduction of Monitoring Project Lifecycles allows you to simulate a multi-environment process (Configure > Test > Send for approval > Make public, and repeat) within a single physical environment. While you can create two physical environments, the non-production environment should primarily be used for specific purposes, such as testing new sources, integration plans, model changes, and service re-configurations. It's important to note that non-production environments are not intended for developing data quality monitoring projects and migrating between environments. For the best results, we recommend having one production environment for all metadata and a non-production environment for model changes, API work, and testing. This setup differs from other modules like RDM and MDM, w

1142
A
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
Working with Time Series Data 📈Best Practice

Hello Community! As part of our ongoing Data Quality Best Practice series, today we'll dive into the world of time series data analysis.Time series data analysis is a powerful tool that enables you to unlock valuable insights from your transaction data by identifying trends and detecting anomalies. To make the most of this feature, here are some best practices to follow: Understanding Transaction Data and Time Series AnalysisTransaction data encompasses internal and external events or transactions within a business, often related to financial and logistical records. Common examples include sales orders, invoices, purchase orders, shipping documents, payments, and insurance claims.The benefits of time series analysis include:Visualizing time series data. AI-driven anomaly detection.   Keep in mind that time series analysis supports columns with timestamp information, such as DATETIME, DATE, or DAY data types. Configure and Run Time Series AnalysisTo get started with time series analysis

1362
A
B
Data Voyager
BhumiData Voyager
asked in Data Quality & Catalog ⚙️
Can ‘SQL Select’ stage only be used to connect with SQL Server DB

Can ‘SQL Select’ stage only be used to connect with SQL Server DB ?

823
anna.spakova
Ataccamer
A
Ataccama Community AdminIntergalactic Expert
published in Data Quality & Catalog ⚙️
ONE Desktop: Projects Part 1️⃣Best Practice

Happy Wednesday community! The Power of Projects: A PrimerBefore you dive into the exciting world of data transformations and analysis, let's lay down the foundation: projects. Think of projects as your organized toolkit – a collection of plans, components, data files, workflows, and custom scripts, all neatly arranged within logical folders. These projects live within a workspace folder, ensuring everything is in its right place. Structuring ProjectsImagine the harmony of a well-structured project. Here's how it can look:A Glance at a Well-Structured Projectbat: Home to batch or shell scripts for triggering plans. bin: Houses DQC plans for data cleansing and deduplication. components: Stores the goldmine of component plans for main plans. data: Nurtures your data directories. ext: Contains external lookup files and replacement lists. bat: Serves batch or shell scripts to trigger plans for lookup files. build: Holds DQC plans for crafting lookups. lkp: The indexed version of looku

930
A
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Сhanging stewards for all business terms

Hi everyone!How can we avoid manually changing stewards for all business terms and catalog items in Ataccama One? I will be very grateful for your helpThank you!

621
Albert de Ruiter
Rocket Pioneer L1

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings