Data Quality & Catalog ⚙️

Hi community! In this post, we will cover a fundamental practice of configuring the profiling step 👣If you have missed it you can check the first post on the introduction to data profiling here👇🏽 The Profiling StepWhen you create a profile using a plan, the Profiling step becomes a pivotal part of your data journey, connecting your data source to insights. Let's explore how to make the most of it.General CategoryBasic TabIn the Basic tab, you can set the step's name, output file name, location, and the default locale for generated files. Masks TabThe Masks tab is where you define and edit masks. Masks help reveal data structure without displaying the actual content. For instance, you can use "D" to represent a digit and "L" for a letter. Configure characters, symbols, repeated symbols, and thresholds for your masks here. Right-click on the row to add, delete, or edit the characters: Drill-through TabThe Drill-through tab enables drill-through functionality, allowing you to inspect i

Happy Monday everyone! This week we are delving deep into one of the most important steps in working with data - profiling 🚀 We will start with the basics and move on to more advanced practices so don’t forget to follow the forum by hitting the Subscribe button to get the latest article in your inbox 🔔 When embarking on a data project, understanding your data's current state is paramount. It's the compass that guides you through the winding path of data cleansing and transformation. This understanding also serves as a yardstick to measure the effectiveness of your cleansing rules. Enter the Data Quality Profile – a snapshot of your data's health. It unveils duplicates, dependencies, business rule evaluations, data patterns, and much more. In this post, we will cover how to create and configure a data quality profile, complete with additional analyses like masks, dependencies, and business rules. Step 1: Select Data to Profile Launch ONE Desktop. In the File Explorer view, pick one

Hi everyone! This week’s last best practice is also the final post on components! Check out yesterday’s post to catch on if you have missed the series so far:  Nested components are like puzzle pieces that fit together to create sophisticated data transformations within a single entity. In this guide, we'll explore how to work with nested components in Ataccama ONE Desktop, focusing on their creation, import, configuration, and use cases. Understanding Nested ComponentsImagine having a main (parent) component that uses multiple component files to achieve complex data transformations. Nested components enable you to do just that. This approach allows you to maintain the original data while applying intricate logic to your datasets. For instance, you can use predefined components provided by Ataccama ONE Desktop to cleanse, filter, or standardize specific data attributes, all while preserving the integrity of your original data. Configuring a Nested Component in ONE DesktopIn Ataccama ON

Happy Monday community! Today we are going to talk about Remote Plans. Let’s start with the basics, what are remote plan executions? The remote plan execution lets you run any ONE plan on the Ataccama ONE Platform. Plans are typically developed and tested on small data sets or on dedicated environments. This helps you validate the plans and assess how well they perform so that you can make the necessary adjustments before applying them on your actual data. Once the plan works as expected, you can run it remotely on production data in ONE.Jobs are first sent to the Data Processing Manager (DPM) module, which forwards the job information to a suitable Data Processing Engine (DPE). Actual data processing takes place only in DPE instances. How to configure Remote Plan ExecutionRemote plan execution leverages existing controls and concepts of ONE Desktop. Once you connect to the Ataccama ONE Platform, you can choose between the following launch options:Local: A suitable DPE is selected auto

Happy Wednesday Community! We are halfway into the week and ready with Part 2 of Component Steps. If you have missed Part 1 don’t worry, check it out on the link below 🔽 In the second part of our journey into Component Steps, we will delve deeper into their creation, configuration, parameter mapping, running from the command line, and working with nested components. Creating a ComponentBuilding a component step is akin to constructing a plan, but certain rules dictate its placement within a plan: At the Beginning: A component at the start of a plan should commence with an input step connected to actual data, not Integration Input. In the Middle: For components in the middle of a plan, they must have at least one input endpoint with Integration Input and one output endpoint with Integration Output. At the End: Components concluding a plan must start with an input endpoint with Integration Input and conclude with an output step that isn't Integration Output (ensuring data is written

Hi community, happy Tuesday! Today, we are covering one of the essential functions of ONE Desktop - Expressions & Steps ✨Steps and functions enable you to craft plan files tailored to your project's unique needs. Here, we provide a comprehensive overview of the frequently used steps and functions, guiding you towards best practices.Frequently Used Steps:Flow Control StepsData Parsing StepsAnalysis StepsMatch and Merge StepsUsing FunctionsWithin these steps, you can leverage various functions to further enhance your data processing capabilities. Here are some commonly used functions:matches: Performs a full match of input data with a regular expression, returning true or false. find: Conducts a partial match of a regular expression within the input string, returning true or false. substr: Extracts a substring from the input string, starting from zero.Using Regular ExpressionsAtaccama's products support regular expressions for powerful pattern matching. Here are some fundamental regu

Happy Monday Community! We are continuing our ONE Desktop best practices series this week as well! Today on the menu is - DQ Scoring!Data quality scoring is a vital process in Ataccama Desktop, enabling you to assess data quality based on specific rules and assign a numerical score. This score helps differentiate between high-quality and low-quality records, making it an essential step in data processing.Here are the best practices for data quality scoring in Ataccama Desktop:1. Categorize Scoring into Four LevelsTo effectively score data quality, categorize your cleansing scores into four quantification levels:Scoring Level Scoring Description Scoring Result 0 No modification was done to the input value. VALID < 10,000 Small modification was done to the input value. VALID/CORRECTED < 10,000,000 Huge modification was done to the input value. UNSAFE/UNKNOWN > 10,000,000 Input value or pre-cleansed value is null. NOT VALID Each cleansing operation, as de

Hi everyone!Today, we have a quick best practice on covering the basics of column naming and debugging in ONE Desktop plans.When working with ONE Desktop, maintaining a consistent and meaningful naming convention for columns in your output files is crucial. This practice ensures clarity, organization, and seamless data processing. In this guide, we'll provide recommendations for naming conventions and debugging your expressions and steps for efficient plan execution. Column Naming ConventionsOutput files in ONE Desktop can contain numerous columns, especially after processing input data in DQC. To keep your data well-organized, we recommend grouping columns by content. Below are the key prefixes and suffixes used in our naming convention: PrefixesAttribute Prefix Explanation src_xxx Source input values (read-only attribute) meta_xxx Source input metadata dec_xxx Decoded source input values dic_xxx Translated master value storage (src_xxx is the source) pur_xxx Pre-clea

Hi Community!We are back with part 2 of One Desktop Plans! If you have missed our first post on how to build plans you can find it by clicking on the link below 👇 When it comes to executing plans within ONE Desktop, understanding the various methods and choosing the right approach is essential for a seamless experience. In this guide, we'll explore the best practices for running plans effectively. Understanding Execution OptionsONE users have multiple ways to run a plan: Local Launch: This is done using Ataccama ONE runtime (formerly known as DQC) from the ONE Desktop interface. ONE Platform Launch: You can employ DPE or Spark DPE for remote execution via the ONE Platform. Command Line Execution: Utilize the runcif.sh|.bat script via command line when employing Ataccama ONE runtime. DPM CLI Tool: Employ the DPM CLI tool for remote execution using DPE or Spark DPE. This article primarily focuses on executing plans. For information on running plans within workflows, please refer t