Data Quality & Catalog ⚙️

Hi everyone! This week’s last best practice post is on Monitoring Project Results Retention! Managing data retention is crucial for maintenance and security purposes. Whether you want to keep results for a specific duration or limit data storage based on the number of runs, Ataccama's Monitoring Project Results Retention feature has got you covered. Flexible Retention SettingsWith Monitoring Project Results Retention, you have the flexibility to choose how long you want to store the results of your monitoring projects. This can be defined in two ways: Per project configuration: Keep the last X runs. Remove runs after X days. Global configuration (available to power-users only) that takes precedence over project configuration: Keep the last X runs. Remove runs after X days. When global settings are configured, you'll receive notifications in the web application.For more details on configuring global settings, check out Monitoring Projects Global Retention.When Data Will Be DeletedMo

Hi Team,I have tried to create Accuracy rule for Date datatype to check whether Date is in the standard format. Accurate rule should be as follows:Standard date format is ‘MM-DD-YYYY’. If date is entered in any format other than the above mentioned (‘DD-MM-YYYY’ or ‘YYYY-MM-DD’ or ‘YYYY-DD-MM’) then it should be considered as inaccurate.When I tried using regex expression, I am getting error. Any ideas?

Is there any way that we restrict accessing the data that are imported to Ataccama Thin client ?, as we do role based access , we are able to see all others catalogue items imported into Thin Client.Such catalogue item view breach the data classification compliance which is managed in main data factory where we store the data.Anyone suggest here your thoughts to how restrict the data by giving right access / restriction while we importing the catalogue item or any measure we can take to restrict in viewing the catalogue items within Thin client w.r.t catalogue item import owner?  

How can we implement concatenation of the last 10 rows in a column?I am looking for a way of storing previous values in a variable and adding the current row value to the variable like that.

As per requirement I want to read a marker value from a file and pass that value as parameter in select SQL of a JDBC reader to pull only delta records. Is this possible in 13.9.1 version? Please advise with documentation link if available.

How to implement a DQ rule to check if an attribute's value is present in another attribute (for eg. Attribute1 'Product' value is present in Attribute2 'Quantity' field) using Thin client

Hi everyone! In the last part of Monitoring Projects Configurations, we will talk about how to easily import configurations from one monitoring project to another, making your data quality monitoring more efficient. If you’ve missed Part 1&2 you can find them below:Import the Configuration Instead of creating a new project from scratch, you can copy the configuration of one monitoring project and apply it to another. This is particularly useful when you want to replicate settings from a testing environment to a production environment. The copied configuration includes all checks, invalid samples, reports structure, and DQ filters.To import the configuration:Navigate to the project where you want to apply the configuration.   Expand the more options menu and select Import configuration. Choose the monitoring project from which you want to copy the configuration, from the list provided. Check if the preselected catalog items are correct. You can modify the list by deleting items or

The source csv file has splitted into 13 rows for each actual row in target.By using Ataccama one desktop we would like to pivot the data by sending source 13 rows into one row in target. I am trying to implement the logic below in the Text file reader step but it's not working . v_RowNum =  iif(v_RowNum IS  NULL,0,v_RowNum+1)                    -------- Initialize v_RowNum to 0 if NULL; otherwise, increment it by 1 for each row.v_CurrentNumber =  iif(v_CurrentNumber%13= 0, v_CurrentNumber+1,  v_CurrentNumber  ).    -- Increment v_CurrentNumber by 1 for every 10 rows.GENERATED_NUMBER= v_CurrentNumber GENERATED_NUMBER INTEGER   Please  help us  to make it work in Ataccama one desktop .

Hi everyone! In this continuation of our best practices post on Monitoring Project Configuration, we will explore additional settings and actions to ensure seamless data quality monitoring.Check out the part one of the series if you’ve missed it:Disable Rule SuggestionsPower users can disable rule suggestions in monitoring projects. For more details, refer to the link provided in the text. Set Partition DefaultsFor partitioned tables in monitoring projects, identify them with the provided icon. Define default analysis for partitioned tables using the more options icon and selecting either Last partition analysis as default or Full analysis as default. These settings will be applied when running the monitoring project. See the link provided for more information. Anomaly DetectionWhen running anomaly detection during monitoring projects, choose between the time-dependent model or time-independent model. Configure the sensitivity of the detection as well. For further details, check out th

My input file is in the below structure and looking to pivot all rows into columns. No name 1 value1 2 value2 3 value3 4 value4 5 value5    Expecting output should be like  Col1 Col2 Col3 Col4 Col5 value1 value2 value3 value4 value5   What will be the best approach in Ataccama One desktop? Looking for support and help..

Json : [   {      "Headers":[         "Empid",         "Name",         "dept",         "EMAIL"      ],      "ID":"Table_112",      "Rows":[         [            "100",            "John",            "Finance",            "John@gmail.com"         ],         [            "101",            "Smith",            "HR",            "Smith@gmail.com"         ],         [            "102",            "Derek",            "Marketing",            "Derek@gmail.com"         ]      ]   }] Expecting the output in below format:Empid Name dept EMAIL ID 100 John Finance John@gmail.com Table_112 101 Smith HR Smith@gmail.com   102 Derek Marketing Derek@gmail.com      I have created a plan with below details:After running a plan with JSON CALL step below response file got created.   Based on response file i have given below columns in Data Streams    I am not getting the out put what i expected. Appreciate your help and support . 

Hi Community!In the next few days, we will discuss Monitoring Projects, a powerful tool to evaluate and monitor data quality over time for selected catalog items. Monitoring projects use DQ checks, automatic anomaly detection, and structure checks to maintain data quality. We also have a short video introduction to Monitoring Projects, go check it out here 👇 If you are ready, let's dive in 🤿 Accessing Monitoring ProjectsTo access Monitoring Projects:Go to the left-hand menu and select Data Quality. Choose Monitoring Projects.     In this section, you'll find all configured projects. If you want to start a new project, click Create.Each project listed will show the following information:Name: The project's name. Terms: The glossary terms associated with the project. Items: The number of items monitored within the project. Latest run: The date of the project's last evaluation.   The current state of a project is indicated through row highlighting, showing if it's published, recently ad

I would like as an example to have one ‘MS Teams’ channel for one DQ Monitoring to reach out different groups of people tus set up diffirent Teams connections  Is there any option to set it up in the current version? cheersPiotr 

Hello Community! In this second part of the lookup best practices series, we'll cover how to schedule automatic rebuilds of Lookup Items and provide some useful tips for managing your Lookup Files effectively. If you’ve missed the first post you can check it out here: Scheduling Lookup Item RebuildsAutomating the rebuild process of your Lookup Items can save time and effort. To schedule automatic rebuilds, follow these steps:In the Lookup Item, select the more options icon, and then click Schedule.   In the sidebar, choose Add scheduled event.   Set up the scheduled events for the rebuilding of lookups. For more details, refer to the Scheduler documentation.Managing Lookup FilesEditing Lookup FilesOnce Lookup Items have been uploaded, you can edit them manually or reupload the files if necessary. To edit the lookup manually: Select the required item. Click the more options icon, then choose Edit.   To reupload the lookup item: Select the required item. Click Upload new version

Hi guys, I'm doing the dq foundation course and I'm stuck.I have tried to look it up, but no result.This is on an environment in the portal.When I'm doing, with AVG or avg, this:It says:How do I make this work? Kind regards,Marcel

I have a legacy database without primary keys.  I know how to evaluate a single column for uniqueness, and thus a primary key. How do I evaluate multiple columns which together would be unique and thus be a composite primary key?

HiI’m running your free version of DQ Analyzer, but when I want to run my profiles this can’t be done.   As I understand it, it is due to a needed version update.Can you advice how I get my license key updated and not losing my existing profiles and still running the free version?I do not want to install Ataccama One.

Hi Community!We are starting the week by exploring best practices for effectively working with Lookup Items and integrating them into your Data Quality (DQ) rules. Lookups are essential in ONE Web Application as they provide a list of predefined values for attributes, enhancing data validation and detection capabilities.If you remember we have previously covered how to create Lookup files in rules in our Working with DQ Rules series, today we will expand on the lookup items feature. Overview of Lookup ItemsLookup Files in ONE Web Application enable you to use reference data, offering a catalog of predefined values for an attribute. You can utilize Lookup Files in both DQ Evaluation rules and Detection rules. To find Lookup Files, go to Data Quality → Lookup Items.  Uploading and Creating Lookup FilesCreating Lookup FilesTo create a Lookup File, follow these steps:In Knowledge Catalog, navigate to Data Quality. Select Lookup Items and click Create. Provide the required details, such as

Hello everyone, Anna here! I work as a projects consultant in Ataccama and today I put together a guide for you on how to migrate permissions after upgrade of v13 to v14. In v14, we introduced a fresh new permissions model introducing Groups and Stewardship. The downside is, though, that after the upgrade the old permissions model cannot be automatically migrated into the new setup.Attached to this guide is a migration package – a set of plans that you can use that will help you through the process of migrating the old permissions into the new setup in more automated way. It is not meant for fresh v14 installations.The article is split to two parts:Preparation phase – steps you need to do BEFORE you upgrade your environment to v14. Post-upgrade phase – steps you need to do AFTER the upgrade, meaning in your new v14 environment.Preparation phaseFollowing steps should be run on version 13 prior running the migration scripts and upgrade of the environment.1. Create server connection in ON

Happy Wednesday Community!Today’s post will cover how to work with component rules on ONE Desktop Application. If you missed yesterday’s article we had an introduction to working with components on ONE Web. Check it out here: Component rules play a crucial role in tackling complex data validation tasks that may go beyond what standard DQ rules in the web application can handle. By leveraging component rules, you can define custom validation logic tailored to your specific data quality requirements.To ensure accurate and effective results, follow these best practices when working with component rules:1. Creating a Component Rule on ONE DesktopTo create a component rule, follow these steps: Open ONE Desktop and add Ataccama ONE Platform Server by creating a new server connection. Navigate to the ONE Metadata Explorer tab. Select the component you've created in ONE Web Application (e.g., "Email Check" in this example). For more information on creating a component in ONE Web Applicatio

Hi,I want to compare the number of not null records in source (custom sql using JDBC writer) and target (salesforce object) using a VCI. Is there any way to do that in Ataccama version 14.1.0. The aim is to get the records which are in source and not in target and records which are in target but not in source.Thanks.

I am using desktop app, and I want to schedule the workflow. Can anyone give some idea of scheduling workflows? I am trying to achieve the task ie. run the flows as per scheduled time(either weekly or monthly - like 5th day of each month)Below is the screenshot that I am having the file, but some how it is not working for me.Thanks in advance. 

Hello all, i have a task of validating email with the help of api. I want to know is there any documentation which properly explains the integration of postman api to ataccama desktop? or is there any tutorial for working on with apis' using ataccama. Graphql is there but i'm in need of a very basic one to execute and practice.Thank you!