Data Quality & Catalog ⚙️

Australian Tax Files Numbers, commonly known as TFNs, are 9-digit numbers where the last digit is a validation check-digit.How would we go about setting up Detection and DQ Evaluation Rules for TFNs?The following article applies to ONE version 13.x. The screenshots are from v13.9.For the Detection Rule, we might want to ignore any extraneous characters in the data, for example, the data that is profiled might contain, TFN:123456782, that could be easily transformed into a valid TFN by removing the prefix.For the DQ Evaluation Rule, we might want to consider values that contain the TFN: prefix like in previous example as invalid, to indicate that further processing/cleansing of the data is required.The good news is that we can use almost the same condition for detecting and validating the numeric part of the TFN. For the DQ Evaluation Rule, we’ll use an extra condition to check that only numbers are present.Detection RuleTo configure the Detection Rule in ONE, head to the Rules section

I have a plan file in which there are multiple flows which generate multiple result csv files having different  number/size of column .I want to store the output of each flow into a single csv file by appending the rows.I want to this happen every time I run the plan file Example - On 1st run of the plan file each result csv file have 10 rows and there are 5 flows . This should lead to a total of 50 rows in the appended file . Now on the 2nd run of the plan file the 10 rows are generated again for each flow , this should lead to a total of 100 rows( 50(1st run) + 50 (2nd run) )  in appended file .

 "When one column have countries, another column have city" This city should be in that country else it is invalid. Please help.

Hello everyone!Did anyone enable and configure Lineage via by configuring Lineage Database with 3 tables? I don't know where to create, how to fill in the table and how to start setting up. What conditions are needed? What needs to be created?I will be very grateful for your help

How one can create a counter with rule for e.g.I am trying to add validation for a column value which will have two rules.Based on three inputs column value,range lower limit and range uper limit.1) If the column value is within the specified range of value (input). Count it's sum how many passed the rule ?2)If the column is out of the specified range of value (input).Count it's sum how many failing the rule?.  

I created a lookup file from existing attributes in the table but some values are missing so I need to update the look up file. How to do that? How to create lkp file outside of web interface and upload to ataccama?

I have rule having 2 attributes but it cannot be linked to business term. How can make use of this rule to validate the atrributes?

I have to parse the below JSON GET response to get “email” and "is_valid_format.value” as columns in Json Call step with Data Streams, How can I perform this? {    "email": "xxxx@abc.com",    "autocorrect": "",    "deliverability": "DELIVERABLE",    "quality_score": "0.99",    "is_valid_format": {        "value": true,        "text": "TRUE"    },    "is_free_email": {        "value": false,        "text": "FALSE"    },    "is_disposable_email": {        "value": false,        "text": "FALSE"    },    "is_role_email": {        "value": false,        "text": "FALSE"    },    "is_catchall_email": {        "value": null,        "text": "UNKNOWN"    },    "is_mx_found": {        "value": true,        "text": "TRUE"    },    "is_smtp_valid": {        "value": true,        "text": "TRUE"    }}

Based on data sensitivity classification of the data source, we would like to validate if the data is encrypted or in clear.  If the classification is confidential and the data is in the clear (unecrypted) we want to generate a report for the data custodian.  Requirements:  workflow, scheduler, alert notification  

INTROSometimes you need to enrich your entity view with the combination of attribute values or attributes that belong to some other entity… and there is a way to do it using computed content feature.Following are the two use cases I happen to encounter most often. Show parent attribute in a child entityYou have report and reportColumn entities. Entity reportColumn is an array of embedded objects inside report. Entity report has a reportName attribute.The goal is to add report.reportName to reportColumn entity as reportName attribute. High level steps:Create a new entity named reportNameCC, save it, **DO NOT PUBLISH**. In the Computed Content tab of the entity add parameters and SQL. Edit reportColumn entity  add embedded object with type reportNameCC;  add delegate property to embedded reportNameCC.  Computed Content EntityGoto Organization → Metadata Model Press Create entity and create empty reportNameCC entity. Save but **DO NOT PUBLISH!** Open Computed Content tab and fill in: Inp

I tried to profile flat file in ADLS but profiling failed with the below error :    {"status":"FAILURE"}Detail: org/apache/hadoop/conf/Configurationorg.apache.hadoop.conf.Configuration Please help

Hi,Have gotten some good answers here. So here is a case. It is a requirements to merge data.This involves steps:Compare to sets of data - say Target and Source. Any record not in Target but in Source will be added (insert). Any record in Target but not in Source will be flagged ‘deleted’ or deleted (delete) Any record in both sets will be updated - say a timestamp will be updated (update)In SQL it is known as MERGE.One way would be an full outer join and then use filters, with three branches in the plan, but I hope I miss an easier way ;) 

Hi all!There is a table in the source that contains about 100 million records and is updated every day. Is there any way to profile the table so that it adds only new data (for example, those that have appeared today) while not profiling all 100 million rows, but conditionally adding 1 million new records to the 99 million existing ones?

Hi!Just started to learn how to work with Scheduler and Workflows and ran into an error when starting workflow through scheduler:ERROR: Data source 'GP' specified by property dataSourceName must exist. [Jdbc Reader (Jdbc Reader, offending property path=/dataSourceName)]The error occurs in the "run build file" task. This is a normal plan containing a database connection. Because of this, all other tasks remain with the "WAITING" status. I will attach the configuration of my .sch file:<?xml version='1.0' encoding='UTF-8'?><scheduleDefinition><description>test</description><enabled>true</enabled><job class="com.ataccama.adt.scheduler.job.WorkflowJob"><workflow>WF00:missed_products_flow_test.ewf</workflow></job><scheduling>40 16 * * *</scheduling></scheduleDefinition>Can you please tell me what I could be missing? Where to write database connection settings in .sch configuration? I just created a workflow, .sch w

While doing unit testing i can’t exclude special characters in Alphanumeric check.Please let me know the best way to do it.

How can we do decimal validation in Ataccama one web DQ rules. input attribute data type should be float for this rule.  

HiI need an example of using JSON Call.I try a simple REST API call, but it does not work.Thanks 

 I have created Business term employee id and detection and DQ rules and linked the rule to business term wherever possible but when I tried to evaluate the following error displayed always: Please help! Couldn't run DQ evaluationNo matching rules found for term attributes.

How can we write case statements in Ataccama one web DQ rules?As of now we are writing multiple conditions with distinct invalid statements. I am looking for a better approach rather than going the traditional way. Thanks,Mahesh

@anna.spakova @Catherine I need some sample look up file so I can understand the format of the look up file. Could you please provide. Thanks!

Hi all!Please tell me where to look to cope with the error "Cannot submit job: Cannot execute task 'preprocess job' for job xxx. Caused by: Cannot retrieve metadata node" when starting remote plan execution. When the plan is run, it goes through the SUBMITTED and PREPROCESSING stages and then ends up in the FAILURE stage with the above error.For clarification: my account has full admin rights in ONE Web, so I have access to the catalog item.

simply I use here two columns date and kpi and sorting of date its fine but if we use record descriptor builder, its missing the date wise sorting.out: Steps:Sort → have date column sorting.Alter→ have rd1 <String> and record_counter<Interger> Column additionRecord descriptor builder → have rd1 binding with date and kpi column  

I would like to add new column e.g. Description to the column bar, next to Abbreviation, but not familiar how to achieve it?  current layout syntax 

Hi there, From the official website, I  found the business terms could show data quality score on from category to element term level.  I wonder how the sub-category item do an aggregation data quality score based on the element(column items). I understand that data quality rule could apply on column level or table level to show the quality result.  For our current environment, I’ve set the hierarchy relationship, Data Domain is parent relationship to Data set, Data Set is the Parent relationship to Tables, which looks like Data Domain > Data Set > Data Table. How can it show data quality score on Data Set(Sub-category) and Data Domain level(Category).   Thank you! Best regards,Vicky