Learn & Share

Australian Tax Files Numbers, commonly known as TFNs, are 9-digit numbers where the last digit is a validation check-digit.How would we go about setting up Detection and DQ Evaluation Rules for TFNs?The following article applies to ONE version 13.x. The screenshots are from v13.9.For the Detection Rule, we might want to ignore any extraneous characters in the data, for example, the data that is profiled might contain, TFN:123456782, that could be easily transformed into a valid TFN by removing the prefix.For the DQ Evaluation Rule, we might want to consider values that contain the TFN: prefix like in previous example as invalid, to indicate that further processing/cleansing of the data is required.The good news is that we can use almost the same condition for detecting and validating the numeric part of the TFN. For the DQ Evaluation Rule, we’ll use an extra condition to check that only numbers are present.Detection RuleTo configure the Detection Rule in ONE, head to the Rules section

We have one use case where we are looking for a feature of looping the records and reading it record by record ..  Can you please share any template plan for the same.. 

I have a plan file in which there are multiple flows which generate multiple result csv files having different  number/size of column .I want to store the output of each flow into a single csv file by appending the rows.I want to this happen every time I run the plan file Example - On 1st run of the plan file each result csv file have 10 rows and there are 5 flows . This should lead to a total of 50 rows in the appended file . Now on the 2nd run of the plan file the 10 rows are generated again for each flow , this should lead to a total of 100 rows( 50(1st run) + 50 (2nd run) )  in appended file .

 "When one column have countries, another column have city" This city should be in that country else it is invalid. Please help.

Hello everyone!Did anyone enable and configure Lineage via by configuring Lineage Database with 3 tables? I don't know where to create, how to fill in the table and how to start setting up. What conditions are needed? What needs to be created?I will be very grateful for your help

I am trying to connect Ataccma to Azure Data lake. Can someone help me on this?I know the container name & Storage Account.. but unable to find Client ID, Authentication token endpoint & client key.  

How one can create a counter with rule for e.g.I am trying to add validation for a column value which will have two rules.Based on three inputs column value,range lower limit and range uper limit.1) If the column value is within the specified range of value (input). Count it's sum how many passed the rule ?2)If the column is out of the specified range of value (input).Count it's sum how many failing the rule?.  

I created a lookup file from existing attributes in the table but some values are missing so I need to update the look up file. How to do that? How to create lkp file outside of web interface and upload to ataccama?

I have rule having 2 attributes but it cannot be linked to business term. How can make use of this rule to validate the atrributes?

Hi All, I have a question about the connection in JSON Call.I had given the url (GraphQL url )to connect in the url section,and in the http headers section, I had given the Authorization in Name and credentials in value.The same thing works in Postman but not in JSON calls.Please make any suggestions. ThanksAvi

I have to parse the below JSON GET response to get “email” and "is_valid_format.value” as columns in Json Call step with Data Streams, How can I perform this? {    "email": "xxxx@abc.com",    "autocorrect": "",    "deliverability": "DELIVERABLE",    "quality_score": "0.99",    "is_valid_format": {        "value": true,        "text": "TRUE"    },    "is_free_email": {        "value": false,        "text": "FALSE"    },    "is_disposable_email": {        "value": false,        "text": "FALSE"    },    "is_role_email": {        "value": false,        "text": "FALSE"    },    "is_catchall_email": {        "value": null,        "text": "UNKNOWN"    },    "is_mx_found": {        "value": true,        "text": "TRUE"    },    "is_smtp_valid": {        "value": true,        "text": "TRUE"    }}

 While working in a live project, there might be a requirement to close or change the status of existing issues in the MDM Administrator (MDA) interface in bulk or on ad-hoc basis automatically whenever certain criteria are met.Some example scenarios when MDA issues require closure could be: Close a split proposal whenever matching attributes of instance records become the same Close all issues with specific data quality criteria that is no longer relevant for client. Example – Company Established Date data quality checks are not required, and client wants to close issues raised for Company Established Date. Close all resolved issues To implement a solution to close issues, certain information is required: The database and tables where issue information is stored:Key Issues details visible in MDA UI are stored in “dqit_issue” table in “it_db” default databases for MDA. Note: Details of configured database can be found in the runtimeConfig file. Key fields in the dqit_issue database:

Based on data sensitivity classification of the data source, we would like to validate if the data is encrypted or in clear.  If the classification is confidential and the data is in the clear (unecrypted) we want to generate a report for the data custodian.  Requirements:  workflow, scheduler, alert notification  

I am using version 13.7 and trying to import two table that consists binary values in two different columns and as a part of check I need to check if the two columns have same values. However, there is no option to import binary values and they are getting imported as string by default. Please help me if there is a way to convert the values into binary or they can be converted back to their original binary values

INTROSometimes you need to enrich your entity view with the combination of attribute values or attributes that belong to some other entity… and there is a way to do it using computed content feature.Following are the two use cases I happen to encounter most often. Show parent attribute in a child entityYou have report and reportColumn entities. Entity reportColumn is an array of embedded objects inside report. Entity report has a reportName attribute.The goal is to add report.reportName to reportColumn entity as reportName attribute. High level steps:Create a new entity named reportNameCC, save it, **DO NOT PUBLISH**. In the Computed Content tab of the entity add parameters and SQL. Edit reportColumn entity  add embedded object with type reportNameCC;  add delegate property to embedded reportNameCC.  Computed Content EntityGoto Organization → Metadata Model Press Create entity and create empty reportNameCC entity. Save but **DO NOT PUBLISH!** Open Computed Content tab and fill in: Inp

I am writing custom steps in Java. the help has some examples. I am wondering to know if the are any javadoc about the ataccama core classes. I would like to test my new custom step using junit. Any reference will help a lot.

Hi,  I am using a parquet file writer and I would like to set the size of the file, how can I do that? 

I tried to profile flat file in ADLS but profiling failed with the below error :    {"status":"FAILURE"}Detail: org/apache/hadoop/conf/Configurationorg.apache.hadoop.conf.Configuration Please help

Hi,Have gotten some good answers here. So here is a case. It is a requirements to merge data.This involves steps:Compare to sets of data - say Target and Source. Any record not in Target but in Source will be added (insert). Any record in Target but not in Source will be flagged ‘deleted’ or deleted (delete) Any record in both sets will be updated - say a timestamp will be updated (update)In SQL it is known as MERGE.One way would be an full outer join and then use filters, with three branches in the plan, but I hope I miss an easier way ;) 

Hi all!There is a table in the source that contains about 100 million records and is updated every day. Is there any way to profile the table so that it adds only new data (for example, those that have appeared today) while not profiling all 100 million rows, but conditionally adding 1 million new records to the 99 million existing ones?

Hi Maksim,Can anyone help me view the broken new_database_connection image at MS SQL Connection error - DQ Analyzer - Ataccama Community? The png image’s status is 'Page not found.'I couldn't connect DQ Analyzer to an Azure SQL database using jTDS- MS SQL. I am using Active Directory - Universal with MFA support authentication.Thank you very much for your time and assistance!JP

Hi!Just started to learn how to work with Scheduler and Workflows and ran into an error when starting workflow through scheduler:ERROR: Data source 'GP' specified by property dataSourceName must exist. [Jdbc Reader (Jdbc Reader, offending property path=/dataSourceName)]The error occurs in the "run build file" task. This is a normal plan containing a database connection. Because of this, all other tasks remain with the "WAITING" status. I will attach the configuration of my .sch file:<?xml version='1.0' encoding='UTF-8'?><scheduleDefinition><description>test</description><enabled>true</enabled><job class="com.ataccama.adt.scheduler.job.WorkflowJob"><workflow>WF00:missed_products_flow_test.ewf</workflow></job><scheduling>40 16 * * *</scheduling></scheduleDefinition>Can you please tell me what I could be missing? Where to write database connection settings in .sch configuration? I just created a workflow, .sch w

While doing unit testing i can’t exclude special characters in Alphanumeric check.Please let me know the best way to do it.

How can we do decimal validation in Ataccama one web DQ rules. input attribute data type should be float for this rule.  

imagine we have 100 views for which we need to find the count difference in 2 different db’s .we need to create a dynamic query in a while loop which looks up view names one by one and runs until the requirement is met.  I have a few questions to ask :Can we have a “JDBC Reader” which can read dynamic queries? For example, using a “Global Variable” in the JDBC Reader query to pick up view_names for every iteration. As per the solution above, we need to create a while loop to cater multiple iteration. Is there any method/function in Ataccama to achieve this?