Learn & Share

Hello everyone!Did anyone enable and configure Lineage via by configuring Lineage Database with 3 tables? I don't know where to create, how to fill in the table and how to start setting up. What conditions are needed? What needs to be created?I will be very grateful for your help

I am trying to connect Ataccma to Azure Data lake. Can someone help me on this?I know the container name & Storage Account.. but unable to find Client ID, Authentication token endpoint & client key.  

How one can create a counter with rule for e.g.I am trying to add validation for a column value which will have two rules.Based on three inputs column value,range lower limit and range uper limit.1) If the column value is within the specified range of value (input). Count it's sum how many passed the rule ?2)If the column is out of the specified range of value (input).Count it's sum how many failing the rule?.  

I created a lookup file from existing attributes in the table but some values are missing so I need to update the look up file. How to do that? How to create lkp file outside of web interface and upload to ataccama?

I have rule having 2 attributes but it cannot be linked to business term. How can make use of this rule to validate the atrributes?

Hi All, I have a question about the connection in JSON Call.I had given the url (GraphQL url )to connect in the url section,and in the http headers section, I had given the Authorization in Name and credentials in value.The same thing works in Postman but not in JSON calls.Please make any suggestions. ThanksAvi

I have to parse the below JSON GET response to get “email” and "is_valid_format.value” as columns in Json Call step with Data Streams, How can I perform this? {    "email": "xxxx@abc.com",    "autocorrect": "",    "deliverability": "DELIVERABLE",    "quality_score": "0.99",    "is_valid_format": {        "value": true,        "text": "TRUE"    },    "is_free_email": {        "value": false,        "text": "FALSE"    },    "is_disposable_email": {        "value": false,        "text": "FALSE"    },    "is_role_email": {        "value": false,        "text": "FALSE"    },    "is_catchall_email": {        "value": null,        "text": "UNKNOWN"    },    "is_mx_found": {        "value": true,        "text": "TRUE"    },    "is_smtp_valid": {        "value": true,        "text": "TRUE"    }}

 While working in a live project, there might be a requirement to close or change the status of existing issues in the MDM Administrator (MDA) interface in bulk or on ad-hoc basis automatically whenever certain criteria are met.Some example scenarios when MDA issues require closure could be: Close a split proposal whenever matching attributes of instance records become the same Close all issues with specific data quality criteria that is no longer relevant for client. Example – Company Established Date data quality checks are not required, and client wants to close issues raised for Company Established Date. Close all resolved issues To implement a solution to close issues, certain information is required: The database and tables where issue information is stored:Key Issues details visible in MDA UI are stored in “dqit_issue” table in “it_db” default databases for MDA. Note: Details of configured database can be found in the runtimeConfig file. Key fields in the dqit_issue database:

Based on data sensitivity classification of the data source, we would like to validate if the data is encrypted or in clear.  If the classification is confidential and the data is in the clear (unecrypted) we want to generate a report for the data custodian.  Requirements:  workflow, scheduler, alert notification  

I am using version 13.7 and trying to import two table that consists binary values in two different columns and as a part of check I need to check if the two columns have same values. However, there is no option to import binary values and they are getting imported as string by default. Please help me if there is a way to convert the values into binary or they can be converted back to their original binary values

INTROSometimes you need to enrich your entity view with the combination of attribute values or attributes that belong to some other entity… and there is a way to do it using computed content feature.Following are the two use cases I happen to encounter most often. Show parent attribute in a child entityYou have report and reportColumn entities. Entity reportColumn is an array of embedded objects inside report. Entity report has a reportName attribute.The goal is to add report.reportName to reportColumn entity as reportName attribute. High level steps:Create a new entity named reportNameCC, save it, **DO NOT PUBLISH**. In the Computed Content tab of the entity add parameters and SQL. Edit reportColumn entity  add embedded object with type reportNameCC;  add delegate property to embedded reportNameCC.  Computed Content EntityGoto Organization → Metadata Model Press Create entity and create empty reportNameCC entity. Save but **DO NOT PUBLISH!** Open Computed Content tab and fill in: Inp

I am writing custom steps in Java. the help has some examples. I am wondering to know if the are any javadoc about the ataccama core classes. I would like to test my new custom step using junit. Any reference will help a lot.

Hi,  I am using a parquet file writer and I would like to set the size of the file, how can I do that? 

I tried to profile flat file in ADLS but profiling failed with the below error :    {"status":"FAILURE"}Detail: org/apache/hadoop/conf/Configurationorg.apache.hadoop.conf.Configuration Please help

Hi,Have gotten some good answers here. So here is a case. It is a requirements to merge data.This involves steps:Compare to sets of data - say Target and Source. Any record not in Target but in Source will be added (insert). Any record in Target but not in Source will be flagged ‘deleted’ or deleted (delete) Any record in both sets will be updated - say a timestamp will be updated (update)In SQL it is known as MERGE.One way would be an full outer join and then use filters, with three branches in the plan, but I hope I miss an easier way ;) 

Hi all!There is a table in the source that contains about 100 million records and is updated every day. Is there any way to profile the table so that it adds only new data (for example, those that have appeared today) while not profiling all 100 million rows, but conditionally adding 1 million new records to the 99 million existing ones?

Hi Maksim,Can anyone help me view the broken new_database_connection image at MS SQL Connection error - DQ Analyzer - Ataccama Community? The png image’s status is 'Page not found.'I couldn't connect DQ Analyzer to an Azure SQL database using jTDS- MS SQL. I am using Active Directory - Universal with MFA support authentication.Thank you very much for your time and assistance!JP

Hello,How to write the following condition in Ataccama Desktop (Join step):left join anl.productsjur p on (asd.creditpurpose = p.cp or coalesce(asd.creditpurpose, p.cp) is null)                        and (asd.creditpurpose2 = p.cp2 or coalesce(asd.creditpurpose2, p.cp2) is null)                        and (asd.creditobject = p.co or coalesce(asd.creditobject, p.co) is null)The difficulty is that in the join step only one field for left key and right key is available for selection..

Hi!Just started to learn how to work with Scheduler and Workflows and ran into an error when starting workflow through scheduler:ERROR: Data source 'GP' specified by property dataSourceName must exist. [Jdbc Reader (Jdbc Reader, offending property path=/dataSourceName)]The error occurs in the "run build file" task. This is a normal plan containing a database connection. Because of this, all other tasks remain with the "WAITING" status. I will attach the configuration of my .sch file:<?xml version='1.0' encoding='UTF-8'?><scheduleDefinition><description>test</description><enabled>true</enabled><job class="com.ataccama.adt.scheduler.job.WorkflowJob"><workflow>WF00:missed_products_flow_test.ewf</workflow></job><scheduling>40 16 * * *</scheduling></scheduleDefinition>Can you please tell me what I could be missing? Where to write database connection settings in .sch configuration? I just created a workflow, .sch w

While doing unit testing i can’t exclude special characters in Alphanumeric check.Please let me know the best way to do it.

How can we do decimal validation in Ataccama one web DQ rules. input attribute data type should be float for this rule.  

imagine we have 100 views for which we need to find the count difference in 2 different db’s .we need to create a dynamic query in a while loop which looks up view names one by one and runs until the requirement is met.  I have a few questions to ask :Can we have a “JDBC Reader” which can read dynamic queries? For example, using a “Global Variable” in the JDBC Reader query to pick up view_names for every iteration. As per the solution above, we need to create a while loop to cater multiple iteration. Is there any method/function in Ataccama to achieve this? 

We were trying to get the first name and middle name parsed out of an embedded array object using the json parser, but for some reason, anytime we called name_info.name It would only return the middle name, we saw this patter on all records, if name has middle name, then middle name only would show, if no middle name, then first name would show. With the json parser, it should have returned one line for first name and one line for middle but it didnt Solution: create a data stream that goes from root into name, then for the attributes in that data stream, you will use an @ as the path for the attribute. This will then give you a separate line for first and middle name. The documentation for 13 has this, but since we were working with 12.6 we went through trail and error  "name_info": [ { "name": [ "John", "A" ] }]Data structure example

Is there a stdout writer, or anything that can echo output?This would be the perfect compliment to the Random Record Generator. The ability to specify an input (Random Record Generator) and output (StdOut Writer) without having to use Text File Reader / Writer would allow the help to be much better without having to provide the tutorials (which are complicated). I want just the simplest hello world example for the Pattern Parser to simply split two words separated by white space (please supply!). But the help is inadequate with too much text and no simple example, and the tutorial example “06.04 Pattern Parser” is far too complicated to decode.If there was a StdOut Writer then it would be much easier to supply examples in the help. Simply paste the entire xml plan into the help. Each plan would only need a Random Record Generator to create one record, the step being displayed (Pattern Parser) and a StdOut Write to write the output to the console of the eclipse desktop.

HiI need an example of using JSON Call.I try a simple REST API call, but it does not work.Thanks