Learn & Share

I’d like to implement a dynamically provisioning Ataccama environment for supporting automated unit testing capability. I.eBuilding a Container image of Ataccama software stored in a private cloud and. Developers can rapidly iterate code, commit in a repo, CI triggers application build, container image build and Automated unit testing of new features via Azure pipelines.And It seems Ataccama is working on a Containerized deployments soon according to their web site,Does anybody know the timeline of when this feature can be available to do the above goals? Also can we use it as a preview for unit testing environment?

IntroductionThis post was created to answer the following questions:When moving data through data pipelines, how can the Ataccama ONE DQ platform support Data Integration platforms other than its own. How do other Integration Platforms access the DQ rules? How do other Integration Platforms flag records for Data Stewards to review?DQ platform in context of integration processesThere are three key patterns that are typically used for integration of the ETL process with Data Quality:Option 1The integration platform calls Ataccama DQ rules exposed as real-time API, sending in batches or micro-batches of data for evaluation.Option 2The Integration platform loads data into a staging/landing layer and triggers (via API call) a batch process in Ataccama, Ataccama evaluates/improves the data and populates the cleansed layer in the data lake, including DQ scores and flags. Option 3Ataccama can be connected to a data stream, such as Kafka, and on the fly evaluating data in the stream - every mes

Hello, communityI’m currently working on a project which requires a workflow on the runtime server to be triggered after the monitoring project on Ataccama ONE is finished. Can anyone recommend a solution to achieve this? Thank you in advance

One of the customers had experienced the error upon entering Approval Requests page:And it happened after metadata changes that looked reasonably safe.So what happened?It turns out that if you have an entity that is a part of workflow (draft, approve, …, publish) and you already have published some of them -- deleting attribute which is an embedded array of other entities would lead to this kind of error.Luckily, there is a way to fix it if you have access to MMM database: delete from "mdArbitraryReference_r" where id in ( select id_i from "mdArbitraryReference_q" where "referencedNodePath_s" not in (select id from "_MmdDictionary" where type = 'PATH')); delete from "mdArbitraryReference_q" where "referencedNodePath_s" not in (select id from "_MmdDictionary" where type = 'PATH');These SQL queries fix DB inconsistency removing invalid records from arbitrary references tables.I would still suggest to consult with support if you encounter such an error first, though. :)

I see column level quality score in monitoring project. Do we have row level quality score?

I want to export DQ Evaluation Rules from Ataccama ONE and import them onto another Ataccama environment. To my knowledge, this will require an export plan. However, I’m unsure how to configure the JSON file for the plan.Could anyone guide me on this, please? Thank you in advance

Hi,We'd like to compare two string-fields with several words. Is there a fuzzy matching or approximate string matching step for a plan?In this case it is not reference of lookup data. They are name fields. One is in the DB the other in a file. Kind regards, Marnix

IntroductionSometimes you realize you just performed an unwanted merge & push to origin/remote Git repository. E.g. you made a mistake in selecting "target" and "source" branch and merged the "env" branch to some "feature" instead the other way round. And most probably you want to remedy this as soon as possible and before other team members use such "broken" branch any further.One of the options can be a revert commit but that would undo a previous (or selected) commit, i.e. undo a single commit. But you might have merged a series of commits which you would like to undo at once and set the git branch to a certain state in the past. On top of it, you can even add (cherry-pick) any individual commits that happened after (contributions of your colleagues you want to keep). ProcedureDisclaimer and preparationFollowing procedure uses a “hard” reset; it is recommended not to perform this in the project phase when rapid development is happenning and/or to communicate your intention with

I’m running a monitoring project and it is failing. Here is what I am receiving as a message: Caused by: java.io.IOException: strange number of lookup header sections: 0What does this mean?

You can optimize your plans by simple things. In this case I would like to show how the number of columns/fields influences the performance of the processing. Test conditionslocal environment, stable conditions 14M records generated by Random Record Generator 78 fields containing fixed strings or dates No parallelism level set  Test Cases One stream – simple reading - BASELINEOverall progress: 117k/s Duration: 3:08min Split into 4 streams, same number of fields – you really need ALL fields in ALL streamsOverall progress: 91k/s Duration: 3:42min  Split into 4 streams, reduced number of field to 1 – maybe you just need 1 field in each streamOverall progress: 109k/s Duration: 3:19min  Split into 4 streams, 3 of them with reduced number of fields to 1 – you might not need all fields in other streamsOverall progress: 101k/s Duration: 3:33min  In this example it’s just about simple transformation flow however in case you use JOINs, RepresentativeCreators, Aggregators etc. and you reduce the

I need to connect to an AS400 database. I have an odbc connection working for that but I can use ODBC with DQA. No OBDC driver is available. it is not like what is displayed on the FAQ support page when they explain how to connect to MS access using ODBC.it seems the driver has disapeared :-(

Australian Tax Files Numbers, commonly known as TFNs, are 9-digit numbers where the last digit is a validation check-digit.How would we go about setting up Detection and DQ Evaluation Rules for TFNs?The following article applies to ONE version 13.x. The screenshots are from v13.9.For the Detection Rule, we might want to ignore any extraneous characters in the data, for example, the data that is profiled might contain, TFN:123456782, that could be easily transformed into a valid TFN by removing the prefix.For the DQ Evaluation Rule, we might want to consider values that contain the TFN: prefix like in previous example as invalid, to indicate that further processing/cleansing of the data is required.The good news is that we can use almost the same condition for detecting and validating the numeric part of the TFN. For the DQ Evaluation Rule, we’ll use an extra condition to check that only numbers are present.Detection RuleTo configure the Detection Rule in ONE, head to the Rules section

We have one use case where we are looking for a feature of looping the records and reading it record by record ..  Can you please share any template plan for the same.. 

I have a plan file in which there are multiple flows which generate multiple result csv files having different  number/size of column .I want to store the output of each flow into a single csv file by appending the rows.I want to this happen every time I run the plan file Example - On 1st run of the plan file each result csv file have 10 rows and there are 5 flows . This should lead to a total of 50 rows in the appended file . Now on the 2nd run of the plan file the 10 rows are generated again for each flow , this should lead to a total of 100 rows( 50(1st run) + 50 (2nd run) )  in appended file .

 "When one column have countries, another column have city" This city should be in that country else it is invalid. Please help.

Hello everyone!Did anyone enable and configure Lineage via by configuring Lineage Database with 3 tables? I don't know where to create, how to fill in the table and how to start setting up. What conditions are needed? What needs to be created?I will be very grateful for your help

I am trying to connect Ataccma to Azure Data lake. Can someone help me on this?I know the container name & Storage Account.. but unable to find Client ID, Authentication token endpoint & client key.  

How one can create a counter with rule for e.g.I am trying to add validation for a column value which will have two rules.Based on three inputs column value,range lower limit and range uper limit.1) If the column value is within the specified range of value (input). Count it's sum how many passed the rule ?2)If the column is out of the specified range of value (input).Count it's sum how many failing the rule?.  

I created a lookup file from existing attributes in the table but some values are missing so I need to update the look up file. How to do that? How to create lkp file outside of web interface and upload to ataccama?

I have rule having 2 attributes but it cannot be linked to business term. How can make use of this rule to validate the atrributes?

Hi All, I have a question about the connection in JSON Call.I had given the url (GraphQL url )to connect in the url section,and in the http headers section, I had given the Authorization in Name and credentials in value.The same thing works in Postman but not in JSON calls.Please make any suggestions. ThanksAvi

I have to parse the below JSON GET response to get “email” and "is_valid_format.value” as columns in Json Call step with Data Streams, How can I perform this? {    "email": "xxxx@abc.com",    "autocorrect": "",    "deliverability": "DELIVERABLE",    "quality_score": "0.99",    "is_valid_format": {        "value": true,        "text": "TRUE"    },    "is_free_email": {        "value": false,        "text": "FALSE"    },    "is_disposable_email": {        "value": false,        "text": "FALSE"    },    "is_role_email": {        "value": false,        "text": "FALSE"    },    "is_catchall_email": {        "value": null,        "text": "UNKNOWN"    },    "is_mx_found": {        "value": true,        "text": "TRUE"    },    "is_smtp_valid": {        "value": true,        "text": "TRUE"    }}

 While working in a live project, there might be a requirement to close or change the status of existing issues in the MDM Administrator (MDA) interface in bulk or on ad-hoc basis automatically whenever certain criteria are met.Some example scenarios when MDA issues require closure could be: Close a split proposal whenever matching attributes of instance records become the same Close all issues with specific data quality criteria that is no longer relevant for client. Example – Company Established Date data quality checks are not required, and client wants to close issues raised for Company Established Date. Close all resolved issues To implement a solution to close issues, certain information is required: The database and tables where issue information is stored:Key Issues details visible in MDA UI are stored in “dqit_issue” table in “it_db” default databases for MDA. Note: Details of configured database can be found in the runtimeConfig file. Key fields in the dqit_issue database:

Based on data sensitivity classification of the data source, we would like to validate if the data is encrypted or in clear.  If the classification is confidential and the data is in the clear (unecrypted) we want to generate a report for the data custodian.  Requirements:  workflow, scheduler, alert notification  

I am using version 13.7 and trying to import two table that consists binary values in two different columns and as a part of check I need to check if the two columns have same values. However, there is no option to import binary values and they are getting imported as string by default. Please help me if there is a way to convert the values into binary or they can be converted back to their original binary values