Data Quality & Catalog ⚙️

We created a number of data quality business rules that will fail records from the source file we are reading.  1.5 Million records.How do you port the failed records to a flat file?  There is no drill down in One Desktop so we are having issues getting business users approval on code without seeing the failed records?

I have a rule binded to 2 columns in a catalog item.By mistake i have changed the data type of those columns without unbinding the rule.Now i am unable to unbind thise rules from that column.i am getting error while saving or publishing the project : server refused sent data as invalid. Please guide me,what i will do here

​How to tune/optimize a DQC load plan?In day-to-day life, the consultants may come across several performance issues of various magnitude in their load plans at some point in the projects. There are several optimization tips mentioned below for tuning the long running plans.Check the memory allocationDQC plans store the data while processing, which means that the more input data there is, the more memory DQC will require. Therefore the intensive executions can result in a memory overflow error if the default memory allocated for DQC is consumed. In such cases, an increase of memory could present a solution.Related Errors: OutOfMemoryError: Java Heap Space or GC Overhead Limit Exceeded... java.lang.OutOfMemoryError: unable to create new native thread... The memory allocation can be done in several ways,Increasing the memory of the DQC IDE(Desktop) Increasing the memory allocation to specific plan in the IDE Increasing the memory allocation for the entire project in IDE Increasing the me

If you would like to filter e.g. business terms, use:$type == 'businessTerm'Switch to Advanced filtering with AQL (Ataccama Query Language)Terms list before applying a filterFiltering resultsLink to documentation:Searching Using AQL Expressions

Every step that allows for the input of an expression shows a button called Template.... This button can be used to pre-fill the Expression field with one from an out-of-the-box list.You can create your own expressions for common or complex repeatable operations, follow these steps:Go to Window in the top navigation menu and select Preferences Navigate the menu to Ataccama DQC and go to Expression Editor and select TemplatesChoose between Expression Templates and RegEx TemplatesSelect FolderClick on Add Template..Fill in Name, Expression and DescriptionConfirm with OK

Any one can help to create this template using version 13.6?  It show the items clearly.  Our users can know the data presentation without explanation.  However, I can find this template in our current version.

Any best practices for using the Purpose function in Ataccama One (webapp)? I see that it is recommended for monitoring projects but when I create a monitoring project I want to be able to search for that tag. Any tips for searching just based on the purpose tag? I’m very new to Ataccama so any and all tips would be appreciated!

I’m trying to read the file name of a file so I can write it in an email within a Workflow. I’m using the “Operate On File” task. The Operation I selected is “INFO”. I then point to the Source File. Next I’m trying to determine which variable I use to supply to the Send Email task. The Operate On File Documentation says you can use task variables and the example they give is “name”, “localName”, “size”, and “timestamp”. When I try using these variables in the Send Email task they just come across as blank in the email (variables appear to be null). Here is the body of the email I’m using: Name = ${name}localName = ${localName} Two questions:Does any one happen to know what I’m doing wrong in above?Additionally is there a way to pull out a file name within a Plan. I’m using the “Excel File Reader” step. I want to pass along the file name in the flow so I can use it later on. Thanks!

Hi, I’ve just started data profiling with Ataccama One so please bear with me.I have data with technical IDs (e.g. “000123456”, “009876543”,...) in a csv that I imported. The IDs are however interpreted as a number.How exactly can I change the attribute type to text? I have done so when importing the file: but it seems to go to numbers nonetheless.The original file has comma-separated values that are all between double quotes. Thanks for any help! edit: typo

I opened the on-prem AtaccamaONE environment this morning and was looking at recent (auto) profiled catalog items. A majority of them are not found - the error is “Catalog item not found” and the provided explanation is “This catalog item doesn’t exist anymore or someone has revoked your access”. Neither of these cases should be true as I am the only one who currently works in the environment and I’m the one who launched the data discovery on 3/09. This is a recent error - I have not seen this before. Any help you could provide would be appreciated.Thanks.

How can we use functionality like ‘for’ and ‘while’ loop in detection rule field while writing/creating rules for capturing data/records.

Hello, I’m new with Ataccama and I have a simple problem. I want call a Rest service:https://api.bulkemailverifier.com/api/a/v1?key=xxxxxxxxxx&email=myemail.comI’m using the JSON Call step and If I put the full URL with my email it’s working, but now I want read the emails forma txt file and I don’t know how to make the url dynamic.In the step I have 3 parameters: URL: Resource  I set it as https://api.bulkemailverifier.com/api/a/v1?key=xxxxxxxxxx&email=Method: GETURL: HERE IS WHERE i DONT KNOW WHAT TYPE. I have tried a lot of combinations without successThe input file has one column with the mail: src_mailThanks 

How can we download files from remote path ( connected through vpn) on to local?I tried with Operate on File component but it is not working.Is there any work around for it?

Enterprise Data Governance as the management of data across its pipeline - from production to consumption (and retirement). How to enable our “data fellows” to leverage on data to save money, save time and reduce risk?What are the key elements to success? What are your thoughts? 

I was wondering if there is a way in Ataccama that I can move my rows data into columns?I’ve data as such:Element:          Dimension:                 Values:         Code: XYZ            Validity                        200               678XYZ            Completeness              250               678 YUP            Completeness            300                555 HIO             Accuracy                     400                100 I’d like it to look like:Element:          Validity:                 Completeness:        Accuracy;         Code: XYZ                 200                               250                         250               678 YUP                500                                300                        300                555 HIO                 200                                450                        400                100 Thank you!  

Data Stories provides a wide range of charts, it includes also Pie and Donut ones, even though I do not recommend using them, and the same applies to a Gauge chart. Let's look at the reasoning why and why I'd recommend using a (stacked) bar chart instead.Space usageIn data visualization (reporting, data storytelling) you have limited space you can use. Your task is to fill the limited space with the most value for the end viewer/user. Note, most value does not necessarily mean most charts, sometimes one curated chart is far better than five. The problem with Pie, Donut and Gauge charts is that they use more space in comparison to an alternative that provides the same value - a bar chart. Sales distribution on a Pie ChartMore space is used due to 2 reasons:Pie and Donut charts use Area to give value to a single category. Bar chars use just distance. An Area just takes more space All these charts are circular, which creates unnecessary space around the chart objectBellow find a better al

I am trying to build a component rule where I can compare primary_id from catalog table A to another table B using join and send primary_id existing in both tables, existing only in catalog table A and existing only in B table in the one report. Also checking if that is possible through Lookup

Did you know that you can reorder or add custom columns to Listings within ONE? Here is an example where we move the validity score to be the second column. You could also disable various columns here if you would like. To do this, you would add the following in Debugging Tools, for more information, please see our documentation. { "_type": "entity.page.listing", "children": { "_type": "relationships.hierarchyListing", "enableSelection": true, "columns": { "name":{ "name": "Name" }, "dqEvalTermAggr": { "name": "Overall Validity" }, "type": { "dataPath": [], "name": "Type", "after": "dqEvalTermAggr", "renderer": { "_type": "glossary.termType.tableCell" } }, "abbreviation":{ "name":"Abbreviation", "after":"type" } } }} 

DQA offers a way to monitor the processing visually, providing counts of processed records, percentages on completion of the different operations (steps) and visual cues to the flow of data to identify bottlenecks. This can be achieved by opening a special view of any plan being executed, which will mirror the original plan, showing the processing of records and other aforementioned visual features.To open the monitoring, click the Show progress button, which appears in the Console when the plan is being executed.You can also configure this monitoring window to be opened automatically whenever a plan is executed. The default configuration for the monitoring can be changed by:Open Window > Preferences Navigate to Ataccama DQ Analyzer > Launching > Progress Viewer. Change Automatically open progress viewer for launched plans to Always. You can also change how the Progress Viewer should behave once the execution finishes. 

Table profile creation can be automated after buying the more advanced Data Quality Center (DQC) software.Schedule a call | Ataccama On the other hand, with Data Quality Analyzer (DQA, free software), the automation is not possible.  

You might experience problems when installing DQA on a computer with limited permissions or high restrictions. Even if the installation is done in a folder where you have rights for installation, DQA might fail to execute plans. The reason is that DQA makes use of a temporary folder (usually in the path to the user cache folder) for placing instances of executed plans. This folder is the default Java temporary folder. The execution might throw an exception due to being unable to create a temp plan file.You can override the path to this folder by passing its chosen value as part of the special environment variable JAVA_OPTS , which serves as the Java Virtual Machine arguments. This should be created as follows:Open for editing the dqa.ini file placed in the root folder of the DQA installation, using an editor sensitive to End Of Lines (EOLs) in Windows, WordPad is an option, Notepad++ is another option. Add the variable java.io.tempdir . The variable must be passed using the -D notation

If you experience out of memory errors when processing a large dataset. You can increase the amount of system memory that is available to the Ataccama DQA application.Open dqa.ini located in the root folder of the DQA installation. This file contains the JAVA_OPTS that will be used by the virtual machine on start.Edit Xmx and substitute the number (256 by default) by a higher number for reserving more memory, e.g. -Xmx1024