Skip to main content

Product Updates
Events
Training Resources
Groups

Home
Learn & Share
Data Quality & Data Governance ⚙️

Data Quality & Data Governance ⚙️

Discuss, learn, ask, suggest, share all things DQ and DG 🔍

533 Topics
1,276 Replies

When you subscribe we will email you when there is a new topic in this category

533 Topics

Recently active

Most replies Most views Newest first

A

Anonymous

published in Data Quality & Data Governance ⚙️

How Ataccama ONE uses Snowpark data frames to automate profiling

Ataccama is known and appreciated by our customers for complex, big deployments where massive amounts of data are processed. We were one of the first to support MapReduce and Spark as native pushdown and enable distributed processing across different cluster nodes.Ataccama’s engine works with the major of enterprise data sources and processes them to provide valuable insight, such as statistics of the data, detection of domains, and data quality.To provide the best experience for our customers that use Snowflake, we developed a tighter integration between our technologies. Specifically, we use the Snowflake data processing capabilities in a pushdown manner which means that data is processed directly on Snowflake.The key advantages of this integration are:Lower data transfer costs for the client. Snowflake’s computational capabilities deliver results faster because of reduced IO operations. Customers don’t incur additional costs to maintain external systems for Big Data. Improved securi

A

RoberTallitschData Voyager

asked in Data Quality & Data Governance ⚙️

Creating flat file using Ataccama One Desktop where records failed business rule

We created a number of data quality business rules that will fail records from the source file we are reading. 1.5 Million records.How do you port the failed records to a flat file? There is no drill down in One Desktop so we are having issues getting business users approval on code without seeing the failed records?

SmrutiData Voyager

asked in Data Quality & Data Governance ⚙️

error while publishing the project

I have a rule binded to 2 columns in a catalog item.By mistake i have changed the data type of those columns without unbinding the rule.Now i am unable to unbind thise rules from that column.i am getting error while saving or publishing the project : server refused sent data as invalid. Please guide me,what i will do here

VaniAtaccamer

posted in Data Quality & Data Governance ⚙️

How to tune/optimize a DQC load plan?

How to tune/optimize a DQC load plan?In day-to-day life, the consultants may come across several performance issues of various magnitude in their load plans at some point in the projects. There are several optimization tips mentioned below for tuning the long running plans.Check the memory allocationDQC plans store the data while processing, which means that the more input data there is, the more memory DQC will require. Therefore the intensive executions can result in a memory overflow error if the default memory allocated for DQC is consumed. In such cases, an increase of memory could present a solution.Related Errors: OutOfMemoryError: Java Heap Space or GC Overhead Limit Exceeded... java.lang.OutOfMemoryError: unable to create new native thread... The memory allocation can be done in several ways,Increasing the memory of the DQC IDE(Desktop) Increasing the memory allocation to specific plan in the IDE Increasing the memory allocation for the entire project in IDE Increasing the me

A

Anonymous

published in Data Quality & Data Governance ⚙️

How would you filter the terms (AQL) by a type?

If you would like to filter e.g. business terms, use:$type == 'businessTerm'Switch to Advanced filtering with AQL (Ataccama Query Language)Terms list before applying a filterFiltering resultsLink to documentation:Searching Using AQL Expressions

A

A

Anonymous

published in Data Quality & Data Governance ⚙️

How to Add Predefined Templates for ExpressionsF.A.Q.

Every step that allows for the input of an expression shows a button called Template.... This button can be used to pre-fill the Expression field with one from an out-of-the-box list.You can create your own expressions for common or complex repeatable operations, follow these steps:Go to Window in the top navigation menu and select Preferences Navigate the menu to Ataccama DQC and go to Expression Editor and select TemplatesChoose between Expression Templates and RegEx TemplatesSelect FolderClick on Add Template..Fill in Name, Expression and DescriptionConfirm with OK

A

A

Anonymous

published in Data Quality & Data Governance ⚙️

Showing Column Names when Profiling Excel FilesF.A.Q.

Do you profile an Excel file and are you unable to see the column names? DQ Analyzer is not processing the column header and in the profiling results, you see the names of the columns as Field_0, Field_1, etc.You can achieve the desired result by editing the metadata of the file you want to profile.Find the necessary file under the File Explorer window. Right-click on it and choose the Edit Metadata option. Specify the column you want it to start with. You should be able to see that the Name of the columns will change automatically. You can verify the column type, as well. Then you can profile the file.

A

ahmingData Voyager

asked in Data Quality & Data Governance ⚙️

Default template

Any one can help to create this template using version 13.6? It show the items clearly. Our users can know the data presentation without explanation. However, I can find this template in our current version.

A

Anonymous

published in Data Quality & Data Governance ⚙️

Data Type Conversions in DQCF.A.Q.

Data type conversions occurrences when working with databases are the natural behaviour of DQC. When reading data from a database (e.g. using JDBC Reader step), the data type is read from the database and transformed into DQC data type.DQC internally works with its own data types. When writing the data back to the database (in such case, JDBS Writer step is commonly used), data type alternation takes place. The rules for data types conversions are defined in SQL Transformation Rules. There is a possibility to change from what exact database types there will be a conversion to DQC types. To redefine the database types from which DQC will transform data into its own data types:Go to Window > Preferences Select Ataccama DQC > Database > SQL Transformation Rules Modify SQL transformation rules of the specific database product you use

A

A

Anonymous

published in Data Quality & Data Governance ⚙️

How can I find and replace all values without opening the plans?F.A.Q.

You can find and replace using the Search option in the upper panel of DQC.Click on Search and enter the text you would like to replace. Click on Replace, and enter the new text you wish to use as a replacement. Clicking on Preview will display a dialog showing where text was found, and a preview of the XML representation of the plans, before and after the change. You may want to uncheck some of the locations. Click OK.

A

jdc122Data Voyager

asked in Data Quality & Data Governance ⚙️

How best to use the Purpose function in Ataccama?

Any best practices for using the Purpose function in Ataccama One (webapp)? I see that it is recommended for monitoring projects but when I create a monitoring project I want to be able to search for that tag. Any tips for searching just based on the purpose tag? I’m very new to Ataccama so any and all tips would be appreciated!

A

jhoppeschData Voyager

asked in Data Quality & Data Governance ⚙️

Operate on File: INFO Operation

I’m trying to read the file name of a file so I can write it in an email within a Workflow. I’m using the “Operate On File” task. The Operation I selected is “INFO”. I then point to the Source File. Next I’m trying to determine which variable I use to supply to the Send Email task. The Operate On File Documentation says you can use task variables and the example they give is “name”, “localName”, “size”, and “timestamp”. When I try using these variables in the Send Email task they just come across as blank in the email (variables appear to be null). Here is the body of the email I’m using: Name = ${name}localName = ${localName} Two questions:Does any one happen to know what I’m doing wrong in above?Additionally is there a way to pull out a file name within a Plan. I’m using the “Excel File Reader” step. I want to pass along the file name in the flow so I can use it later on. Thanks!

semoData Voyager

asked in Data Quality & Data Governance ⚙️

Data profiling - set attribute type

Hi, I’ve just started data profiling with Ataccama One so please bear with me.I have data with technical IDs (e.g. “000123456”, “009876543”,...) in a csv that I imported. The IDs are however interpreted as a number.How exactly can I change the attribute type to text? I have done so when importing the file: but it seems to go to numbers nonetheless.The original file has comma-separated values that are all between double quotes. Thanks for any help! edit: typo

ddw_MH22Data Voyager

asked in Data Quality & Data Governance ⚙️

catalog item not found error

I opened the on-prem AtaccamaONE environment this morning and was looking at recent (auto) profiled catalog items. A majority of them are not found - the error is “Catalog item not found” and the provided explanation is “This catalog item doesn’t exist anymore or someone has revoked your access”. Neither of these cases should be true as I am the only one who currently works in the environment and I’m the one who launched the data discovery on 3/09. This is a recent error - I have not seen this before. Any help you could provide would be appreciated.Thanks.

kumaritarData Voyager

asked in Data Quality & Data Governance ⚙️

For or while loop in detection rule

How can we use functionality like ‘for’ and ‘while’ loop in detection rule field while writing/creating rules for capturing data/records.

onipledData Voyager

asked in Data Quality & Data Governance ⚙️

JSON Call. Get Method dynamic parameter

Hello, I’m new with Ataccama and I have a simple problem. I want call a Rest service:https://api.bulkemailverifier.com/api/a/v1?key=xxxxxxxxxx&email=myemail.comI’m using the JSON Call step and If I put the full URL with my email it’s working, but now I want read the emails forma txt file and I don’t know how to make the url dynamic.In the step I have 3 parameters: URL: Resource I set it as https://api.bulkemailverifier.com/api/a/v1?key=xxxxxxxxxx&email=Method: GETURL: HERE IS WHERE i DONT KNOW WHAT TYPE. I have tried a lot of combinations without successThe input file has one column with the mail: src_mailThanks

ashpoghoshData Voyager

asked in Data Quality & Data Governance ⚙️

Copy File from remote location

How can we download files from remote path ( connected through vpn) on to local?I tried with Operate on File component but it is not working.Is there any work around for it?

LucaAtaccamer

posted in Data Quality & Data Governance ⚙️

Data Governance & why we need it

Enterprise Data Governance as the management of data across its pipeline - from production to consumption (and retirement). How to enable our “data fellows” to leverage on data to save money, save time and reduce risk?What are the key elements to success? What are your thoughts?

anu1237Data Voyager

asked in Data Quality & Data Governance ⚙️

Rows to Columns

I was wondering if there is a way in Ataccama that I can move my rows data into columns?I’ve data as such:Element: Dimension: Values: Code: XYZ Validity 200 678XYZ Completeness 250 678 YUP Completeness 300 555 HIO Accuracy 400 100 I’d like it to look like:Element: Validity: Completeness: Accuracy; Code: XYZ 200 250 250 678 YUP 500 300 300 555 HIO 200 450 400 100 Thank you!

hooAtaccamer

posted in Data Quality & Data Governance ⚙️

Why Pie charts are evil?

Data Stories provides a wide range of charts, it includes also Pie and Donut ones, even though I do not recommend using them, and the same applies to a Gauge chart. Let's look at the reasoning why and why I'd recommend using a (stacked) bar chart instead.Space usageIn data visualization (reporting, data storytelling) you have limited space you can use. Your task is to fill the limited space with the most value for the end viewer/user. Note, most value does not necessarily mean most charts, sometimes one curated chart is far better than five. The problem with Pie, Donut and Gauge charts is that they use more space in comparison to an alternative that provides the same value - a bar chart. Sales distribution on a Pie ChartMore space is used due to 2 reasons:Pie and Donut charts use Area to give value to a single category. Bar chars use just distance. An Area just takes more space All these charts are circular, which creates unnecessary space around the chart objectBellow find a better al

Neeraj SinghData Voyager

posted in Data Quality & Data Governance ⚙️

Use of Joiner in component rule

I am trying to build a component rule where I can compare primary_id from catalog table A to another table B using join and send primary_id existing in both tables, existing only in catalog table A and existing only in B table in the one report. Also checking if that is possible through Lookup

nathaniel.reid-smithAtaccamer

published in Data Quality & Data Governance ⚙️

Reorder Columns in ONE ListingTutorial

Did you know that you can reorder or add custom columns to Listings within ONE? Here is an example where we move the validity score to be the second column. You could also disable various columns here if you would like. To do this, you would add the following in Debugging Tools, for more information, please see our documentation. { "_type": "entity.page.listing", "children": { "_type": "relationships.hierarchyListing", "enableSelection": true, "columns": { "name":{ "name": "Name" }, "dqEvalTermAggr": { "name": "Overall Validity" }, "type": { "dataPath": [], "name": "Type", "after": "dqEvalTermAggr", "renderer": { "_type": "glossary.termType.tableCell" } }, "abbreviation":{ "name":"Abbreviation", "after":"type" } } }}

DannyRyanHead of Technical Training

published in Data Quality & Data Governance ⚙️

How to monitor the current execution of a plan?Tutorial

DQA offers a way to monitor the processing visually, providing counts of processed records, percentages on completion of the different operations (steps) and visual cues to the flow of data to identify bottlenecks. This can be achieved by opening a special view of any plan being executed, which will mirror the original plan, showing the processing of records and other aforementioned visual features.To open the monitoring, click the Show progress button, which appears in the Console when the plan is being executed.You can also configure this monitoring window to be opened automatically whenever a plan is executed. The default configuration for the monitoring can be changed by:Open Window > Preferences Navigate to Ataccama DQ Analyzer > Launching > Progress Viewer. Change Automatically open progress viewer for launched plans to Always. You can also change how the Progress Viewer should behave once the execution finishes.

DannyRyanHead of Technical Training

published in Data Quality & Data Governance ⚙️

Is it possible to automate table profile creation?F.A.Q.

Table profile creation can be automated after buying the more advanced Data Quality Center (DQC) software.Schedule a call | Ataccama On the other hand, with Data Quality Analyzer (DQA, free software), the automation is not possible.

DannyRyanHead of Technical Training

published in Data Quality & Data Governance ⚙️

How to configure the temporary folder used by DQA (permissions error)?F.A.Q.

You might experience problems when installing DQA on a computer with limited permissions or high restrictions. Even if the installation is done in a folder where you have rights for installation, DQA might fail to execute plans. The reason is that DQA makes use of a temporary folder (usually in the path to the user cache folder) for placing instances of executed plans. This folder is the default Java temporary folder. The execution might throw an exception due to being unable to create a temp plan file.You can override the path to this folder by passing its chosen value as part of the special environment variable JAVA_OPTS , which serves as the Java Virtual Machine arguments. This should be created as follows:Open for editing the dqa.ini file placed in the root folder of the DQA installation, using an editor sensitive to End Of Lines (EOLs) in Windows, WordPad is an option, Notepad++ is another option. Add the variable java.io.tempdir . The variable must be passed using the -D notation

1
...
18
19
20
21
22

Page 21 / 22

Badge winners

GastonQhas earned the badge First idea 💡
may_kwokhas earned the badge 10 ideas 💡
DiBetzhas earned the badge First idea 💡
Lindsay Owenshas earned the badge First idea 💡
AndrewH_LBGhas earned the badge First idea 💡

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more

Powered by Gainsight

Community Terms of Use & Conditions Cookie settings

Sign up

Already have an account? Login

For Ataccama Customers and Partners

or

Username *

E-mail address *

Company

(Private)

Only you and moderators can see this information

*

Job Title *

A bit about yourself

How did you hear about the Data People Community? *

Ataccama version

Password *

I accept the terms & conditions

loginBox.register.email_repeat

Login to the Ataccama Community

No account yet? Create an account

For Ataccama Customers and Partners

or

Username or Email

Password

Remember me

Forgot password?

Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.

Enter your e-mail address

Back to overview

Scanning file for viruses.

Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.

OK

This file cannot be downloaded

Sorry, our virus scanner detected that this file isn't safe to download.

OK

Lead your team forward OCT 24 / 9AM ET

×