Skip to main content

Data Quality & Catalog ⚙️

Discuss, learn, ask, suggest, share all things DQ&C 🔍

  • 788 Topics
  • 2,012 Replies
788 Topics
K
Data Voyager
KostroskData Voyager
asked in Data Quality & Catalog ⚙️
Can DQ Analyzer connect to Redshift?

I was thinking that this should be possible with a hack to the postgreSQL jdbc configuration using the redshift jdbc driver. Does anyone know if this is possible or if there is another avenue?Edit: I found the connection after the install and properly configured the Redshift driver and connection settings.

411
K
Data Voyager
A
Data Voyager
ADB59Data Voyager
asked in Data Quality & Catalog ⚙️
RFC 5322 Internet Message Format incompatible with DQC

Hi,We have a problem in a workflow in DQC, with task “Send Mail”, because the generated message is incompatible with the internet message format RFC 5322.When there are at least 2 recipients, in the generated message, the header “to” is duplicate as many times as there are recipients, but the RFC 5322 standard does not allow this.How to correct this problem ?Thanks for your help.

1043
A
Data Voyager
M
Data Voyager
mukeshData Voyager
asked in Data Quality & Catalog ⚙️
How to get Aggregate functions in DQ Analyzer

I have Ataccama 12.x version and I do not see their Aggregate functionPlease help, how to get it enabled or patch it Thanks,Mukesh

471
M
Data Voyager
M
Data Pioneer L2
MayKwokData Pioneer L2
published in Data Quality & Catalog ⚙️
How to configure team based permissions in v13.8Tutorial

Full support for team based permissions in Ataccama ONE is coming in v14. However, you may need to implement team based permissions sooner than v14. This document is a guide to how you could achieve this in v13.8. BackgroundPermissions in ONE can be configured as role based or user based. In most cases, configuration by role is preferred as it will usually be a one-time set up, where the role only needs to be configured once in the beginning, then any users allocated to that role will inherit all permissions on role.The out of the box roles in ONE have some capabilities already allocated. These are auto-sharing permissions, which allows the role to have access to all items of that entity. E.g. Your role could have the capability of Rule Owner. This means the user will be automatically be owner of all rules that would ever be created on the platform.But what if your user should only be owner to rules (or any other entities) within their team / division / domain?As it stands in v13.8, th

2400
M
Data Pioneer L2
A
Anonymous
published in Data Quality & Catalog ⚙️
State of Data Quality report is here!
State of Data Quality report is here!Report

Our new State of Data Quality report is HERE!  We've partnered with TDWI to survey 1,000+ executives and business users and understand key factors that help today's companies succeed with data quality.Access key findings from this year’s survey in the e-book, including: The current state of enterprise data quality management in 2022 Common challenges organizations encounter when managing their data quality What’s required to establish a mature enterprise DQ management practice Recommendations from TDWI analysts for data management professionals... and more! Download the free report here: https://www.ataccama.com/ebook/2022-state-of-data-quality

340
A
A
Anonymous
published in Data Quality & Catalog ⚙️
How Ataccama ONE uses Snowpark data frames to automate profiling

Ataccama is known and appreciated by our customers for complex, big deployments where massive amounts of data are processed. We were one of the first to support MapReduce and Spark as native pushdown and enable distributed processing across different cluster nodes.Ataccama’s engine works with the major of enterprise data sources and processes them to provide valuable insight, such as statistics of the data, detection of domains, and data quality.To provide the best experience for our customers that use Snowflake, we developed a tighter integration between our technologies. Specifically, we use the Snowflake data processing capabilities in a pushdown manner which means that data is processed directly on Snowflake.The key advantages of this integration are:Lower data transfer costs for the client. Snowflake’s computational capabilities deliver results faster because of reduced IO operations. Customers don’t incur additional costs to maintain external systems for Big Data. Improved securi

4022
A
R
Data Voyager
RoberTallitschData Voyager
asked in Data Quality & Catalog ⚙️
Creating flat file using Ataccama One Desktop where records failed business rule

We created a number of data quality business rules that will fail records from the source file we are reading.  1.5 Million records.How do you port the failed records to a flat file?  There is no drill down in One Desktop so we are having issues getting business users approval on code without seeing the failed records?

1432
S
Data Pioneer L2
S
Data Voyager
SmrutiData Voyager
asked in Data Quality & Catalog ⚙️
error while publishing the project

I have a rule binded to 2 columns in a catalog item.By mistake i have changed the data type of those columns without unbinding the rule.Now i am unable to unbind thise rules from that column.i am getting error while saving or publishing the project : server refused sent data as invalid. Please guide me,what i will do here

371
anna.spakova
Ataccamer
V
Ataccamer
VaniAtaccamer
posted in Data Quality & Catalog ⚙️
How to tune/optimize a DQC load plan?

​How to tune/optimize a DQC load plan?In day-to-day life, the consultants may come across several performance issues of various magnitude in their load plans at some point in the projects. There are several optimization tips mentioned below for tuning the long running plans.Check the memory allocationDQC plans store the data while processing, which means that the more input data there is, the more memory DQC will require. Therefore the intensive executions can result in a memory overflow error if the default memory allocated for DQC is consumed. In such cases, an increase of memory could present a solution.Related Errors: OutOfMemoryError: Java Heap Space or GC Overhead Limit Exceeded... java.lang.OutOfMemoryError: unable to create new native thread... The memory allocation can be done in several ways,Increasing the memory of the DQC IDE(Desktop) Increasing the memory allocation to specific plan in the IDE Increasing the memory allocation for the entire project in IDE Increasing the me

5790
V
Ataccamer
A
Anonymous
published in Data Quality & Catalog ⚙️
How would you filter the terms (AQL) by a type?

If you would like to filter e.g. business terms, use:$type == 'businessTerm'Switch to Advanced filtering with AQL (Ataccama Query Language)Terms list before applying a filterFiltering resultsLink to documentation:Searching Using AQL Expressions

1500
A
A
Anonymous
published in Data Quality & Catalog ⚙️
How to Add Predefined Templates for ExpressionsF.A.Q.

Every step that allows for the input of an expression shows a button called Template.... This button can be used to pre-fill the Expression field with one from an out-of-the-box list.You can create your own expressions for common or complex repeatable operations, follow these steps:Go to Window in the top navigation menu and select Preferences Navigate the menu to Ataccama DQC and go to Expression Editor and select TemplatesChoose between Expression Templates and RegEx TemplatesSelect FolderClick on Add Template..Fill in Name, Expression and DescriptionConfirm with OK

360
A
A
Anonymous
published in Data Quality & Catalog ⚙️
Showing Column Names when Profiling Excel Files
Showing Column Names when Profiling Excel FilesF.A.Q.

Do you profile an Excel file and are you unable to see the column names? DQ Analyzer is not processing the column header and in the profiling results, you see the names of the columns as Field_0, Field_1, etc.You can achieve the desired result by editing the metadata of the file you want to profile.Find the necessary file under the File Explorer window. Right-click on it and choose the Edit Metadata option. Specify the column you want it to start with. You should be able to see that the Name of the columns will change automatically. You can verify the column type, as well. Then you can profile the file.

370
A
A
Data Voyager
ahmingData Voyager
asked in Data Quality & Catalog ⚙️
Default template

Any one can help to create this template using version 13.6?  It show the items clearly.  Our users can know the data presentation without explanation.  However, I can find this template in our current version.

1032
A
Data Voyager
A
Anonymous
published in Data Quality & Catalog ⚙️
Data Type Conversions in DQC
Data Type Conversions in DQCF.A.Q.

Data type conversions occurrences when working with databases are the natural behaviour of DQC. When reading data from a database (e.g. using JDBC Reader step), the data type is read from the database and transformed into DQC data type.DQC internally works with its own data types. When writing the data back to the database (in such case, JDBS Writer step is commonly used), data type alternation takes place. The rules for data types conversions are defined in SQL Transformation Rules. There is a possibility to change from what exact database types there will be a conversion to DQC types. To redefine the database types from which DQC will transform data into its own data types:Go to Window > Preferences Select Ataccama DQC > Database > SQL Transformation Rules Modify SQL transformation rules of the specific database product you use

690
A
A
Anonymous
published in Data Quality & Catalog ⚙️
How can I find and replace all values without opening the plans?
How can I find and replace all values without opening the plans?F.A.Q.

You can find and replace using the Search option in the upper panel of DQC.Click on Search and enter the text you would like to replace. Click on Replace, and enter the new text you wish to use as a replacement. Clicking on Preview will display a dialog showing where text was found, and a preview of the XML representation of the plans, before and after the change. You may want to uncheck some of the locations. Click OK.

460
A
J
Data Voyager
jdc122Data Voyager
asked in Data Quality & Catalog ⚙️
How best to use the Purpose function in Ataccama?

Any best practices for using the Purpose function in Ataccama One (webapp)? I see that it is recommended for monitoring projects but when I create a monitoring project I want to be able to search for that tag. Any tips for searching just based on the purpose tag? I’m very new to Ataccama so any and all tips would be appreciated!

1112
J
Data Voyager
jhoppesch
Data Voyager
jhoppeschData Voyager
asked in Data Quality & Catalog ⚙️
Operate on File: INFO Operation

I’m trying to read the file name of a file so I can write it in an email within a Workflow. I’m using the “Operate On File” task. The Operation I selected is “INFO”. I then point to the Source File. Next I’m trying to determine which variable I use to supply to the Send Email task. The Operate On File Documentation says you can use task variables and the example they give is “name”, “localName”, “size”, and “timestamp”. When I try using these variables in the Send Email task they just come across as blank in the email (variables appear to be null). Here is the body of the email I’m using: Name = ${name}localName = ${localName} Two questions:Does any one happen to know what I’m doing wrong in above?Additionally is there a way to pull out a file name within a Plan. I’m using the “Excel File Reader” step. I want to pass along the file name in the flow so I can use it later on. Thanks!

1661
F
Ataccamer
S
Data Voyager
semoData Voyager
asked in Data Quality & Catalog ⚙️
Data profiling - set attribute type

Hi, I’ve just started data profiling with Ataccama One so please bear with me.I have data with technical IDs (e.g. “000123456”, “009876543”,...) in a csv that I imported. The IDs are however interpreted as a number.How exactly can I change the attribute type to text? I have done so when importing the file: but it seems to go to numbers nonetheless.The original file has comma-separated values that are all between double quotes. Thanks for any help! edit: typo

1122
S
Data Voyager
D
Data Voyager
ddw_MH22Data Voyager
asked in Data Quality & Catalog ⚙️
catalog item not found error

I opened the on-prem AtaccamaONE environment this morning and was looking at recent (auto) profiled catalog items. A majority of them are not found - the error is “Catalog item not found” and the provided explanation is “This catalog item doesn’t exist anymore or someone has revoked your access”. Neither of these cases should be true as I am the only one who currently works in the environment and I’m the one who launched the data discovery on 3/09. This is a recent error - I have not seen this before. Any help you could provide would be appreciated.Thanks.

562
M
Ataccamer
K
Data Voyager
kumaritarData Voyager
asked in Data Quality & Catalog ⚙️
For or while loop in detection rule

How can we use functionality like ‘for’ and ‘while’ loop in detection rule field while writing/creating rules for capturing data/records.

953
A
Ataccamer
O
Data Voyager
onipledData Voyager
asked in Data Quality & Catalog ⚙️
JSON Call. Get Method dynamic parameter

Hello, I’m new with Ataccama and I have a simple problem. I want call a Rest service:https://api.bulkemailverifier.com/api/a/v1?key=xxxxxxxxxx&email=myemail.comI’m using the JSON Call step and If I put the full URL with my email it’s working, but now I want read the emails forma txt file and I don’t know how to make the url dynamic.In the step I have 3 parameters: URL: Resource  I set it as https://api.bulkemailverifier.com/api/a/v1?key=xxxxxxxxxx&email=Method: GETURL: HERE IS WHERE i DONT KNOW WHAT TYPE. I have tried a lot of combinations without successThe input file has one column with the mail: src_mailThanks 

2673
O
Data Voyager
A
Data Voyager
ashpoghoshData Voyager
asked in Data Quality & Catalog ⚙️
Copy File from remote location

How can we download files from remote path ( connected through vpn) on to local?I tried with Operate on File component but it is not working.Is there any work around for it?

471
Anna_Nebesova
Ataccamer
Luca
Data Enthusiast
LucaData Enthusiast
posted in Data Quality & Catalog ⚙️
Data Governance & why we need it

Enterprise Data Governance as the management of data across its pipeline - from production to consumption (and retirement). How to enable our “data fellows” to leverage on data to save money, save time and reduce risk?What are the key elements to success? What are your thoughts? 

1833
D
Data Voyager
A
Data Voyager
anu1237Data Voyager
asked in Data Quality & Catalog ⚙️
Rows to Columns

I was wondering if there is a way in Ataccama that I can move my rows data into columns?I’ve data as such:Element:          Dimension:                 Values:         Code: XYZ            Validity                        200               678XYZ            Completeness              250               678 YUP            Completeness            300                555 HIO             Accuracy                     400                100 I’d like it to look like:Element:          Validity:                 Completeness:        Accuracy;         Code: XYZ                 200                               250                         250               678 YUP                500                                300                        300                555 HIO                 200                                450                        400                100 Thank you!  

1171
T
Ataccamer
hoo
Data Voyager
hooData Voyager
posted in Data Quality & Catalog ⚙️
Why Pie charts are evil?

Data Stories provides a wide range of charts, it includes also Pie and Donut ones, even though I do not recommend using them, and the same applies to a Gauge chart. Let's look at the reasoning why and why I'd recommend using a (stacked) bar chart instead.Space usageIn data visualization (reporting, data storytelling) you have limited space you can use. Your task is to fill the limited space with the most value for the end viewer/user. Note, most value does not necessarily mean most charts, sometimes one curated chart is far better than five. The problem with Pie, Donut and Gauge charts is that they use more space in comparison to an alternative that provides the same value - a bar chart. Sales distribution on a Pie ChartMore space is used due to 2 reasons:Pie and Donut charts use Area to give value to a single category. Bar chars use just distance. An Area just takes more space All these charts are circular, which creates unnecessary space around the chart objectBellow find a better al

1584
hoo
Data Voyager

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings