Data Quality (DQ)

Discuss, learn, ask, suggest, share 🔍

  • 30 Topics
  • 38 Replies

30 Topics

A
alinAtaccamer
published in Data Quality (DQ)

Data Quality and Pipeline Execution

Hello, here is a short video to illustrate how an organization might use Ataccama Data Quality to run checks as part of an existing data pipeline, and verify data integrity before further processing. With Ataccama, you can use various data warehouses or ETL tools, and GraphQL API integration allows conditional scheduling and more complex orchestration.In this demonstration, we use DataBricks Notebook to trigger DQ execution and retrieve the results. These determine whether to halt or commence the pipeline, and in-app dashboards display the metrics. All results and metadata can also be exported to a data mart table for further analysis and visualization.The platform is designed to provide visible and usable data quality insights, and we offer flexible integration capabilities to suit any data stack. Thanks for watching. 

300
A
15 days ago
A
aiwaliaRocket Pioneer
asked in Data Quality (DQ)

Divide results from Invalid Rule column in Monitoring Project Report Export into multiple columns

The idea is to divide the invalid_rules column as obtained in export for Monitoring Project into two columns- one containing the rule name and the other containing the explanation; and implement this for all the rules mentioned in a cell.For example, The below cell would be divided into invalid_rule_name column containing two records-validation [USA] State and calidation[Noth America] Phone Numbers, and invalid_rule_explanation column containing explanation for the respective rules.The challenge I am facing here is that the rule ID mentioned in Monitoring Project report export is not the same as the rule ID mentioned in the list of rules I export; which I am trying to use to identify the rule in invalid_rules column from Monitoring Project report.Please help me find a solution for this

2
A
8 days ago
E
Evgeny ChernoskutovSpace Explorer
asked in Data Quality (DQ)

How to increase the speed of profiling data for using Oracle DB?

Hello, I have a speed of profiling data for using Oracle DB of about 4000-7000 rows per minute.I use Ataccama DQ Analyzer (Version: 11.1.1.ga-2017-06-14).  Please, can you take me to advise me on how to increase the speed of profiling data for using Oracle DB?I hope to take the profiling speed to about 100 000 rows per minute...  Connation config:   Log:17.03.2023 12:56:04 [INFO] Switching to logging setup from C:\Users\ECHERN~1\AppData\Local\Temp\loggingConfig3307204922660014311.tmp17.03.2023 12:56:04 [INFO] Using following licenses:17.03.2023 12:56:04 [INFO] C:\Ataccama DQ Analyzer 11\runtime\license_keys\license.plf17.03.2023 12:56:04 [INFO] Using DQ Analyzer configuration file C:\Ataccama DQ Analyzer 11\workspace\sqb_prof_pilot\data\out\sqb_all_table.plan17.03.2023 12:56:04 [INFO] Using runtime configuration file C:\Users\echernoskutov\AppData\Local\Temp\runtimeCfg1065525328424309714.tmp17.03.2023 12:56:04 [INFO] Changing CPU affinity to 4 CPU(s).17.03.202

2
E
10 days ago
S
smunotSpace Explorer
asked in Data Quality (DQ)

MCQs in Certification

What kind of MCQs are asked in Certification exam?

1
DannyRyan
13 days ago
S
sgillaRocket Pioneer
asked in Data Quality (DQ)

NULL Valued fields on DQS

How are NULL’s considered? I have a case statement which gives the right value matching measure -used debug to test. But when it comes to matching rules, it skips the rule. Example:Name      NPIRamona  12345678NULL       12345678the 2nd row should match the 1st on NPI value and considering on of them is null.Even the scenarioName      NPINULL       12345678NULL       12345678these two should match together based on NPI. I do not see a special character. If I replace NULL with a value ‘A’ , they match with rule Matching on NPI. but does not work for NULL.  Any thoughts or suggestions appreciated. 

6
S
15 days ago
J
JerinSpace Explorer
asked in Data Quality (DQ)

Results message not showing as message.

Hello, I’ve got a question while using DQ evaluation rules.I have been creating rules with valid and invalid messages but just the invalid message at the end does not display as intended. But when I view this in the results tab it just says OTHER as the explanation.    Is there a workaround to this problem? Should I edit my rule logic to include some specific condition and leave the rest to fall under the ‘IF none of the conditions above apply THEN’ TIA for the replies.  

2
J
15 days ago
E
EllieSpace Explorer II
asked in Data Quality (DQ)

Export profiling result

Hi Team, Is there anyway that we can export the result of profiling in ataccama? Thanks, Ellie

1
O
18 days ago
S
sgillaRocket Pioneer
asked in Data Quality (DQ)

subject_matching.online: Error message: Online configuration is not valid!

During bundle deployment I see this error. I have tried different combination ofUpdate to old bundle. Manually update the component again on the existing old bundle and upload Copying extended unification Updating OmnigenServer folders (bundle,cmd,dbms) with the backup. restarting services as needed.Any thoughts or suggestions? Thank you.

3
S
26 days ago
S
sgillaRocket Pioneer
asked in Data Quality (DQ)

DQC match - slow incremental load

The matching component is slow at the extended unifications step for a incremental load. When a record like ‘WALGREEN%’ is loaded and there are 20K records with similar name on the instance, the records are all considered through the process of matching. Hence the process is slow when processing the increment and even if the incremental load has 11 records to process 30K records have the transaction Id and omni_modified date updated with the work order related on the instance table.  When I check history, nothing changes on these records except the omni_modified_date and transaction_id to that of the work order. Tweaking the union keys to create combination of name and key factors or adding separate groups for the high count records is not helping. Any thoughts or suggestions?

6
S
29 days ago
Maxim Kim
Maxim KimAtaccamer
posted in Data Quality (DQ)

Export Data Quality results

Eventually, you might need all your data quality results evaluated in One monitoring projects or within regular catalog items (via business terms and related DQ rules) exported to some reporting system or into excel. And it would be nice if it looked like this:  TLDR; There are 2 components attached you can use to export DQ results, one for monitoring projects, another one for regular catalog items. Monitoring ProjectsThe idea is to read all monitoring projects metadata, join it with their check results as well as rules names, catalog item names and voilà, we can save it to excel/csv or to some database table.It sounds like something very simple, and indeed there is no black magic, but nonetheless component looks a bit more elaborated: Catalog Items DQ (based on Term evaluations)The idea is to find all catalog items that have aggregated overall data quality and join them with the DQ rules results:This component has a bit simpler implementation:  Of course, the output of the components

1
M
1 month ago
K
kriangsaksumthongSpace Explorer
asked in Data Quality (DQ)

importing DQ rules fail.

I’m having this import error note, how to fix this? thank you.  

1
K
2 months ago
M
matthewtaylorukAtaccamer
posted in Data Quality (DQ)

Data Quality and Integration Scenarios

IntroductionThis post was created to answer the following questions:When moving data through data pipelines, how can the Ataccama ONE DQ platform support Data Integration platforms other than its own. How do other Integration Platforms access the DQ rules? How do other Integration Platforms flag records for Data Stewards to review?DQ platform in context of integration processesThere are three key patterns that are typically used for integration of the ETL process with Data Quality:Option 1The integration platform calls Ataccama DQ rules exposed as real-time API, sending in batches or micro-batches of data for evaluation.Option 2The Integration platform loads data into a staging/landing layer and triggers (via API call) a batch process in Ataccama, Ataccama evaluates/improves the data and populates the cleansed layer in the data lake, including DQ scores and flags. Option 3Ataccama can be connected to a data stream, such as Kafka, and on the fly evaluating data in the stream - every mes

0
M
2 months ago
K
KarthikaSPillaiRocket Pioneer
asked in Data Quality (DQ)

do we have row level quality score?

I see column level quality score in monitoring project. Do we have row level quality score?

5
anna.spakova
2 months ago
O
OjaswiniRocket Pioneer II
asked in Data Quality (DQ)

DQ Monitoring project UAT/Test deployment

Hello,I am new to DQ and Ataccama. Our code is in development environment and soon to be deployed in Test/UAT for UAT testing and later to Prod once we get UAT signoff.My question is what is the way to export DQ monitoring project and its related components (like rules, etc) from Dev environment to UAT/Test environment? Is there any export capability? I have also heard that we may need to recreate it again in UAT environment. Really appreciate any direction in this.Thanks in advance.

2
O
2 months ago
K
KarthikaSPillaiRocket Pioneer
asked in Data Quality (DQ)

Need DQ Rule

How to Check whether the target column has only DATE values or not.?

4
S
6 months ago
Adela Sotonova
Adela SotonovaAtaccamer
published in Data Quality (DQ)

Automate the Delivery of High Quality DataWhitepaper

What makes manual data quality so hard to scale in large organization? Is there a step by step approach to automating data quality? Why is an automated DQM system the future of enterprise data quality?You can find an answer to all these questions (and many more!) in our latest whitepaper.👉 Download it here

320
Adela Sotonova
6 months ago
S
sd_2000Space Explorer
asked in Data Quality (DQ)

Custom Data quality check

Can we create custom check on data as per our need and get stats out of it ??for example.. I want to check whether colA has values between list [‘A’,’AB’,’O’,’B’] and stats about the values that are present in data.

1
Adela Sotonova
7 months ago
Adela Sotonova
Adela SotonovaAtaccamer
published in Data Quality (DQ)

State of Data Quality report is here!Report

Our new State of Data Quality report is HERE!  We've partnered with TDWI to survey 1,000+ executives and business users and understand key factors that help today's companies succeed with data quality.Access key findings from this year’s survey in the e-book, including: The current state of enterprise data quality management in 2022 Common challenges organizations encounter when managing their data quality What’s required to establish a mature enterprise DQ management practice Recommendations from TDWI analysts for data management professionals... and more! Download the free report here: https://www.ataccama.com/ebook/2022-state-of-data-quality

200
Adela Sotonova
8 months ago
N
nathaniel.reid-smithAtaccamer
published in Data Quality (DQ)

Reorder Columns in ONE ListingTutorial

Did you know that you can reorder or add custom columns to Listings within ONE? Here is an example where we move the validity score to be the second column. You could also disable various columns here if you would like. To do this, you would add the following in Debugging Tools, for more information, please see our documentation. { "_type": "entity.page.listing", "children": { "_type": "relationships.hierarchyListing", "enableSelection": true, "columns": { "name":{ "name": "Name" }, "dqEvalTermAggr": { "name": "Overall Validity" }, "type": { "dataPath": [], "name": "Type", "after": "dqEvalTermAggr", "renderer": { "_type": "glossary.termType.tableCell" } }, "abbreviation":{ "name":"Abbreviation", "after":"type" } } }} 

601
S
1 year ago
DannyRyan
DannyRyanCommunity Manager
published in Data Quality (DQ)

How to monitor the current execution of a plan?Tutorial

DQA offers a way to monitor the processing visually, providing counts of processed records, percentages on completion of the different operations (steps) and visual cues to the flow of data to identify bottlenecks. This can be achieved by opening a special view of any plan being executed, which will mirror the original plan, showing the processing of records and other aforementioned visual features.To open the monitoring, click the Show progress button, which appears in the Console when the plan is being executed.You can also configure this monitoring window to be opened automatically whenever a plan is executed. The default configuration for the monitoring can be changed by:Open Window > Preferences Navigate to Ataccama DQ Analyzer > Launching > Progress Viewer. Change Automatically open progress viewer for launched plans to Always. You can also change how the Progress Viewer should behave once the execution finishes. 

190
DannyRyan
1 year ago
DannyRyan
DannyRyanCommunity Manager
published in Data Quality (DQ)

Is it possible to automate table profile creation?F.A.Q.

Table profile creation can be automated after buying the more advanced Data Quality Center (DQC) software.Schedule a call | Ataccama On the other hand, with Data Quality Analyzer (DQA, free software), the automation is not possible.  

200
DannyRyan
1 year ago
DannyRyan
DannyRyanCommunity Manager
published in Data Quality (DQ)

How to configure the temporary folder used by DQA (permissions error)?F.A.Q.

You might experience problems when installing DQA on a computer with limited permissions or high restrictions. Even if the installation is done in a folder where you have rights for installation, DQA might fail to execute plans. The reason is that DQA makes use of a temporary folder (usually in the path to the user cache folder) for placing instances of executed plans. This folder is the default Java temporary folder. The execution might throw an exception due to being unable to create a temp plan file.You can override the path to this folder by passing its chosen value as part of the special environment variable JAVA_OPTS , which serves as the Java Virtual Machine arguments. This should be created as follows:Open for editing the dqa.ini file placed in the root folder of the DQA installation, using an editor sensitive to End Of Lines (EOLs) in Windows, WordPad is an option, Notepad++ is another option. Add the variable java.io.tempdir . The variable must be passed using the -D notation

160
DannyRyan
1 year ago
DannyRyan
DannyRyanCommunity Manager
published in Data Quality (DQ)

How to reserve more memory for the DQA IDE?Best Practice

If you experience out of memory errors when processing a large dataset. You can increase the amount of system memory that is available to the Ataccama DQA application.Open dqa.ini located in the root folder of the DQA installation. This file contains the JAVA_OPTS that will be used by the virtual machine on start.Edit Xmx and substitute the number (256 by default) by a higher number for reserving more memory, e.g. -Xmx1024 

190
DannyRyan
1 year ago
DannyRyan
DannyRyanCommunity Manager
published in Data Quality (DQ)

How to reserve more memory for executing plans?Best Practice

DQA plans store data while processing it. This means the more input data there and the more the plan logic multiplies this data to produce desired outputs, the more memory needs to be reserved it.Some of the most memory-intensive steps are Profiling and JDBC Reader.There steps to change the default settings are:Open Window > Preferences > Ataccama DQ Analyzer > Launching. Change the value of the Default memory for launch configuration (MB) to 1024, which is recommended for modern for modern machines. 

170
DannyRyan
1 year ago
DannyRyan
DannyRyanCommunity Manager
published in Data Quality (DQ)

DQ Analyzer for Windows can help you gain valuable insights into your data.Tutorial

Did you know our DQ Analyser for Windows (32 or 64bit) can help you to gain valuable insights into your data, and the application is free to use.Visit https://www.ataccama.com/download/dq-analyzer Some of the features include;Duplicate count Pattern & mask analysis Business rule analysis Numeric statistics Ability to drill down into the data on a per record basis Foreign key analysis Dependency analysis Primary key analysis Concerned about duplicate values in your datasets?Frequency Analysis can tell us that we have a value ‘SYSTEM MIGRATION’ in the customer name column 4551 times. It is clear that these customer names were not transferred into the current system correct - supposedly during previous migration activity.One of my personal favorites for the identification of duplicate records is Group Frequency analysis.Here we can see the same value of 4551 as a group size, with a group count of 1. We can interpret the group analysis as there is 1 value that appears 4551 times.Or usi

270
DannyRyan
1 year ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Terms & ConditionsCookie settings