Skip to main content

Data Quality & Catalog ⚙️

Discuss, learn, ask, suggest, share all things DQ&C 🔍

  • 555 Topics
  • 1,356 Replies
555 Topics
Catherine
Data Pioneer
CatherineData Pioneer
posted in Data Quality & Catalog ⚙️
How to detect if data source (at-rest) is encryption or not based on sensitivity classification?

Based on data sensitivity classification of the data source, we would like to validate if the data is encrypted or in clear.  If the classification is confidential and the data is in the clear (unecrypted) we want to generate a report for the data custodian.  Requirements:  workflow, scheduler, alert notification  

320
Catherine
Data Pioneer
2 years ago
Maxim Kim
Ataccamer
Maxim KimAtaccamer
posted in Data Quality & Catalog ⚙️
Computed Content: Show attribute values based on existing attributes

INTROSometimes you need to enrich your entity view with the combination of attribute values or attributes that belong to some other entity… and there is a way to do it using computed content feature.Following are the two use cases I happen to encounter most often. Show parent attribute in a child entityYou have report and reportColumn entities. Entity reportColumn is an array of embedded objects inside report. Entity report has a reportName attribute.The goal is to add report.reportName to reportColumn entity as reportName attribute. High level steps:Create a new entity named reportNameCC, save it, **DO NOT PUBLISH**. In the Computed Content tab of the entity add parameters and SQL. Edit reportColumn entity  add embedded object with type reportNameCC;  add delegate property to embedded reportNameCC.  Computed Content EntityGoto Organization → Metadata Model Press Create entity and create empty reportNameCC entity. Save but **DO NOT PUBLISH!** Open Computed Content tab and fill in: Inp

1410
Maxim Kim
Ataccamer
2 years ago
A
Data Voyager
AbbyData Voyager
asked in Data Quality & Catalog ⚙️
Flow of columns after using group Aggregator

Hello All,I am using group aggregator in our plan but I am not sure how to make all the columns flow from input till the end/output through group aggregator. 

1041
alexAguilarMx
Data Pioneer
2 years ago
K
Data Pioneer
KarthikaSPillaiData Pioneer
asked in Data Quality & Catalog ⚙️
ADLS Profiling Failed

I tried to profile flat file in ADLS but profiling failed with the below error :    {"status":"FAILURE"}Detail: org/apache/hadoop/conf/Configurationorg.apache.hadoop.conf.Configuration Please help

592
anna.spakova
Ataccamer
2 years ago
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
The merge of data - pattern for merge

Hi,Have gotten some good answers here. So here is a case. It is a requirements to merge data.This involves steps:Compare to sets of data - say Target and Source. Any record not in Target but in Source will be added (insert). Any record in Target but not in Source will be flagged ‘deleted’ or deleted (delete) Any record in both sets will be updated - say a timestamp will be updated (update)In SQL it is known as MERGE.One way would be an full outer join and then use filters, with three branches in the plan, but I hope I miss an easier way ;) 

961
BrianF
Ataccamer
2 years ago
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Profiling new data without discarding unmodified data

Hi all!There is a table in the source that contains about 100 million records and is updated every day. Is there any way to profile the table so that it adds only new data (for example, those that have appeared today) while not profiling all 100 million rows, but conditionally adding 1 million new records to the 99 million existing ones?

812
F
Data Pioneer
2 years ago
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Сomplicated version of left join

Hello,How to write the following condition in Ataccama Desktop (Join step):left join anl.productsjur p on (asd.creditpurpose = p.cp or coalesce(asd.creditpurpose, p.cp) is null)                        and (asd.creditpurpose2 = p.cp2 or coalesce(asd.creditpurpose2, p.cp2) is null)                        and (asd.creditobject = p.co or coalesce(asd.creditobject, p.co) is null)The difficulty is that in the join step only one field for left key and right key is available for selection..

1142
F
Data Pioneer
2 years ago
F
Data Pioneer
FCB developerData Pioneer
posted in Data Quality & Catalog ⚙️
Database connection issue with scheduler in ONE Runtime Server Admin

Hi!Just started to learn how to work with Scheduler and Workflows and ran into an error when starting workflow through scheduler:ERROR: Data source 'GP' specified by property dataSourceName must exist. [Jdbc Reader (Jdbc Reader, offending property path=/dataSourceName)]The error occurs in the "run build file" task. This is a normal plan containing a database connection. Because of this, all other tasks remain with the "WAITING" status. I will attach the configuration of my .sch file:<?xml version='1.0' encoding='UTF-8'?><scheduleDefinition><description>test</description><enabled>true</enabled><job class="com.ataccama.adt.scheduler.job.WorkflowJob"><workflow>WF00:missed_products_flow_test.ewf</workflow></job><scheduling>40 16 * * *</scheduling></scheduleDefinition>Can you please tell me what I could be missing? Where to write database connection settings in .sch configuration? I just created a workflow, .sch w

1542
F
Data Pioneer
2 years ago
M
Data Pioneer
mahesh ArData Pioneer
posted in Data Quality & Catalog ⚙️
what is the best way to write alphanumaric check DQ rule in one web.

While doing unit testing i can’t exclude special characters in Alphanumeric check.Please let me know the best way to do it.

330
M
Data Pioneer
2 years ago
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
How can we check decimal value in ONE WEB ?

How can we do decimal validation in Ataccama one web DQ rules. input attribute data type should be float for this rule.  

644
M
Data Pioneer
2 years ago
N
Data Voyager
NarehhData Voyager
asked in Data Quality & Catalog ⚙️
JSON Call example

HiI need an example of using JSON Call.I try a simple REST API call, but it does not work.Thanks 

2262
W
Ataccamer
2 years ago
P
Data Voyager
PCappData Voyager
posted in Data Quality & Catalog ⚙️
How do I dynamically feed a "Find" regular expression ?

 

380
P
Data Voyager
2 years ago
K
Data Pioneer
KarthikaSPillaiData Pioneer
asked in Data Quality & Catalog ⚙️
Couldn't run DQ evaluation

 I have created Business term employee id and detection and DQ rules and linked the rule to business term wherever possible but when I tried to evaluate the following error displayed always: Please help! Couldn't run DQ evaluationNo matching rules found for term attributes.

647
K
Data Pioneer
2 years ago
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
Case statement in Ataccama one web DQ rules

How can we write case statements in Ataccama one web DQ rules?As of now we are writing multiple conditions with distinct invalid statements. I am looking for a better approach rather than going the traditional way. Thanks,Mahesh

821
M
Data Pioneer L2
2 years ago
K
Data Pioneer
KarthikaSPillaiData Pioneer
asked in Data Quality & Catalog ⚙️
Need Sample Look up file

@anna.spakova @Catherine I need some sample look up file so I can understand the format of the look up file. Could you please provide. Thanks!

536
M
Data Pioneer L2
2 years ago
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Catalog Item Reader : "cannot retrieve metadata node"

Hi all!Please tell me where to look to cope with the error "Cannot submit job: Cannot execute task 'preprocess job' for job xxx. Caused by: Cannot retrieve metadata node" when starting remote plan execution. When the plan is run, it goes through the SUBMITTED and PREPROCESSING stages and then ends up in the FAILURE stage with the above error.For clarification: my account has full admin rights in ONE Web, so I have access to the catalog item.

1297
S
Ataccamer
2 years ago
M
Data Voyager
mukeshData Voyager
asked in Data Quality & Catalog ⚙️
sorting issues if we use record descriptor builder

simply I use here two columns date and kpi and sorting of date its fine but if we use record descriptor builder, its missing the date wise sorting.out: Steps:Sort → have date column sorting.Alter→ have rd1 <String> and record_counter<Interger> Column additionRecord descriptor builder → have rd1 binding with date and kpi column  

431
M
Data Voyager
2 years ago
Catherine
Data Pioneer
CatherineData Pioneer
asked in Data Quality & Catalog ⚙️
Mongo DB Data Source Connectivity

Is Mongo DB data source supported to connect and profile?

821
S
Ataccamer
2 years ago
V
Data Voyager
vickylau32Data Voyager
asked in Data Quality & Catalog ⚙️
How to add now column on term level page?

I would like to add new column e.g. Description to the column bar, next to Abbreviation, but not familiar how to achieve it?  current layout syntax 

1903
F
Data Pioneer
2 years ago
V
Data Voyager
vickylau32Data Voyager
asked in Data Quality & Catalog ⚙️
How to indicate data quality score on sub category and category terms?

Hi there, From the official website, I  found the business terms could show data quality score on from category to element term level.  I wonder how the sub-category item do an aggregation data quality score based on the element(column items). I understand that data quality rule could apply on column level or table level to show the quality result.  For our current environment, I’ve set the hierarchy relationship, Data Domain is parent relationship to Data set, Data Set is the Parent relationship to Tables, which looks like Data Domain > Data Set > Data Table. How can it show data quality score on Data Set(Sub-category) and Data Domain level(Category).   Thank you! Best regards,Vicky  

944
F
Data Pioneer
2 years ago
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
How to get the metadata for a post-processing integration input?

Hi Experts,Maybe I’m missing something here. I’d like to get the name and id of the monitoring project, catalog item in a post process, but the integration input doesn’t provide those, right? (Why nut? Or am I missing some system pars I could use here?) So how can I get these quickly?Thank you! 

1395
M
Star Blazer L2
2 years ago
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Directory with large values

Hello. everyone! Has anyone encountered a problem with large directory(example: 150 values). How did you solve this issue and make clear to this directory easy in the Ataccama ONE itself, without using other external softwares, services? 

462
F
Data Pioneer
2 years ago
B
Data Voyager
bartlomiej.konopkaData Voyager
asked in Data Quality & Catalog ⚙️
Two data sources for one Catalog Item

Hi, is it possible to use data from two data sources in one catalog item? Is there an instruction regarding such task? Let’s say we have data in ERP and separate data in MDM tool, which are not the part of the ERP itself, and we want to check if data in both are consistent for same records. We’ve heard, that it should be done by converting catalog items in IDE (at least for Gen1), but I’m not sure how should it be working. 

1295
S
Data Pioneer
2 years ago
anna.spakova
Ataccamer
anna.spakovaAtaccamer
published in Data Quality & Catalog ⚙️
valueList:property: Trait with a great power

Did you ever use valueList:property trait? No? Maybe this article will change your mind.What are traitsFirst things first. What are traits? Traits are used to enable additional configuration on our entities in the metadata model. They are either property traits (applied to properties) or entity traits (added to entities). Example can be a trait called fe:term that allows to give a term subtype a different color. Link to the full list of traits as well as how to apply them can be found at the end of this article.Why to use valueList:property traitNow, what is valueList:property trait? It is a trait that will (simply put) make properties look like terms. It will change the design from the underlined text or list of underlined entities to small boxes. While it adds more to the design of the layout, it can also be very beneficial for arrays of embedded objects. If you have many values for this type of property, Ataccama will show only first 10 values and additional will be on other pages.

1495
Catherine
Data Pioneer
2 years ago
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
Is there an expression to verify that a list of EMPNAME contains only characters in Ataccama one web 13.6 version ? like REG_MATCH(column,[a-zA-Z])

i found below string fucntion. in help document but it will check [A-Za-z]  pattern.string matches(string srcRegex, string srcStr [, string caseInsensitive=false]) 

872
M
Data Pioneer
2 years ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings