Skip to main content

Data Quality & Data Governance ⚙️

Discuss, learn, ask, suggest, share all things DQ and DG 🔍

  • 517 Topics
  • 1,234 Replies
517 Topics
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
posted in Data Quality & Data Governance ⚙️
Business glossary - adding relationship types (part 2)

Business glossary - relationship types (part 2)Hi Community,In my previous post I presented the idea to implement new metadata entities for attribute and entity terms in the glossary. The reason for this is that it makes most sense to have more types of terms that best describe either a catalog item attribute (the attribute term) or a catalog item (the entity term).I finished the post with making a comparison to logical data modelling.In a logical data model the entity is described by its properties, so the attributes. An entity not only has descriptive attributes, but obviously also one or more attributes that identify a record of that entity, the primary key. Besides the primary key also an alternate key may exist (for instance an email address can be used as alternate key next to the client id). Or an attribute can act as foreign key to another entity. Such an attribute will then also exist in its parent entity.With the example of an attribute existing both as foreign key in one ent

0
Albert de Ruiter
Star Blazer L2
1 hour ago
maykwok_hamilton
Data Pioneer L2
maykwok_hamiltonData Pioneer L2
asked in Data Quality & Data Governance ⚙️
Runtime server / ONE Desktop connection to Azure Keyvault

I have a use case where I want to store some client id and secret in my own company’s Azure keyvault.Then I have a component which needs to use the client id and secret to authenticate against a 3rd party, so I can issue a GET call to get some data. So I am thinking:Runtime server authenticate against keyvault Runtime server issue a “get secret” call to keyvault Keyvault sends back the secret Runtime server takes the secret and authenticate against 3rd party Runtime server issue a “get data” call to 3rd party 3rd party sends me back the actual data What authentication methods from runtime server to keyvault are supported? I had a look at Azure keyvault’s documentation but I don’t think I understand it fully. https://learn.microsoft.com/en-us/azure/key-vault/general/authentication Has anybody else done it? What’s your experience?

5
maykwok_hamilton
Data Pioneer L2
1 day ago
J
Data Voyager
jcarrData Voyager
asked in Data Quality & Data Governance ⚙️
How does Ataccama interpret duplicate attributes if they have different meanings to different terms?

Good Morning, I have a quick question in regards to loading data and how it appears into Ataccama from a database. I did not see this topic addressed but I wanted to reach out and ask this and see if this question made sense.In the Knowledge Catalog, if terms are “table names” and attributes are “columns” as described in the documentation. How does Ataccama interpret duplicate attributes spread across separate tables of data? For example, how would Ataccama interpret two different tables that have duplicate titled columns within those tables? Especially if those column headers have different meanings across the tables. If any further explanation is needed feel free to send me a message.

2
A
Ataccamer
1 day ago
H
Data Voyager
hbairiData Voyager
asked in Data Quality & Data Governance ⚙️
Need an automation to map Active Directory groups to access to groups and monitoring projects on Ataccama UI

Need an automation to map Active Directory groups to access to groups and monitoring projects on Ataccama UI. 

3
maykwok_hamilton
Data Pioneer L2
1 day ago
M
Data Enthusiast
manikanta vData Enthusiast
asked in Data Quality & Data Governance ⚙️
How to Implement Logic in Ataccama to Process One Record and Flag Another as Error for Duplicate Correlation ID and Version Number

Two records as same correation id and same version number one should process another one should error out how to implement this logic in ataccama 

9
S
Ataccamer
1 day ago
M
Data Enthusiast
mbiteyeData Enthusiast
asked in Data Quality & Data Governance ⚙️
Monitoring Project Specific Notifications - Separated

I have been testing rule specific notifications within monitoring projects and it seems to add those notifications to the bottom part of the general notifications, but we are trying to see if we can receive separate emails due to different users needing to see specific issues.The documentation link is shown in the link below,https://docs.ataccama.com/one/latest/monitoring-projects/results-reports-notifications.html#specific-notifications  

0
M
Data Enthusiast
1 day ago
D
Data Voyager
DhavalMData Voyager
asked in Data Quality & Data Governance ⚙️
How to Transpose columns in a plan?

I have been given a list of emails that I want to transpose—converting the data from rows to separate columns—and then concatenate these columns into a single row. How can I do that in the IDE? Original Data After Transpose Final Output Note: All the emails will be unique, so I don’t want them to be grouped. Could you please suggest how I can transpose these rows into columns and later merge them into one single row?Thank you.Regards,Dhaval

2
Cansu
Community Manager
2 days ago
E
Space Explorer
EmmanuelFSpace Explorer
asked in Data Quality & Data Governance ⚙️
How to add a column from an csv file to the "To" section of the SendMail in workflow?

We are trying to pass a csv file’s column which is generated from the Run DQC and contains a list of emails to the Send Mail’s “to” section. What are the other ways of passing a column to that particular section. Let me know if there are any alternative steps. This is the workflow planConfiguration of Send Mail:Regards,Emmanuel

5
E
Space Explorer
6 days ago
K
Data Pioneer
KarthikeyanData Pioneer
asked in Data Quality & Data Governance ⚙️
Compare data between file and table

we need to achieve completeness DQ control by comparing data in the file and the table irrespective of the position of primary key in either of them or by using composite keys. - This is in continuation with the query posted in ataccama community - Regarding DQ controls (ataccama.com) Do we have any step or document to follow on how we can import that file into ONE web app as catalog item and can we upload the file from local or we need to get that from the file system? Thanks,Karthikeyan.

6
K
Data Pioneer
6 days ago
M
Space Explorer
MelSpace Explorer
asked in Data Quality & Data Governance ⚙️
Duplicate detection with aggregation results config

I have created a rule that detects if a record is in the table more than once based on aggregation of 2 attributes in OneWeb application.  This gives a proper result in Invalid samples data and exports giving 2 errors if the record is in the table 2 times etc.  But the customer requires that only 1 record should be visible in Invalid samples and also in exporting to db.  It does not matter which one.  Is it possible to configure the result to do this?

3
J
Ataccamer
6 days ago
S
Ataccamer
srija piratlaAtaccamer
published in Data Quality & Data Governance ⚙️
How to use Dynamic Expression Assigner?

Hi I'm Srija, Solutions Consultant here at Ataccama.In this post, I’ll walk you through how to use a dynamic expression assigner. What is a dynamic expression assigner?The Dynamic Expression Assigner in Ataccama allows you to define dynamic expressions or rules to determine how values are extracted or assigned to specific attributes based on conditions present in your data. Use Case:The Monitoring project results will give the attributes , Invalid DQ rules and Invalid DQ Attributes for the failed rows. If we need to extract the DQ Error Attribute Value based on the Invalid DQ Attribute then Dynamic Expression Assigner can be used in the post processing plan.In the below example, I have a DQ_Error_Attribute which is causing the issue for that record. If I need to extract the respective value of the DQ_Error_Attribute then, I can use the Dynamic Expression Assigner. We can achieve this using a column assigner, but we must hardcode the attributes to extract the value. If any new attribute

772
S
Ataccamer
6 days ago
K
Data Pioneer
KarthikeyanData Pioneer
asked in Data Quality & Data Governance ⚙️
data reconciliation for snowflake data

Hi Team This is the continuation of the of the below thread about using reconciliation project for comparing a file with the table.Compare data between file and table (ataccama.com) The data compare is working fine when I followed the approach suggested.  But from this link (Data Reconciliation Best Practices 👾 (ataccama.com) - it is mentioned as (except snowflake data).So, what will be the approach for the snowflake related connections. I have one more question - for eg ., I have column fullName in source (which has a full name) in target I have two columns one has the first name and other column has the second name. Can we compare source column to target column1 and 2.  

0
K
Data Pioneer
7 days ago
kayhan.turker
Ataccamer
kayhan.turkerAtaccamer
published in Data Quality & Data Governance ⚙️
Passing a dynamic query into a JDBC reader

Hi community!Today, I have a quick tip to share on how to pass a dynamic query into a JDBC reader. If you are using ONE Desktop as part of your day-to-day this would be a good practice to implement when it comes to using the JDBC reader step.OverviewThe following is a working example of passing a dynamic query into a JDBC reader. Its overall structure consists of two plans that are run sequentially in a workflow. The idea is that the first plan will produce the query that the second plan (that includes our main logic) will use.In the above example, an audit table is read in the second DQC plan, but with a query that depends on information on the current state of the table (for example, the maximum of a particular record, record count, and latest refresh date). First Plan - Retrieving QueriesIn the first plan, functions and variables are applied to produce a .sql file that contains the JDBC reader query to be used.As an example, functions can include getting the current date to apply a

541
V
Data Voyager
9 days ago
S
Data Pioneer
sumeyraData Pioneer
asked in Data Quality & Data Governance ⚙️
DQ rule using complex lookup

Hello,I was wondering if someone has an idea to help my question about one of my DQ check. Basically let me explain what i have and what i want to achieve with them.I have; one list consisting 2 columns. Column A has unique values having 2 or 3 digits with a mixed combination of letters and numbers. Column B has names according to the column A. Here is the example: Column A Column B abc name1 abd name1 abe name1 abf name2 xy name3 1a1 name4 1a2 name4 1a3 name4 yz name5 yz1 name6  As you see, there is no corelation between column A and column B. I can’t assume if the value of column A is yz and value of column B is name5 then i have name5 for yz1 of column A.And I have a field in my data lets call it as column C. Column C has values with a mixed combination of digits and letters and with different length. Such as abc1231422ad , 1a3577afdg , yz4564sdfsfs can be example. And I have another field which is Column D having some other values. I want to check v

4
S
Data Pioneer
9 days ago
R
Data Voyager
Raj YalakaData Voyager
asked in Data Quality & Data Governance ⚙️
How to Add Date Parameter to output file extension

Hi, I’m trying to add the data value to the file extension in the component but I’m not able to get the expected results. I have added a new string column using alter format step (tostring(today(), 'yyyy-MM-dd'). When I pass the column as $date or ${date} in the output filename, I’m not seeing the data on the output filename. 

3
C
Data Voyager
9 days ago
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
published in Data Quality & Data Governance ⚙️
Business glossary - customization (part 1)

Hi Community,I’d like to share some thoughts about the business glossary. After considering how we wanted to use the glossary, we applied customizations in the metamodel, so we have a specific implementation. I suppose that others have a different approach to using the glossary, so it would be great if we could share and discuss our visions. I am looking forward to your feedback!Let’s start.The business glossary provides a way to describe terms that have relevance to the organization. Obvious properties like name, definition and owner describe a term. It is possible to connect catalog items and its attribute to a term, so that physical data elements becomes meaningful.The question is, can we describe these elements with the same properties? (Indeed a rhetorical question)To logically describe a catalog item attribute, besides its name and definition also properties like its (logical) datatype, an indicator to mark a ‘personal identifying information’ term and the GDPR classification.The

471
Cansu
Community Manager
9 days ago
S
Data Pioneer
sgillaData Pioneer
asked in Data Quality & Data Governance ⚙️
matching component - child tables

On a matching component if address table is mapped as shown in the screenshot. Why would the process still run for the address table for both insertion and deletion. while its not being used. How to avoid it run through it and why would it run through? it would not need a master_id associated wither as it is tied to parent by u_parent_id. 

5
Phil Holbrook
Ataccamer
12 days ago
R
Data Voyager
RadziahData Voyager
asked in Data Quality & Data Governance ⚙️
Configure Path in Attach File of Send Email Workflow

HiI have a workflow with Send Email step configured and this workflow will be pushed to orchestration server to run it. I have an attachment need to be included in the email notification generated from the workflow and realized that if I put only the file name for example RESULTS_ABC.xlsx, by default it will find it the same folder of the workflow that is being created. Let’s say if I put the RESULTS_ABC.xlsx in different folder like Data/Out, how do I configure it correctly at the Attach Files configuration as shown in the image below? I saw a discussion to use filePath variable but I am not too familiar with that. Could you please explain how can we achieve running the workflow and read the file that is stored from different folder so that it can attach in the email. 

1
maykwok_hamilton
Data Pioneer L2
12 days ago
Phil Holbrook
Ataccamer
Phil HolbrookAtaccamer
published in Data Quality & Data Governance ⚙️
Building a nested GraphQL API call in ONE Desktop

ONE Desktop provides two ways to access the ONE platform data. Usually, it is easiest to use the Metadata Reader step, but sometimes you need a more complex query, or perhaps you are developing a Json API call to be used from another application entirely.  In this case, you will want to start from a Json Call step.The plan I will describe is this: I’ve used a random record generator to “inject” a single row into the data flow to get things started because the Json Call executes once for each row on its input.The “out” output from the Json Call carries all columns from the input; and can also be used to carry connection error messages. I’ve not set that up, so I just sent all that to Trash.I’ve named each of the remaining outputs using a sequential numbered prefix.  This manipulates the automatic alphanumeric sorting of the output connections so that the ordering is more logical than a list of element names would be. I’ve also set the layout of the step to horizontal so I can read the l

4008
N
Data Voyager
12 days ago
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Data Governance ⚙️
Start Ataccama server without opening Ataccama Desktop tool

I'm currently using the "start-server.bat" file within the Ataccama Desktop tool to manually start Ataccama Server. However, if I forget to start the server, certain workflows fail to execute. I've attempted to automate server startup using Windows Task Scheduler, but it's not working as expected. Is there a reliable method to start Ataccama Server independently of the Desktop tool, perhaps through a scheduled task or another approach?

4
A
Universe Traveller
13 days ago
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Data Governance ⚙️
Can term detection rule use both data and metadata both in the same rule?

Hello: I have a use case specifically for assigning a term. If the data is a 9 digit number and if the field name is NOT one of the list i want to exclude - then that term be assigned. I also have additional field names that i want to be tagged with that term - irrespective of value with in that field.As of now - iam able to create  separate rules one that runs on metadata and other runs on real data - and put a OR condition or AND condition between the two rules. But that is not addressing my use case very well. I need one rule that can refer to both real data and metadata.  OR - i should be able to give a complex join condition with more than two rules(Rule 1 AND Rule 2) OR (Rule 3) Has anyone done something like that?

1
R
Ataccamer
13 days ago
JTH
Data Pioneer
JTH
asked in Data Quality & Data Governance ⚙️
Full Profiling results

Hi,13.9.4In the Monitoring Project there's a way to only display the data quality of the last run via Advanced settings ​​​​​Is there also such a way to do accomplish this with Full Profiling on (SQL) Catalog Items?

0
JTH
Data Pioneer
14 days ago
maykwok_hamilton
Data Pioneer L2
maykwok_hamiltonData Pioneer L2
asked in Data Quality & Data Governance ⚙️
Using runtimeconfig to encrypt OAuth 2.0 token request

I’m building an API call request, where I first need to obtain an access token, before I make my actual API call.I have tested the 2-step process using 2 separate JSON calls successfully. The message body of the OAuth request looks like this. Sensitive info redacted: scope is a required parameter. The above call successfully retrieves my token. I tried to setup this as a Generic server:But there is no available space to specify the scope parameter. When I run my plan: I get:2024-09-04 09:33:16 INFO: Validating model...2024-09-04 09:33:16 INFO: Creating runtime...2024-09-04 09:33:17 INFO: Running runtime...2024-09-04 09:33:17 [JsonCall] DEBUG: IOException when making request GET https://preprod-api.londonmarketgroup.co.uk/reference-data/acord/v2/version HTTP/1.1java.io.IOException: Cannot get credentials for request. at com.ataccama.dqc.io.http.CommonHttpClient.makeRequest(CommonHttpClient.java:351) at com.ataccama.dqc.tasks.io.json.call.JsonCall$Runtime.run(JsonCall.java:425) at com.at

0
maykwok_hamilton
Data Pioneer L2
15 days ago
maykwok_hamilton
Data Pioneer L2
maykwok_hamiltonData Pioneer L2
asked in Data Quality & Data Governance ⚙️
Attaching a client certificate in a Json Call step

I’m trying to build a GET API call to an external data provider.This provider requires me to send my GET call with the certificate.We managed to test it in POSTMAN, where I had to add the certificate in the Settings → Certificates menu: I have a .pfx file with a passphrase, and I needed to specify the host.This works on my laptop in POSTMAN. Now I’m trying to do the same call in ONE Desktop using JSON Call step.By way of googling, I’ve managed to convert my .pfx file into .cer, then I imported the .cer file into {DQC_HOME}/jre/lib/security/cacerts . Then I have restarted ONE Desktop. But when I call the API I’m still getting an error message from the other side saying my certificate is not sent. I have a suspicion that the certificate is sitting in the keystore but it is not used. Is there some way to ensure it is used? Some java parameter somewhere?

6
maykwok_hamilton
Data Pioneer L2
15 days ago
T
Data Voyager
ThomasProData Voyager
asked in Data Quality & Data Governance ⚙️
Microsoft Teams Notifications - incoming webhook is discontinued for new configurations

Hey all!Now that incoming webhooks within Microsoft Teams is discontinued (Existing configurations will work until end of 2025), is there a plan for how teams will continue to integrate with notifications from Monitoring Projects? This was a big sell to a lot of our users, and now that we are unable to use incoming webhooks for new monitoring projects, I want to see if there is a plan to have a work around, or if it will be not supported in the future.    

4
Cansu
Community Manager
20 days ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings
ataccama
arrows
Lead your team  forward  OCT 24 / 9AM ET
Register now
×