Data Quality & Data Governance ⚙️

Discuss, learn, ask, suggest, share all things DQ and DG 🔍

  • 489 Topics
  • 1,125 Replies

489 Topics

S
Universe Traveller
sumeyraUniverse Traveller
asked in Data Quality & Data Governance ⚙️

Adding/Removing Attributes of Catalog item in Monitoring Project?

Hi all! I am working on a monitoring project. I have a catalog item consisting 130 columns. I want to create a custom catalog item selecting only few attributes that I need to have them in my monitoring project.However, based on my previous experience: Once I have created a catalog item with selected attributes. Then I had to add new attributes according to my project needs. I have run the monitoring project again with new attributes of catalog item, then I lost my monitoring project. It showed errors for all DQ rules because as I understood from error message, DQ rules need to have attribute key ids, and with adding new attributes I have changed attribute key ids. Now, again I have a huge catalog item and I want to reduce the volume with selecting few attributes but I don’t want to face with the same problem again. If you have any suggestion it would be so great for me. Thanks !

1
ivan.kozlov
Ataccamer
23 hours ago
A
Universe Traveller
apejkoUniverse Traveller
asked in Data Quality & Data Governance ⚙️

Breaking Excel Sheet into multiple files

Hi everyone,I currently have a plan that is creating a main task and populating it with sub tasks that each link to individual tables. My issue at the moment is that sometimes this creates tasks with a huge amount of sub tasks that may seem daunting for people to try to work through.Currently the tables and their attribute ids are all stored in one excel sheet, so I was wondering if it would be possible to break it into separate excel files each containing 20 rows. If this is possible I would then be able to create additional tasks rather than just grouping everything in one.Thank you for any help!

0
A
Universe Traveller
4 days ago
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Data Governance ⚙️

Day over day changes for certain data

Hello: I have a use case where when the data in a given column - for a key changes from previous day to next day - that should trigger an alert/notification For Ex: Security Ratings, and/or ESG ratings generally dont change for a given security. I have a table that has one record for each day for each security. And the ratings are in the columns of that table. Now - on a regular day - all ratings from previous day should be same for next day, and day after (until it changes). And on the day when it changes - i want to capture that security, ,what field changed - what was yesterday;s value vs todays value. The way - i would like to solve this - is create a SQLCatalog item - with previous day - and current day values combined into one dataset for each security and write scripts to compare them one by one, and report any changes. But I want to know if there is an out of box - date driven compare that can be done for this. 

0
Prasad Rani
Data Pioneer
1 month ago
G
Data Voyager
GloriaData Voyager
asked in Data Quality & Data Governance ⚙️

Concatenating fields on the IDE

I want to create another column using the filter step on the IDE. How do I concatenate foreign characters and fields with different data types into one field.Fields to concatenate is date field, special character (/), string and integer fields

4
ivan.kozlov
Ataccamer
23 hours ago
JTH
Data Pioneer
JTH
asked in Data Quality & Data Governance ⚙️

Delete dataset from report (Monitoring Project)

Hi,ONE Web > Monitoring ProjectI have a MP A which contains 7 CIs, I want to copy this MP to create MP B. For MP B I only want to use 6 out of the 7 CIs as the last CI is unnecessary. I use the import configuration (great qol btw) and for the 7th CI I use same CI.After the import configuration has been completed I delete the 7th CI from the MP. Inside the Report tab I still see that same CI appear which is not ideal. You'd think clicking the three dots besides it lets me Delete it from the page, but to my surprise that option is greyed out. How do I get rid of this CI from the Report page?​​​

2
JTH
Data Pioneer
1 day ago
U
Data Voyager
UmaData Voyager
asked in Data Quality & Data Governance ⚙️

Availability of API's in Data Quality to check results ?

Hi We are planning to use Atacamma DQ as a service and would like to use its DQ Engine to query and provide us counts. We would like to build a API call to Atacamma DQ with necessary SQL Query and connection parameters as part of API call, and would like to know if this SQL query can be executed on the Atacamma DQ and provide the results as an output back to API as a response. Please let us know the possibilities. Regards,Uma

6
R
Data Voyager
2 days ago
R
Data Voyager
Rohith1217Data Voyager
asked in Data Quality & Data Governance ⚙️

New Use case implementation in Ataccama

Hi,We have a use case to accept a request from collibra with details like JDBC URL, username, password and query to Ataccama and I need to return the results and size of the results in reponse from Ataccama.Can you please share usecase/documentation for implementing this usecase in Ataccama.Regards,Rohith

0
R
Data Voyager
2 days ago
P
Data Voyager
PrithikaData Voyager
asked in Data Quality & Data Governance ⚙️

Exporting Catalog item along with the associated terms

Hello, is it possible to export a catalog item along with its associated terms?  As you can see in the screenshot below, I set the attribute called “last_update_hostname”  as confidential temporarily for an example, but when I export it to a file, I was only able to see the data values not the terms associated with it.   

10
maykwok_hamilton
Data Pioneer L2
4 days ago
kayhan.turker
Ataccamer
kayhan.turkerAtaccamer
published in Data Quality & Data Governance ⚙️

Adding a DQ Check to a Monitoring Project with ONE API

OverviewThe following is the procedure to use a ONE API call to add a DQ check to a monitoring project in ONE. It assumes the following: that the monitoring project and DQ rule has already been created, and a catalog item has already been added to the monitoring project.The GraphQL query that will be used will be a mutation that will apply the DQ check to the provided attribute. It will take variables to indicate where this check should be applied. Where and how these variables can be acquired will be outlined as well.The last section will briefly go through how to run the queries used in this guide using Ataccama Playground. Query to Apply DQ CheckThe query used to apply the DQ check is provided in the file mutation SaveEntity_projectCatalogItemInstance.txt and the variables which will be necessary to include for this query to work, are in the mutation variables.txt file.In the variables text file, you must replace the following with the appropriate ids:“MONITORING PROJECT TABLE ID” “

190
kayhan.turker
Ataccamer
4 days ago
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Data Governance ⚙️

Did anyone customize the email notifications to include the name of the individuals in the addressing?

The email notifications starts with “Dear Collegue”… and i was wondering - is there a mail merge kind of thing - that can be done to replace the generic addressing with speicific name of the individual it is being emailed to? And - if it is a role - may be we need to get into role membership to expand into names. Has anyone done that? 

3
W
Data Pioneer
4 days ago
maykwok_hamilton
Data Pioneer L2
maykwok_hamiltonData Pioneer L2
asked in Data Quality & Data Governance ⚙️

DQC online service - override the response code?

Hi DQC online service experts,I have built an online service that writes data into RDM using Rdm importer. I put some validation logic into the component so that it would reject bad records from being written (e.g. value supplied does not exist in parent table). The rdm importer is going straight to confirm mode as we need the data to be automatically published and available. I've configured the integration output to provide the primary key and my explanation in the case that the component rejects the record. The app that is sending the data, are now saying that even if I provide an error explanation, the endpoint still provides a 200 response code. They have some in built error handling capability that they want to use, but they currently can't use it because the server responds 200. Is it possible for me to set a different response code e.g. 400, based on the data that is provided in the integration output?

2
maykwok_hamilton
Data Pioneer L2
7 days ago
W
Data Pioneer
WillohData Pioneer
asked in Data Quality & Data Governance ⚙️

Meta Data Model AD Group Integration

Hello Community! I am looking to make a bit of an adjustment to my MMM in DQG version 14.5. Specifically, I would like to add a field for AD groups, so that they are validated at an entity level. I know I can write in AD groups and add them to the entity description (see image below), but I am also looking to enforce that idea. The entity summary includes a field to list an AD group, but this does not have any sort of legitimate ties to the actual external group.Any insight on how others have approached this topic or otherwise accomplished my goal is greatly appreciated. Thank you for your time.  Willoh

3
W
Data Pioneer
9 days ago
Catherine
Data Pioneer
CatherineData Pioneer
asked in Data Quality & Data Governance ⚙️

Able to use "valueList:property:" Capability On "userManagement" metadata to expand the Properties.

Are we able to use valueList:property: functionality to expand the tags to enable the use to see the full value of the entry as there is a lot of white space. Within the metadata model - usermanagement  

1
K
Ataccamer
10 days ago
W
Data Pioneer
WillohData Pioneer
asked in Data Quality & Data Governance ⚙️

Hiding Global Settings

Hello! I am interested in hiding the global settings from users, at least some of them. I know this question has been asked before for version 13, but it was mentioned the approach is quite different in 14.5. Would appreciate any insight you all have. 

3
K
Ataccamer
11 days ago
A
Universe Traveller
apejkoUniverse Traveller
asked in Data Quality & Data Governance ⚙️

Getting Stewardship information through graphql

Hi everyone!Im not sure if this is possible, but I have not been able to find a query to find who the stewards are of a particular catalogItem. Specifically I am interested in getting the data owner and then using this in another part of my project.I do have the Attribute Id of the catalog item and my atacamma playground is working I just cant seem to find specifically what query I need the make.Thank you for any help!

2
A
Universe Traveller
11 days ago
S
Data Voyager
sriniData Voyager
asked in Data Quality & Data Governance ⚙️

Datetime conversion issues with locale and format

Hi Team,we are just trying to convert the string timestamp format values to dateTime format in the expression transformation, when we tried to use  toDateTime(<column_name>,'YYYY-MM-DD HH:mm:ss','en') returning null- can someone help.

2
ivysakh
Ataccamer
12 days ago
S
Ataccamer
srija piratlaAtaccamer
posted in Data Quality & Data Governance ⚙️

How to save a project to a remote repository from Ataccama ONE Desktop?

Hello everyone!I’m Srija Piratla , Solutions consultant here at Ataccama. In this post, I'll guide you through saving your project to a remote repository using Ataccama ONE Desktop. This setup is beneficial when multiple team members are collaborating on the same project, allowing all code changes to be stored both locally and remotely.To push a project into a remote repository, we need to create a project and start a local repository in ONE Desktop.In Model or File Explorer, right-click the project folder and select Team > Share Project. In the Configure Git Repository dialog, specify the path to the folder where you want to set up your local Git repository.Note: It is recommended to create the repository outside your workspaceHello everyone!In this post, I'll guide you through saving your project to a remote repository using Ataccama ONE Desktop. This setup is beneficial when multiple team members are collaborating on the same project, allowing all code changes to be stored both l

0
S
Ataccamer
14 days ago
K
Data Pioneer
KarthikeyanData Pioneer
asked in Data Quality & Data Governance ⚙️

Removing Html tags

Hi Team,I am reading from collibra using collibra reader, some data has html tags like - <p> , <div> in it. Is there a way where we can filter out the html tags if its present in the data. like strip html tags like that. Let me know if more information is needed.Thanks.

3
Cansu
Community Manager
14 days ago
S
Data Voyager
SoniyaData Voyager
asked in Data Quality & Data Governance ⚙️

Data Quality

Hi team,I am working on a story where a user needs to see the results of the monitoring project which is running.Like where the results are getting saved in the backend,in the tables.Need to create a pipeline for that like when we click the run monitoring button in Ataccama Web,a workflow in Ataccama desktop should run and the results should be saved in the database to look at to see how many records are passing and failing.So how can we create that pipeline?

2
Cansu
Community Manager
14 days ago
M
Data Voyager
Matus_MarinakData Voyager
asked in Data Quality & Data Governance ⚙️

Merge Partitioned Parquet files

Hello all, i have datalake gen2 parquet files in containers and folders. In the folder are delta.log file and parquets. Is there some way to merge these parquets to one table? This table should represent one current(up to date) table contains data from parquets. this is what i see now in ataccama. I want to merge these data to one table. Thanks for replies, Matus

2
Cansu
Community Manager
15 days ago
S
Data Voyager
SoniyaData Voyager
asked in Data Quality & Data Governance ⚙️

Regarding DQ controls

Hi Team,I am trying to find how we can achieve completeness DQ control by comparing data in the file and the table irrespective of the position of primary key in either of them. Also, need to know how we can achieve the same by using the composite keys.

2
Cansu
Community Manager
15 days ago
maykwok_hamilton
Data Pioneer L2
maykwok_hamiltonData Pioneer L2
asked in Data Quality & Data Governance ⚙️

Convert unix timestamp to datetime

Hello,I hoped somebody has already worked this out so I don’t have to 🤣. I have a Epoch Unix Timestamp value that I want to convert into normal DATETIME format in Ataccama.e.g. 1711015200 I was expecting converted value to be 2024-03-21 10:00:00 (GMT).But when I do a toDateTime() on it, it gives me 1970-01-20 20:16:55. So I guessed wrong.I thought about using dateadd() but that one does not support seconds. Any ideas?

9
Joeri
Universe Traveller
16 days ago
S
Ataccamer
srija piratlaAtaccamer
published in Data Quality & Data Governance ⚙️

✍️ How to open a SSL certification file

Hello everyone,Are you encountering a certificate issue while running profiling? Let's dive into resolving this problem step by step.Issue: The monitoring project is encountering an error stating "cannot open ssl cert file /u01/ataccama/one/tmp/postgresql1.crt" during profiling. Solution Steps:Investigate Recent Changes: Check for any recent updates or modifications to configurations, certificates, or permissions in the environment. Confirm SSL Certificate Existence: Verify the presence of the SSL certificate file "postgresql1.crt" in the specified directory "/u01/ataccama/one/tmp/". Review Java Application Configuration: Ensure that the file path "/u01/ataccama/one/tmp/" is correctly specified in the Java application's configuration. Check Permissions: Verify if the Java application has the necessary permissions to access the SSL certificate file and the directory where it is located. Adjust Permissions if Needed: If permissions are lacking, adjust them to allow the application to

220
S
Ataccamer
16 days ago
M
Data Voyager
Matus_MarinakData Voyager
asked in Data Quality & Data Governance ⚙️

merging Partitioned Parquet files

Hello all, i have datalake gen2 parquet files in containers and folders. In the folder are delta.log file and parquets. Is there some way to merge these parquets to one table? This table should represent one current(up to date) table contains data from parquets. this is what i see now in ataccama. I want to merge these data to one table. Thanks for replies, Matus

2
M
Data Voyager
16 days ago
S
Data Pioneer
sgillaData Pioneer
asked in Data Quality & Data Governance ⚙️

DQS matching component results

DQS matching component results are not reflected when bundle is deployed and the entity is re-processed. I did re-build the repo with the new Extended unifier but no effect. What could be a reason?

15
S
Data Pioneer
17 days ago

Badge winners

  • First idea 💡
    sumeyrahas earned the badge First idea 💡
  • First idea 💡
    zayarhas earned the badge First idea 💡
  • First idea 💡
    Alpihas earned the badge First idea 💡
  • First idea 💡
    maykwok_hamiltonhas earned the badge First idea 💡
  • First idea 💡
    JTHhas earned the badge First idea 💡
Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings