Skip to main content

Data Quality & Catalog βš™οΈ

Discuss, learn, ask, suggest, share all things DQ&C πŸ”

  • 791 Topics
  • 2,013 Replies
791 Topics
D
Data Pioneer
DhavalMData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
How to Transpose columns in a plan?

I have been given a list of emails that I want to transposeβ€”converting the data from rows to separate columnsβ€”and then concatenate these columns into a single row. How can I do that in the IDE?Β Original DataΒ After TransposeΒ Final OutputΒ Note: All the emails will be unique, so I don’t want them to be grouped. Could you please suggest how I can transpose these rows into columns and later merge them into one single row?Thank you.Regards,Dhaval

762
A
E
Universe Traveller
EmmanuelFUniverse Traveller
asked inΒ Data Quality & Catalog βš™οΈ
How to add a column from an csv file to the "To" section of the SendMail in workflow?

We are trying to pass a csv file’s column which is generated from the Run DQC andΒ contains a list of emails to the Send Mail’sΒ β€œto” section. What are the other ways of passing a column to that particular section. Let me know if there are any alternative steps.Β This is the workflow planConfiguration of Send Mail:Regards,Emmanuel

1255
E
Universe Traveller
K
Data Pioneer
KarthikeyanData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
Compare data between file and table

we need to achieve completeness DQ control by comparing data in the file and the table irrespective of the position of primary key in either of them or by using composite keys. - This is in continuation with the query posted in ataccama community - Regarding DQ controls (ataccama.com) Do we have any step or document to follow on how we can import that file into ONE web app as catalog item and can we upload the file from local or we need to get that from the file system?Β Thanks,Karthikeyan.

1046
K
Data Pioneer
M
Universe Traveller
MelUniverse Traveller
asked inΒ Data Quality & Catalog βš™οΈ
Duplicate detection with aggregation results config

I have created a rule that detects if a record is in the table more than once based on aggregation of 2 attributes in OneWeb application.Β  This gives a proper result in Invalid samples data and exports giving 2Β errorsΒ if the record is in the table 2Β times etc.Β  But the customer requires that only 1 record should be visible in Invalid samples and also in exporting to db.Β  It does not matter which one.Β  Is it possible to configure the result to do this?

1463
J
Ataccamer
K
Data Pioneer
KarthikeyanData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
data reconciliation for snowflake data

Hi TeamΒ This is the continuation of the of the below thread about using reconciliation project for comparing a file with the table.Compare data between file and table (ataccama.com)Β The data compare is working fine when I followed the approach suggested.Β  But from this link (Data Reconciliation Best Practices πŸ‘Ύ (ataccama.com)Β - it is mentioned as (except snowflake data).So, what will be the approach for the snowflake related connections.Β I have one more question - for eg ., I have column fullName in source (which has a full name) in target I have two columns one has the first name and other column has the second name. Can we compareΒ source column to target column1 and 2.Β Β 

900
K
Data Pioneer
S
Data Pioneer
sumeyraData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
DQ rule using complex lookup

Hello,I was wondering if someone has an idea to help my question about one of my DQ check. Basically let me explain what i have and what i want to achieve with them.I have; one list consisting 2 columns.Β Column AΒ has unique values having 2 or 3 digits with a mixed combination of letters and numbers.Β Column B has names according to the column A. Here is the example:Β Column A Column B abc name1 abd name1 abe name1 abf name2 xy name3 1a1 name4 1a2 name4 1a3 name4 yz name5 yz1 name6 Β As you see, there is no corelation between column A and column B. I can’t assume if the value of column A is yz and value of column B is name5 then i have name5 for yz1 of column A.And I have a field in my data lets call it as column C. Column C has values with a mixed combination of digits and letters and with different length. Such as abc1231422ad ,Β 1a3577afdg , yz4564sdfsfs can be example.Β And I have another field which is Column D having some other values.Β I want to check v

984
S
Data Pioneer
Albert de Ruiter
Rocket Pioneer L1
Albert de RuiterRocket Pioneer L1
published inΒ Data Quality & Catalog βš™οΈ
Business glossary - customization (part 1)
Business glossary - customization (part 1)

Hi Community,I’d like to share some thoughts about the business glossary. After considering how we wanted to use the glossary, we applied customizations in the metamodel, so we have a specific implementation. I suppose that others have a different approach to using the glossary, so it would be great if we could share and discuss our visions. I am looking forward to your feedback!Let’s start.The business glossary provides a way to describe terms that have relevance to the organization. Obvious properties like name, definition and owner describe a term. It is possible to connect catalog items and its attribute to a term, so that physical data elements becomes meaningful.The question is, can we describe these elements with the same properties? (Indeed a rhetorical question)To logically describe a catalog item attribute, besides its name and definition also properties like its (logical) datatype, an indicator to mark a β€˜personal identifying information’ term and the GDPR classification.The

1851
A
S
Data Pioneer
sgillaData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
matching component - child tables

On a matching component if address table is mapped as shown in the screenshot. Why would the process still run for the address table for both insertion and deletion. while its not being used. How to avoid it run through it and why would it run through? it would not need a master_id associated wither as it is tied to parent by u_parent_id.Β 

405
Phil Holbrook
Ataccamer
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked inΒ Data Quality & Catalog βš™οΈ
Start Ataccama server without opening Ataccama Desktop tool

I'm currently using the "start-server.bat" file within the Ataccama Desktop tool to manually start Ataccama Server. However, if I forget to start the server, certain workflows fail to execute. I've attempted to automate server startup using Windows Task Scheduler, but it's not working as expected. Is there a reliable method to start Ataccama Server independently of the Desktop tool, perhaps through a scheduled task or another approach?

314
A
Universe Traveller
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted inΒ Data Quality & Catalog βš™οΈ
Can term detection rule use both data and metadata both in the same rule?

Hello:Β I have a use case specifically for assigning a term. If the data is a 9 digit number and if the field name is NOT one of the list i want to exclude - then that term be assigned.Β I also have additional field names that i want to be tagged with that term - irrespective of value with in that field.As of now - iam able to createΒ  separate rules one that runs on metadata and other runs on real data - and put a OR condition or AND condition between the two rules. But that is not addressing my use case very well.Β I need one rule that can refer to both real data and metadata.Β  OR - i should be able to give a complex join condition with more than two rules(Rule 1Β AND Rule 2) OR (Rule 3)Β Has anyone done something like that?

281
R
Ataccamer
may_kwok
Star Blazer L3
may_kwokStar Blazer L3
asked inΒ Data Quality & Catalog βš™οΈ
Attaching a client certificate in a Json Call step

I’m trying to build a GET API call to an external data provider.This provider requires me to send my GET call with the certificate.We managed to test it in POSTMAN, where I had to add the certificate in the Settings β†’Β Certificates menu:Β I have a .pfx file with a passphrase, and I needed to specify the host.This works on my laptop in POSTMAN.Β Now I’m trying to do the same call in ONE Desktop using JSON Call step.By way of googling, I’ve managed to convert my .pfx file into .cer, then IΒ imported the .cer fileΒ into {DQC_HOME}/jre/lib/security/cacerts . Then I have restarted ONE Desktop. But when I call the API I’m still getting an error message from the other side saying my certificate is not sent.Β I have a suspicion that the certificate is sitting in the keystore but it is not used. Is there some way to ensure it is used? Some java parameter somewhere?

1866
may_kwok
Star Blazer L3
T
Data Pioneer L2
ThomasProData Pioneer L2
asked inΒ Data Quality & Catalog βš™οΈ
Microsoft Teams Notifications - incoming webhook is discontinued for new configurations

Hey all!Now that incoming webhooks within Microsoft Teams is discontinued (Existing configurations will work until end of 2025), is there a plan for how teams will continue to integrate with notifications from Monitoring Projects? This was a big sell to a lot of our users, and now that we are unable to use incoming webhooks for new monitoring projects, I want to see if there is a plan to have a work around, or if it will be not supported in the future.Β Β Β Β 

1264
A
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted inΒ Data Quality & Catalog βš™οΈ
Day over day changes for certain data

Hello: I have a use case where when the data in a given column - for a key changes from previous day to next day - that should trigger an alert/notificationΒ For Ex: Security Ratings, and/or ESG ratings generally dont change for a given security. I have a table that has one record for each day for each security. And the ratings are in the columns of that table. Now - on a regular day - all ratings from previous day should be same for next day, and day after (until it changes). And on the day when it changes - i want to capture that security, ,what field changed - what was yesterday;s value vs todays value.Β The way - i would like to solve this - is create a SQLCatalog item - with previous day - and current day values combined into one dataset for each security and write scripts to compare them one by one, and report any changes. But I want to know if there is an out of box - date driven compare that can be done for this.Β 

381
A
Universe Traveller
S
Ataccamer
srija piratlaAtaccamer
published inΒ Data Quality & Catalog βš™οΈ
How to create a Hyperlink in ONE Web App
How to create a Hyperlink in ONE Web App

Hi everyone, I’m Srija Piratla, Solutions Consultant here at Ataccama.In this post I’m going to talk about how to create a hyperlink for an attribute column in the ONE Web App. Curious? Follow these steps. This example demonstrates how to make a column, such as "Product URL," in the catalog item "Epic Dashboard" clickable, redirecting to the respective web page.Steps to Create a Hyperlink for an Attribute ColumnCreate Metadata Model: First, ensure you have a metadata model set up. You can find detailed instructions for creating a metadata model here. If you need to create a new entity, use the metadata writer step to load the data. Edit the Page Template: Navigate to the page where you want to create the URL hyperlink. In the top right corner of the page, click on the Edit Page Template option. Β  Add Renderers for Hyperlinks: In the page template editor, add the necessary renderers for hyperlinks. This involves specifying the renderer type for the attribute you want to make clic

390
S
Ataccamer
R
Space Explorer
rajeswarimylaSpace Explorer
asked inΒ Data Quality & Catalog βš™οΈ
How to get yesterday date

hi Team, i need to get yesterday date in my workflow global variable, can you please help in getting.when i useΒ toString(now(),"yyyy-MM-dd") -- printed me todays dateΒ when i useΒ toString(now() - 1 ,"yyyy-MM-dd"), getting error as operator β€œ-” can not be used

522
R
Space Explorer
D
Universe Traveller
Delbert71Universe Traveller
asked inΒ Data Quality & Catalog βš™οΈ
How to define data types in Ataccama using regular expression syntax

Hi,Β I am trying to write some rules to check the data type of attributes within sql tables or flat file columns. I am aware that profiling checks this but I have a requirement to check the data type as well. There don’t appear to be any parse functions in the interface. Any help would be appreciated, I am a noob so please be gentle :)

682
D
Universe Traveller
S
Data Pioneer
sgillaData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
DQS matching component results

DQS matching component results are not reflected when bundle is deployed and the entity is re-processed. I did re-build the repo with the new Extended unifier but no effect. What could be a reason?

13618
S
Data Pioneer
JTH
Data Pioneer
JTHData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
Export functionality visiblity

Hi,13.9.4Currently we have a monitoring project with several DQ rules applied. We also build in ONE Desktop an export functionality through components in order for the users to click one button and get a .xlsx file to see all the invalid records.ONE Deskptop component​​​​​When the .xlsx is downloaded and opened you will see the following:ExportAs you can see the same rules is shown multiple times with every time because a record can have multiple invalid rules applied on them hence the several options. This creates a list of invalid rules and for the user it's not a great experience to filter through the file. Is there a solution to this?I also want to take the moment to ask another question. As you can see you’ll notice β€œFLOAT is empty”. That is a generic rule we make available for every user. Input attribute is a float and it's checks whether the field is empty. If yes, it's invalid. This rule can be applied to any attribute that is a float data type. However in the invalid records I

412
JTH
Data Pioneer
G
Universe Traveller
GloriaUniverse Traveller
asked inΒ Data Quality & Catalog βš™οΈ
Attaching export files to Monitoring Project Notifications.

Is it possible to include attachments (containing invalid records) from the export tab of the monitoring project to the notifications sent out to the users?

1023
G
Universe Traveller
T
Data Pioneer L2
ThomasProData Pioneer L2
asked inΒ Data Quality & Catalog βš™οΈ
Is there a way to see the plain values of prebuilt lookup lists?

Hello!I had a user pose the question of being able to view the plain values of lookup lists that came out of the box with Ataccama. When attempting to look at the lookup lists, it is unreadable from the Web UI, and the Desktop application.Β Is there any listed documentation of what these lookup lists contains, or a desktop planΒ  to transform the lookup lists into a plain English format?Thanks!

263
T
Data Pioneer L2
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted inΒ Data Quality & Catalog βš™οΈ
DQ Rules using Lookups

I have about 200+ rules in my rule library that i built over a period of last 2 years.Some of them I used Lookups. Later on I stopped using the rules in my moniotring projects, but the rules are still left in rule library. I want to disable the rebuildingo of lookups.Β Β How do i find out which of my rules are using the Lookups? How to find them?

351
R
Universe Traveller
S
Ataccamer
srija piratlaAtaccamer
published inΒ Data Quality & Catalog βš™οΈ
How to run the profiling using GQL based on the Catalog Item Name and type of profiling
How to run the profiling using GQL based on the Catalog Item Name and type of profiling

Hi Everyone ! I’m Srija ,Solutions Consultant here at ataccama. Today I’m sharing how to profile a catalog item using GraphQLΒ by providing the Catalog item name and type of profiling.Β Β Profiling is usually performed to Catalog Items. There are two types of data profiling in AtaccamaOne.ο‚· Sample Profilingο‚· Full ProfilingWe can do profiling using the Ataccama One Web GUI but there are limitations. Consider a scenario where profiling needs to be executed for multiple catalog items in one Go. In such cases, GraphQL can be leveraged within a plan to facilitate the profiling process. By specifyingthe catalog items and the type of profiling (Sample/Full), the profiling operation can be seamlessly performed.The below GraphQL Query helps to run the specified profiling for the specified Catalog Item. This can be executed in the playground/postman tool or can be done in a plan in ONE desktop.Β Here's the query structure:query ImportCatalog ($connection_id: GID!) {connection(gid: $connection_id) {i

892
S
Ataccamer
A
Universe Traveller
apejkoUniverse Traveller
asked inΒ Data Quality & Catalog βš™οΈ
Breaking Excel Sheet into multiple files

HiΒ everyone,I currently have a plan that is creating a main task and populating it with sub tasks that each link to individual tables. My issue at the moment is that sometimes this creates tasks with a huge amount of sub tasks that may seem daunting for people to try to work through.Currently the tables and their attribute ids are all stored in one excel sheet, so I was wondering if it would be possible to break it into separate excel files each containing 20 rows. If this is possible I would then be able to create additional tasks rather than just grouping everything in one.Thank you for any help!

983
A
Universe Traveller
S
Data Pioneer
sumeyraData Pioneer
asked inΒ Data Quality & Catalog βš™οΈ
Adding/Removing Attributes of Catalog item in Monitoring Project?

Hi all!Β I am working on a monitoring project. I have a catalog item consisting 130 columns. I want to create a custom catalog item selecting only few attributes that I need to have them in my monitoring project.However, based on my previous experience: Once I have created a catalog item with selected attributes. Then I had to add new attributes according to my project needs.Β I have run the monitoring project again with new attributes of catalog item, then I lost my monitoring project. It showed errors for all DQ rules because as I understood from error message, DQ rules need to have attribute key ids, and with adding new attributes I have changed attribute key ids.Β Now, again I have a huge catalog item and I want to reduce the volume with selecting few attributes but I don’t want to face with the same problem again. If you have any suggestion it would be so great for me.Β Thanks !

805
S
Data Pioneer
M
Data Voyager
Matus_MarinakData Voyager
asked inΒ Data Quality & Catalog βš™οΈ
merging Partitioned Parquet files

Hello all,Β i have datalake gen2 parquet files in containers and folders. In the folder are delta.log file and parquets. Is there some way to merge these parquets to one table? This table should represent one current(up to date) table contains data from parquets.Β this is what i see now in ataccama. I want to merge these data to one table.Β Thanks for replies, Matus

583
S
Ataccamer

Badge winners

  • 10 ideas πŸ’‘
    akumahas earned the badge 10 ideas πŸ’‘
  • First idea πŸ’‘
    romizhas earned the badge First idea πŸ’‘
  • First idea πŸ’‘
    kellymremzilybhas earned the badge First idea πŸ’‘
  • First idea πŸ’‘
    Francois Ascanihas earned the badge First idea πŸ’‘
  • 20 ideas πŸ’‘
    ayounghas earned the badge 20 ideas πŸ’‘
Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, andΒ have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings