Skip to main content

Learn & Share

Discuss data, share best practices, learn, and grow with Ataccama's Community.

876 Topics
K
Data Pioneer
KarthikeyanData Pioneer
asked in Data Quality & Catalog ⚙️
Compare two different columns from two different files

I am working on the usecase of comparing two different columns in two different csv files. I need to compare these column, remove any duplicates and merge them into single column. I am trying to use join component but not sure about this approach will work or any better solution is available to achieve this use case - If anyone can suggest a better apprach it will be helpful. - Thanks.

456
A
Ataccamer
anna.spakova
Ataccamer
anna.spakovaAtaccamer
published in Data Quality & Catalog ⚙️
Integrating git with orchestration server

Hi everyone, me again 😊 I work as a professional services consultant in Ataccama and I prepared another article for you, this time about some recommendations for integration with git, mainly for self-managed deployments. Specifically, when you have the DQG suite and you want to integrate your orchestration server. You can probably reuse some of these recommendations for other modules as well.Before you start reading, please note that I reused some of the already existing articles:https://community.ataccama.com/master-data-management-reference-data-management-92/one-desktop-git-recommendations-187?tid=187&fid=92I also recommend checking out this article in our documentation for working with git in our IDE:https://docs.ataccama.com/one-desktop/latest/projects/git.html#introduction-to-git-and-egitAlso please note, that this article doesn’t cover how to work with git in general, but more on how to work with the Ataccama projects, what files to version, what to look out for etc.Recomme

1650
anna.spakova
Ataccamer
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
asked in Data Quality & Catalog ⚙️
Can i get notification on term assignments

I have certain Business terms that i want to track on which fields are getting mapped from which datasets on a given day.Is there a way, i can send a notification to myself (and any others interested) on the occurance details of a term - and i schedule this to run periodically like every monday? To be specific, Iam want to track Social Security Number. I want to get a report every monday morning - with list of Catalog items and attributes that are mapped to this term. (from the occurances details). Is it possible?? How??  

414
Cansu
Community Manager
V
Universe Traveller
Vanja GilbertUniverse Traveller
asked in Data Quality & Catalog ⚙️
Possible to create filters to rules?

Hello! I am looking for assistance with applying filters to rules. Currently, I understand that the "where" clause can be used in rules, but it seems to apply to all records uniformly. What I need is a way to create a rule that only affects a subset of the data, so that the total count of records affected by the rule is different from the total count of records in the Catalog Item. For example, if I have a dataset with 1000 records, I want to create a rule that only applies to, say, 200 of those records based on specific criteria, without affecting the other 800 records. So if there is 100 records failing the result should be 100/200 instead of 100/1000.Is there a way to achieve this selective application of rules? Any guidance or examples on how to set this up would be greatly appreciated. Thank you!

663
Cansu
Community Manager
O
Data Voyager
olayinkadaramolaData Voyager
asked in Data Quality & Catalog ⚙️
Date Format

is there a way to change the date format in ataccama one. i want the format to be  dd mm yyyy, what i have currently is mm dd yyyy

1222
Cansu
Community Manager
K
Data Voyager
Kelvin FongData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Possible to show all (> 10) properties without the show all button?

Say now we have an term with more than 10 properties to display at the same time. Is it possible to (configure the metadata model so once we click the term it will) show all the properties without having to click this Show all button? Thanks in advance. 

252
Cansu
Community Manager
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Catalog ⚙️
How to run .vbs file in workflow

How can I run a .vbs file workflow. May be either ‘Run Shell Script’ or ‘Run Windows Command’. Or is there any other options available? I tied to run this through ‘Run Shell Script’ but it showing an issue with Interpreter(/bin/sh) 

622
Cansu
Community Manager
Y
Data Voyager
Yasharth MisraData Voyager
asked in Data Quality & Catalog ⚙️
how to find duplicates in ONE Desktop

Hello can you please assist in finding out how to find  the duplicates in a column using one desktop.

462
Cansu
Community Manager
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Catalog ⚙️
AI enabled term detection is giving significant false positives. How can i tune it

For terms like USA Zipcode, USA Social Security Number etc., - the term is enabled with AI detection and it is getting tagged to hundreds of fields - most of them are NOT zipcode or SSN.  If it was giving a score and recommendation, I would have discarded the recommendation, but it is assigning the term automatically and I have to go in and remove that - which became painful. I have now disabled the AI detection but is there a better way to tune it to make it work better? 

424
PetrD
Ataccamer
K
Data Voyager
Kelvin FongData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Possible to access Reference Objects by only clicking the object once?

Let’s say a data consumer in my workplace is getting familiar with a data element (as term), and he/she want to know who the data owner is. Since we set the person reference for the roles in stewardship, the consumer need to 1. click the display name of the data owner, see the username of the referred user, and then 2. click that username to see the detail of that user, namely his/her user ID and email address.(1) Is it possible in any way the data consumers only need to click once from the display name of the data owner to his/her user detail?(2) Is the above workaround applicable for other reference objects (e.g. the policy reference)?Thanks a lot. 

412
Cansu
Community Manager
Y
Data Voyager
Yasharth MisraData Voyager
asked in Data Quality & Catalog ⚙️
how to use representative creator

hello everyone , I am unable to understand how to use representative creator step can you please explain it in detail with an example so that I am able to use it . Thank you.

952
Cansu
Community Manager
K
Data Pioneer
KarthikeyanData Pioneer
posted in Data Quality & Catalog ⚙️
Recovery Audit in One Metadata Writer

HI Team,I am using one metadata writer for one of my use-case. In one-metadata writer there is an out option called recovery audit.  May I know the use of this functionality. I tried checking in help contents I was not able to find any document related to this. If I don’t want to use this recovery_audit is it possible?  This is the error I am getting while running the plan. Let me know if more information is needed from my end. Thanks.

273
K
Data Pioneer
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Catalog ⚙️
Can I run .vbs file from Workflow?

How can I run a .vbs file workflow. May be either ‘Run Shell Script’ or ‘Run Windows Command’. Or is there any other options available? I tied to run this through ‘Run Shell Script’ but it showing an issue with Interpreter(/bin/sh)  

202
A
Universe Traveller
S
Ataccamer
Samuel MuvdiAtaccamer
published in Data Quality & Catalog ⚙️
GIT Version Control Best Practices

Hi everyone!I wanted to share some best practices when it comes to GIT version control, how you can organize your branches in a way that merges and rollbacks are more granular, making it easier as well to see what code changes were done where Typically, what I tell clients when they are using GIT integrated with Ataccama, is that they would essentially need to have a standard flow control process for their branches. What I mean by that, is that they should set up 1 main production branch, 1 development branch, and many feature branches. You can think about it like this, your Master branch can be thought of as your “Production” branch, your Development branch can be thought of as your “Test” server, and each of those feature branches you can see it as your “Dev” server where you develop new code. How this works is simple yet effective:You start off with a main branch in your repo, we can call it “master” Master can have either prior code or be empty, let's say it has some code in it alr

1000
S
Ataccamer
R
Data Voyager
rpainte1Data Voyager
asked in Data Quality & Catalog ⚙️
Advanced Expression in IDE (Atacaama Desktop) to perform a uniqueness chaeck

I have been trying for some time to build a data quality check/rule in Ataccama Desktop (using an advanced expression) to check where a defiend attribute is unique/distinct and have been struggling to find the correct function to use for this.In a tool like excel I woudl simply count the number of instances of each value and following that I would ensure that anything where the count was >1 failed this check but I have thus far been unable to find a way to perform a distinct count in this way. e.g. DistinctCount([Install_And_Leave_Flag]) = 1I understand that the profiling step nativley gives you this information but I require a way to do that via an advanced expression for my use case.  

3076
Cansu
Community Manager
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
Permission/Security setup via the OneDesktop IDE (RDM)

Setting permissions for groups via the IDE.Ensure your groups exists in KeyCloak first In the IDE go to your RDM Project and complete the following under "Security" Right click on Roles Select New role   Add the name and description of the role (be sure to use the same names that you setup in KeyCloak) then click Apply and OK Expand the new role, and right click on Entities then select Edit…  Add the Tables and the Table level permissions in the next window  NOTE: Table permissions do not give column level permissions. Double click the number next to the table that you wish to setup permissions for Notice that there are options for "View for All Columns" and "Modify for All Columns." However, if you want to select only certain columns, you can do that below with the following steps. Add the column names you want permissions to be given for, and select View and/or Modify for the given column   When complete, click Apply then OKTo ensure these permissions are applied to the Web Admin do

835
Cansu
Community Manager
S
Data Pioneer
suryakanth.EmishaData Pioneer
asked in Data Quality & Catalog ⚙️
How to export ONE Webapp Audit logs using ONE Desktop IDE?

Hello everyone, I have a requirement of exporting ONE webapp's audit logs but not using the MinIO storage, is there any other way to export log files through ONE Desktop IDE using any Api calls such as graphql converted into curl command in json call? Thanks!

661
anna.spakova
Ataccamer
Anna_Nebesova
Ataccamer
Anna_NebesovaAtaccamer
published in Master Data Management & Reference Data Management 🛰️
How to use graphQL queries in ONE DesktopTutorial

Let‘ say you have a graphQL query that works in a Postman or in Ataccama playground, and you want to use it in a plan in ONE Desktop. How to do it?You cannot simply copy and paste the JSON body into JSON call step. You have to reformat it. You can use the COPY CURL button in Playground (in the upper right corner) or similar button Code in Postman. That will create a curl command that also contains the body:PlaygroundCurl commandThis body can be then pasted into JSON call. The query method needs to be set as POST.JSON call step configuration Want to learn more about the graphQL API? Check out the documentation: https://support.ataccama.com/home/docs/aip/latest/development-guides/one-development-guide/one-api

5967
anna.spakova
Ataccamer
aysel_jafarzade
Data Pioneer
aysel_jafarzadeData Pioneer
asked in Data Quality & Catalog ⚙️
Combination of Email Notifications

Hello,I want to set up a single general email notification for various monitoring projects. Typically, I receive separate email notifications for each monitoring project, but I would like to receive only one email notification that consolidates the results of different monitoring projects. I know that I am supposed to use One Desktop but I don’t know which steps exactly. I would be grateful if you could help.Thanks in advance😊

1183
aysel_jafarzade
Data Pioneer
R
Universe Traveller
RiannaUniverse Traveller
asked in Data Quality & Catalog ⚙️
DQ rule and Regular expression - Truncate

Hello Community 😀I am trying to write a DQ rule using a regular expression, I have been using the “ONE Expressions” documentation to help with this but I can’t seem to get it working. Can anyone advise where I am going wrong please??ExampleInput data:Value1 Value2 Value3 -1137.86 -5689.33 0.2  DQ rule I am trying to achieve:Value2 * Value 3 = -1137.866 Take only the first 2 decimals of the above bullet WITHOUT rounding = -1137.86 Check if value1 equals the value in the bullet above. (i.e. -1137.86 = -1137.86 so it should show as a valid result)How I have written this in Ataccama using a regular expression:Value1 is NOT (substituteAll("(\d+\.\d{2})(\d+)","${1}",toString(Value2 * Value3))) Then result is INVALIDError I get:Using the test rule feature with the example input data above, it says that the result is INVALID when I am expecting it to be VALID. And only if I change Value1 to equal -1137.866 then it will show VALID which is not what I want to achieve.Can anyone see wh

613
R
Universe Traveller
may_kwok
Star Blazer L3
may_kwokStar Blazer L3
asked in Master Data Management & Reference Data Management 🛰️
RDM Extended Reader row filter syntax

We are on RDM v14.5.1, on PaaS, so underlying db is postgres.I need to query a table using the RDM Extended Reader step with a filter. The volume of this table will grow, so I much prefer to filter at the reader step, rather than bring the whole table into the plan and then filter, to manage performance.I have a column, from_date, that is datetime type. I have from_date in my data flow, and I want to query RDM for a day prior.Expecting that this is an SQL where clause, I tried to put in syntax compatible with Postgres So I created a column called “row_filter”, with type string. Using postgres to_timestamp functionday_prior:=dateAdd(from_date,-1,'DAY');day_prior_string:=tostring(day_prior,'yyyy-MM-dd HH:mm:ss');"from_date=to_timestamp('"+day_prior+"','YYYY-MM-DD HH:MI:SS')" This one gave an error of:13.05.2024 13:10:00 [INFO] Ataccama DQC engine initialized.13.05.2024 13:10:00 [INFO] Creating runtime...13.05.2024 13:10:00 [INFO] Starting runtime...13.05.2024 13:10:00 [INFO]

303
may_kwok
Star Blazer L3
JTH
Data Pioneer
JTH
asked in Data Quality & Catalog ⚙️
Record having max one empty attribute

Hi,This may sound very easy, but: how do I retrieve records that have code = ‘A’ and more than one empty start dates? So in the end I want to see records 3 & 4 since they have the same id_#, code = ‘A’ and more than 1 empty start date. Would be ideal if this was in ONE Web and not through ONE Desktop.Person id_# code start_date 1 1 A 1-1-2000 2 1 A 1-2-2000 3 1 A   4 1 A   5 2 A 1-1-2000 6 2 A 1-2-2000 7 2 A    select *from Personwhere start_date is emptyand code = ‘A’group by id_#, codehaving count (*) > 1I tried to use count in advanced expressions, but can't seem to get it to work.

814
JTH
Data Pioneer
Y
Data Voyager
Yasharth MisraData Voyager
asked in Data Quality & Catalog ⚙️
combining the output of more than 3 or more text file write or component output

I am facing the problem that I have output from 4 components and the I am writing them on different text file writers is there any way I can combine the outputs into a single text file writer if not can I combine the outputs from the different components and write it in a single text file writer.

291
A
Ataccamer
kayhan.turker
Ataccamer
kayhan.turkerAtaccamer
published in Data Quality & Catalog ⚙️
Export Monitoring Project Metedata + Results For Multiple Runs And Versions

Export Monitoring Project Metedata + Results For Multiple Runs And Versions Introduction For DQ monitoring and reporting, it may be useful to extract the results of monitoring project reports into a spreadsheet, or to an external database. It may also be useful to extract multiple or all reports/runs for a monitoring project at once.The following topic covers a component created to export results, how it can be used, and how it was created (in case modifications need to be made).  Scope of the Component The component extracts DQ results from a monitoring project. The output is on the rule instance level, where each rule included in the monitoring project across all tables is output with its dq validity percentage, and the number of valid and invalid records for said rule. Each record is also output with a date field, to organize results by the report run date (as multiple reports may be included in one export) Using the Component To use the component, move the component files to a fold

2241
kayhan.turker
Ataccamer
A
Data Voyager
akumaData Voyager
asked in Data Quality & Catalog ⚙️
Mass Delete Catalog Items

Hi Community,We have a scenario where an entire schema was imported as an Ataccama source. This resulted in over 700 tables/views brought in as catalog items.Is there a way to bulk delete all 700?Thank you.

1714
anna.spakova
Ataccamer

Badge winners

  • First idea 💡
    sumishahas earned the badge First idea 💡
  • First idea 💡
    BarbaraBhas earned the badge First idea 💡
  • First idea 💡
    jbinghas earned the badge First idea 💡
  • First idea 💡
    samsTmobilehas earned the badge First idea 💡
  • First idea 💡
    ViLashas earned the badge First idea 💡
Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings