Skip to main content

Data Quality & Governance ⚙️

Discuss, learn, ask, suggest, share all things DQG 🔍

  • 534 Topics
  • 1,285 Replies
534 Topics
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Governance ⚙️
AI enabled term detection is giving significant false positives. How can i tune it

For terms like USA Zipcode, USA Social Security Number etc., - the term is enabled with AI detection and it is getting tagged to hundreds of fields - most of them are NOT zipcode or SSN.  If it was giving a score and recommendation, I would have discarded the recommendation, but it is assigning the term automatically and I have to go in and remove that - which became painful. I have now disabled the AI detection but is there a better way to tune it to make it work better? 

354
PetrD
Ataccamer
4 months ago
Y
Data Voyager
Yasharth MisraData Voyager
asked in Data Quality & Governance ⚙️
how to use representative creator

hello everyone , I am unable to understand how to use representative creator step can you please explain it in detail with an example so that I am able to use it . Thank you.

642
Cansu
Community Manager
4 months ago
K
Data Pioneer
KarthikeyanData Pioneer
posted in Data Quality & Governance ⚙️
Recovery Audit in One Metadata Writer

HI Team,I am using one metadata writer for one of my use-case. In one-metadata writer there is an out option called recovery audit.  May I know the use of this functionality. I tried checking in help contents I was not able to find any document related to this. If I don’t want to use this recovery_audit is it possible?  This is the error I am getting while running the plan. Let me know if more information is needed from my end. Thanks.

173
K
Data Pioneer
5 months ago
A
Universe Traveller
Ajeesh_GUniverse Traveller
asked in Data Quality & Governance ⚙️
Can I run .vbs file from Workflow?

How can I run a .vbs file workflow. May be either ‘Run Shell Script’ or ‘Run Windows Command’. Or is there any other options available? I tied to run this through ‘Run Shell Script’ but it showing an issue with Interpreter(/bin/sh)  

182
A
Universe Traveller
5 months ago
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Governance ⚙️
I would like to get alert when any new data flagged with PII

Hello: I would like to create an alert - while scanning the datasets - if it flags any data to be PII (such as Social Security Number, Data of Birth, Tax Identification Number) etc., Is there a way to get alerted when a specific term is assigned to any catalog item attribute? 

180
Prasad Rani
Data Pioneer
5 months ago
S
Ataccamer
Samuel MuvdiAtaccamer
published in Data Quality & Governance ⚙️
GIT Version Control Best Practices

Hi everyone!I wanted to share some best practices when it comes to GIT version control, how you can organize your branches in a way that merges and rollbacks are more granular, making it easier as well to see what code changes were done where Typically, what I tell clients when they are using GIT integrated with Ataccama, is that they would essentially need to have a standard flow control process for their branches. What I mean by that, is that they should set up 1 main production branch, 1 development branch, and many feature branches. You can think about it like this, your Master branch can be thought of as your “Production” branch, your Development branch can be thought of as your “Test” server, and each of those feature branches you can see it as your “Dev” server where you develop new code. How this works is simple yet effective:You start off with a main branch in your repo, we can call it “master” Master can have either prior code or be empty, let's say it has some code in it alr

790
S
Ataccamer
5 months ago
R
Data Voyager
rpainte1Data Voyager
asked in Data Quality & Governance ⚙️
Advanced Expression in IDE (Atacaama Desktop) to perform a uniqueness chaeck

I have been trying for some time to build a data quality check/rule in Ataccama Desktop (using an advanced expression) to check where a defiend attribute is unique/distinct and have been struggling to find the correct function to use for this.In a tool like excel I woudl simply count the number of instances of each value and following that I would ensure that anything where the count was >1 failed this check but I have thus far been unable to find a way to perform a distinct count in this way. e.g. DistinctCount([Install_And_Leave_Flag]) = 1I understand that the profiling step nativley gives you this information but I require a way to do that via an advanced expression for my use case.  

2706
Cansu
Community Manager
5 months ago
S
Data Pioneer
suryakanth.EmishaData Pioneer
asked in Data Quality & Governance ⚙️
How to export ONE Webapp Audit logs using ONE Desktop IDE?

Hello everyone, I have a requirement of exporting ONE webapp's audit logs but not using the MinIO storage, is there any other way to export log files through ONE Desktop IDE using any Api calls such as graphql converted into curl command in json call? Thanks!

441
anna.spakova
Ataccamer
5 months ago
aysel_jafarzade
Data Pioneer
aysel_jafarzadeData Pioneer
asked in Data Quality & Governance ⚙️
Combination of Email Notifications

Hello,I want to set up a single general email notification for various monitoring projects. Typically, I receive separate email notifications for each monitoring project, but I would like to receive only one email notification that consolidates the results of different monitoring projects. I know that I am supposed to use One Desktop but I don’t know which steps exactly. I would be grateful if you could help.Thanks in advance😊

1063
aysel_jafarzade
Data Pioneer
5 months ago
R
Universe Traveller
RiannaUniverse Traveller
asked in Data Quality & Governance ⚙️
DQ rule and Regular expression - Truncate

Hello Community 😀I am trying to write a DQ rule using a regular expression, I have been using the “ONE Expressions” documentation to help with this but I can’t seem to get it working. Can anyone advise where I am going wrong please??ExampleInput data:Value1 Value2 Value3 -1137.86 -5689.33 0.2  DQ rule I am trying to achieve:Value2 * Value 3 = -1137.866 Take only the first 2 decimals of the above bullet WITHOUT rounding = -1137.86 Check if value1 equals the value in the bullet above. (i.e. -1137.86 = -1137.86 so it should show as a valid result)How I have written this in Ataccama using a regular expression:Value1 is NOT (substituteAll("(\d+\.\d{2})(\d+)","${1}",toString(Value2 * Value3))) Then result is INVALIDError I get:Using the test rule feature with the example input data above, it says that the result is INVALID when I am expecting it to be VALID. And only if I change Value1 to equal -1137.866 then it will show VALID which is not what I want to achieve.Can anyone see wh

543
R
Universe Traveller
5 months ago
JTH
Data Pioneer
JTH
asked in Data Quality & Governance ⚙️
Record having max one empty attribute

Hi,This may sound very easy, but: how do I retrieve records that have code = ‘A’ and more than one empty start dates? So in the end I want to see records 3 & 4 since they have the same id_#, code = ‘A’ and more than 1 empty start date. Would be ideal if this was in ONE Web and not through ONE Desktop.Person id_# code start_date 1 1 A 1-1-2000 2 1 A 1-2-2000 3 1 A   4 1 A   5 2 A 1-1-2000 6 2 A 1-2-2000 7 2 A    select *from Personwhere start_date is emptyand code = ‘A’group by id_#, codehaving count (*) > 1I tried to use count in advanced expressions, but can't seem to get it to work.

704
JTH
Data Pioneer
5 months ago
Y
Data Voyager
Yasharth MisraData Voyager
asked in Data Quality & Governance ⚙️
combining the output of more than 3 or more text file write or component output

I am facing the problem that I have output from 4 components and the I am writing them on different text file writers is there any way I can combine the outputs into a single text file writer if not can I combine the outputs from the different components and write it in a single text file writer.

251
A
Ataccamer
5 months ago
kayhan.turker
Ataccamer
kayhan.turkerAtaccamer
published in Data Quality & Governance ⚙️
Export Monitoring Project Metedata + Results For Multiple Runs And Versions

Export Monitoring Project Metedata + Results For Multiple Runs And Versions Introduction For DQ monitoring and reporting, it may be useful to extract the results of monitoring project reports into a spreadsheet, or to an external database. It may also be useful to extract multiple or all reports/runs for a monitoring project at once.The following topic covers a component created to export results, how it can be used, and how it was created (in case modifications need to be made).  Scope of the Component The component extracts DQ results from a monitoring project. The output is on the rule instance level, where each rule included in the monitoring project across all tables is output with its dq validity percentage, and the number of valid and invalid records for said rule. Each record is also output with a date field, to organize results by the report run date (as multiple reports may be included in one export) Using the Component To use the component, move the component files to a fold

1331
kayhan.turker
Ataccamer
5 months ago
A
Data Voyager
akumaData Voyager
asked in Data Quality & Governance ⚙️
Mass Delete Catalog Items

Hi Community,We have a scenario where an entire schema was imported as an Ataccama source. This resulted in over 700 tables/views brought in as catalog items.Is there a way to bulk delete all 700?Thank you.

1374
anna.spakova
Ataccamer
5 months ago
Alan Raubenheimer
Data Voyager
Alan RaubenheimerData Voyager
asked in Data Quality & Governance ⚙️
Data Asset from ETL job

Is it possible to create a data asset from an ETL job (SSIS or similar) so that this is included in the data lineage to allow for full impact analysis 

231
F
Ataccamer
5 months ago
A
Data Voyager
AkuljotData Voyager
asked in Data Quality & Governance ⚙️
Data stories on synapse source catalog items

Currently only 4 sources are supported for data stories and they are mentioned below: Amazon Aurora PostgreSQL ONE Data PostgreSQL Snowflake We also use synapse as source for a lot of catalog items and it would be great if you can add synapse as well to the above. 

191
Cansu
Community Manager
5 months ago
aysel_jafarzade
Data Pioneer
aysel_jafarzadeData Pioneer
asked in Data Quality & Governance ⚙️
Deleting Component

Hello,How we can delete one of the components in One Desktop? I created virtual catalog item through One Desktop. Virtual catalog item has already been deleted but its component still remains.  I want to delete its component as well.  For example, I want to delete component “branch_dwh_vs_lake”. How can I delete it from desktop?Thanks in advance, 

312
aysel_jafarzade
Data Pioneer
5 months ago
jdordregter
Data Voyager
jdordregterData Voyager
asked in Data Quality & Governance ⚙️
Term insights (term suggestions)

Hi, I notice that for some of my terms, the visual Insights is visible and for others it is not. What is the reason for this?Second question: I would want to automatically assign terms. if the column is named wnplts then I want to assign the term Residence to it. Is there an easy way to do this? Detection rules seem to look at the value and not at the column name.kind regards, Jur Dördregter

272
jdordregter
Data Voyager
5 months ago
jdordregter
Data Voyager
jdordregterData Voyager
asked in Data Quality & Governance ⚙️
Masking of sensitive data

Hi,I have certrain fields in my data that should be masked for all users. This is information like social security number and IBAN. Now users can see this data from the sample data tab and from the profile. Can I apply some some masking of these attributes without have to create a VCI or SCI for this? Ideally masking should be applied based on the term assignment: as soon as I map the data to IBAN it should be masked for all users.Regards, Jur Dördregter

1175
Cansu
Community Manager
5 months ago
JTH
Data Pioneer
JTH
asked in Data Quality & Governance ⚙️
Priority DPM

Hi,We have several Monitoring Projects that are scheduled to run at 07:00. If for instance 1000 jobs are still in the queue those scheduled MP will run after the 1000 jobs (assuming the priority is the same; 0). In order to change this I have to go to DPM and change the priority of the scheduled MPs. Is there a way where I can schedule a Monitoring Project at a specific time, 07:00, with a certain priority (automatically)?

532
ivan.kozlov
Ataccamer
5 months ago
Prasad Rani
Data Pioneer
Prasad RaniData Pioneer
posted in Data Quality & Governance ⚙️
how do i do an exact field search in data catalog?

how do i do an exact field search?For Ex: I have a field named trd_settle_date  - and when i search for that in the data catalog, it is Giving me all fields in catalog that either started with “trd” or had “settle” word some where in it etc.,Because of that, the search is giving so many unnecesary values in the list, and making it difficult to find the right one.How can i get Ataccama to perform exact match to my search string? If it doesnt find - its OK for it to give me “no items found”

605
M
Ataccamer
5 months ago
Catherine
Data Pioneer
CatherineData Pioneer
asked in Data Quality & Governance ⚙️
Total Record Volume (all Data Source)

Please advise if there is an easy way to capture how much volume of data in total we have to report up to our CIO? I was thinking creating a plan but that will execute forever.  Any ideas?

332
Catherine
Data Pioneer
5 months ago
Marcel-Jan
Star Blazer L2
Marcel-JanStar Blazer L2
asked in Data Quality & Governance ⚙️
Profiling with row based access

We're currently working on a setup where different customers can see their own data in our Cloudera data lake. So when a customer looks at sample data or profiling data they should only see the row of their own data. I'm currently working on a solution for the profiling data. For this I want a service account for that customer to do the profiling. This service account can also only see the rows for that customer. I was thinking of creating a component for this that reads a database table with lists of customers, url resources and CIs (Hive tables) to profile. And then use a Json call that runs a GraphQL command to run the profiling as the correct service account. Unfortunately the url resource in the Json Call is the only part that cannot be parameterized (as far as I can see). The url resource has the authentication (One Desktop server connection), so that will not be the way to go then. Alternatively I was thinking of using authorization in the HTTP Header. Like this.But some experim

411
A
Ataccamer
5 months ago
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Governance ⚙️
Getting Started with ONE DG: Search for Data Assets 🔍🕵️Best Practice

Hi everyone, Today we will cover how to use the search function in ONE Web Application 🔍Whether you're a business user looking for specific information or a technical user executing complex queries, our search engine has you covered. Let's dive in!There are two areas for search:Global Search: full-text search is enabled on the whole application and is a good starting point for searching entities and locating the area for the future detailed search if necessary. Search in Navigation Sections: combines full-text and filtering capabilities to query entities. Search options are restricted to the currently selected navigation section: for example, using the search bar in Catalog Items will only return results found within that entity and configured areas of search on this entity.1️⃣ Global Search: Discovering Catalogued EntitiesThe Global Search is your go-to tool for searching use cases across cataloged entities. It allows users to explore preconfigured entities like catalog items, terms,

1552
Cansu
Community Manager
5 months ago
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Governance ⚙️
🎙️ Ask me anything: Data quality and governance webinar coming soon 🎙️

Hi community 👋 We are so excited to announce our next event, our first Ask Me Anything session on Data quality and governanceWe are bringing our best experts to gather together and discuss the best practices, use cases, challenges, and workarounds on all things DQG.You'll have the opportunity to ask our Product Managers, Professional Services Consultants, and Community DQG experts anything related to data quality and governance in Ataccama. Whether you're looking for advice on overcoming specific challenges, seeking insights into improving your processes, wanting to broaden your knowledge, or simply finding inspiration from other members, this AMA is the perfect opportunity to connect with like-minded professionals and gain valuable insights.👇Find more about the event and how to register here 👇 Already have your questions lined up? Great! Share with us what you’d like to learn more about, your current challenges in DQG, and anything you’d like us to cover in the comments below.  ⚠️A

531
M
Space Explorer
5 months ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings
ataccama
arrows
Lead your team  forward  OCT 24 / 9AM ET
Register now
×