Skip to main content

Data Quality & Governance ⚙️

Discuss, learn, ask, suggest, share all things DQG 🔍

  • 552 Topics
  • 1,330 Replies
552 Topics
R
Data Voyager
RoelantVosData Voyager
asked in Data Quality & Governance ⚙️
Reproducing the Term level Data Quality overview from GraphQL

Hi Ataccama community,I’m hoping someone with better grip on the Ataccama GraphQL model can shine a light on this. For internal reporting, we are looking to retrieve data quality details via the Ataccama GraphQL API. One of the requests it to produce an overview of the DQ score across the (rule’s) DQ dimensions. This is basically the same as is shown in the web UI when navigating to the ‘Data Quality’ tab for a Term. It shows the score broken down by dimensions, based on how the rules are defined.What I’m trying to do is retrieve the passed/failed rows at this level, so for each term and each dimension. This would allow me to summarise this into a score per dimension.However,  I am unable to get this out of the API.I’ve approached this from both the ‘terms’ and ‘rules’ pathway. In both approaches I always end up with the ‘Overall Quality’ score for each term - not the per dimension details.The ‘rules’ query looks promising, but ultimately only returns the passed/failed records for the

683
OGordon100
Ataccamer
22 hours ago
E
Data Voyager
evan.sasowskyData Voyager
asked in Data Quality & Governance ⚙️
Date Field Format Data Quality Check

Hey y’all, I was wondering if anyone had a quality rule or advice on how to construct a quality rule to check a date type field for formatting. I know there is the “matches” function for strings and that you can convert a date field to a string using “toString”, but that requires you to specify a format, which invalidates the point of the rule.The desired format is “MM-dd-yyyy”, as shown in what I’ve got so far.matches("^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$",toString(ATTRIBUTE_1, "MM-dd-yyyy"))

201
anna.spakova
Ataccamer
1 day ago
L
Data Voyager
Lindsay OwensData Voyager
asked in Data Quality & Governance ⚙️
Slice Time Ranges

Hi - I wanted more information on how to use the slices vs a SQL Catalog Item.The use case being often when I’m leveraging a slice I want to be able to do a start date of  >2023 , if it is part of a  quality program I want the data to keep updating i.e as new records come in. However, it seems like the slice is only really designed for the ‘previous’ so ignores current records i.e previous year = 2023, and I can’t set the custom date without an end date so it would be limited to a fixed time range.It seems like for the quality program is better served with the SQL CAT ITEM but I was curious if there was value with the slice I was missing or if I could accomplish the same objective with the slice ? Thank you, Lindsay  

151
Cansu
Community Manager
12 days ago
R
Data Voyager
RoelantVosData Voyager
asked in Data Quality & Governance ⚙️
Query the monitoring report results at item level

I’m hoping anyone has some good GraphQL examples to share. For us, it is necessary to integrate Ataccama into our standard reporting. To do so I’m trying to get a GraphQL working that returns the scores for each item in a monitoring project. Ideally, listing out all monitoring projects and items with their scores - for the latest processing (although I’ll take all at this point and filter the old ones later). This is all about the statistics (# records passed/failed etc. not the individual results - I am aware I can create post-processing plans to export data, but this is not applicable here).It’s basically the overview from the report tab in the monitoring project, at item level:I can get everything I need, including the total row count except the passed or failed figure. I have tried and looked at all GraphQL examples everywhere, including ONE API :: Ataccama ONE, and have pieced this together based on the structures visible in the desktop app. Things like DqAggregationResultOverTime

60
R
Data Voyager
13 days ago
may_kwok
Star Blazer L3
may_kwokStar Blazer L3
asked in Data Quality & Governance ⚙️
Setup connection to an SFTP server in ONE Desktop

Hi folks,I’ve been given details to an SFTP server, and I can successfully connect to it using Winscp on my laptop: I tried to setup connection in ONE desktop:I put workflow task:But I get an error:14.11.2024 15:15:21 [INFO] Using following licenses:14.11.2024 15:15:21 [INFO] C:\Users\mkwok\license_hamilton_USER_v14_subscription_2025-01-29.plf14.11.2024 15:15:21 [INFO] ============================================================14.11.2024 15:15:21 [INFO] This is a subscription USER license for developers' laptops for "Hamilton". It expires on 2025-01-29.14.11.2024 15:15:21 [INFO] ============================================================2024-11-14 15:15:21 [INFO] com.ataccama.lib.securityutil.javaencryption.DefaultEncryptionEngine messageId=createDefaultKeyProvider reason=the 'properties.encryption.keystore' property not specified2024-11-14 15:15:21 [INFO] com.ataccama.lib.securityutil.javaencryption.DefaultEncryptionEngine messageId=createDefaultKeyProvid

145
may_kwok
Star Blazer L3
14 minutes ago
T
Data Voyager
ThanujaData Voyager
asked in Data Quality & Governance ⚙️
Audit config in One desktop

Can anyone guide me on how to generate an audit for every action, log it, and integrate the logs with the UI in Ataccama One Desktop? Any best practices or examples would be greatly appreciated!

91
Albert de Ruiter
Star Blazer L2
5 hours ago
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
posted in Data Quality & Governance ⚙️
Business glossary - synonyms (part 5)

Hi Community,The first posts about the business glossary dealt about applying more term types (attribute and entity terms), and how they get context by adding relationship types. Following posts described an implementation for multilingualism and how to apply computed content. Especially the last one was rather tough to digest, so for my last post about the glossary I promise a soft landing.I think one thing needs some further attention and that regards synonyms.Defining a synonym for a term is existing functionality: you can define a synonym by pointing to another term.This means though that when you want to define a synonym for a term, you would have to define a new term first. For that term you would also have to define its metadata. In fact the new term including metadata would be redundant with the original term. Or would you leave out the additional metadata?We chose to use a different approach: for a synonym only its name is relevant, as the other metadata would be the same. So

10
Albert de Ruiter
Star Blazer L2
17 hours ago
R
Data Voyager
ritguptaData Voyager
asked in Data Quality & Governance ⚙️
Post Processing Transformation Plan in One Web

Hi,Can some help me with creating a post processing transformation plan that shows a different row for each DQ rule that has passed/failed on a particular field in the transformation plan in One Web UI. For Eg: I have 2 rules, a uniqueness check and a format check on the ID field. In the post processing output file, what’s the best way to see how I can split this for each rule and see the pass/fail metrics for each rule.Thanks

201
Cansu
Community Manager
22 hours ago
R
Data Voyager
RAZORXData Voyager
asked in Data Quality & Governance ⚙️
Policy Tagging to Business term or any other asset

Hi Team need your expert opinion on how to tag Policies to Business Terms or any other assets(Applying policies to asset types) in 14.5 version? I cannot see any options for the same is it any version limitation or any customization needs to be turned on. Any help is appreciated.

274
R
Data Voyager
1 day ago
B
Data Voyager
BhuviData Voyager
asked in Data Quality & Governance ⚙️
Uniqueness DQ Evaluation rule

I am currently using 14.4.0.one-44751-landing-pa-230616-287787-f83b54a4 ONE web application version. While creating a Uniqueness Rule I am not getting WHEN condition as “is unique” and “is not unique” . Could someone help me understand why I am not getting the uniqueness condition  instead Validity condition.  

221
anna.spakova
Ataccamer
1 day ago
D
Data Voyager
DhavalMData Voyager
asked in Data Quality & Governance ⚙️
How to mark unique queries based on their structure?

I have data coming from a SQL database. For most queries, the structure of these queries remains the same, as shown in the picture, with only the attribute name changing. Now, I want to classify these queries as unique based on their structure rather than the attribute name.Demo DataThank you.Regards,Dhaval 

80
D
Data Voyager
1 day ago
S
Data Pioneer
sumeyraData Pioneer
asked in Data Quality & Governance ⚙️
Writing DQ rule for interval values

Hello all,I want to create a DQ rule, In this rule I need to check a numeric field if the value is in the predefined interval or not. Let me give an example of logic. I have a column X contains ‘types of product’ and column Y contains ‘value of product’. And have a list of valid values. If the value of X is ‘abc’ then Y must be in the interval 1-30. If hte value of X is ‘xyz’ then Y must be in 1-200 and so on.I was going to create a lookup file for this one, but couldn’t be sure how to define intervals. Does someone know how to implement my DQ rule for this logic ? Thanks a lot !

173
T
Data Pioneer L2
1 day ago
N
Data Voyager
NBalaguraData Voyager
asked in Data Quality & Governance ⚙️
Bulk editing of metadata using reference excel file

Hi Ataccama community. Could you please share examples of plans used for adding a new metadata entity and bulk loading values from excel file.We have an excel file with sensitivity level defined for each attribute. We need to add an entity called e.g. sensitivity level and to import this from excel file.  

50
N
Data Voyager
2 days ago
greglvaughan
Data Enthusiast
greglvaughanData Enthusiast
asked in Data Quality & Governance ⚙️
monitoring project processing custom label

Issue: We receive files to be tested against our DQ rules (Monitoring Projects) every 2 weeks.  Each set of files is referenced by its date created (aka 2024-10-01,2024-10-14...).  When running Monitoring Projects against these items we only get the Execution Date which can be different from the file date.   When extracting data for reporting from ONE we only have the Extraction Date and don’t necessarily know which file date the plan results are for (sometimes have to rerun old files for new rules etc..).  Question:  Is there a way to add a custom attribute / parameter for when we run the Monitoring Project (at the processing id level) so we can report our DQ results by File date?  The 2 options we could think of: In extraction component connect to the catalog items and get the file date which is in the table itself (tried this but we need to extract history so would need file date, catalog item, and Monitoring Project processing id to join with DQ Aggregation Results tasks in the com

436
Cansu
Community Manager
2 days ago
E
Data Voyager
evan.sasowskyData Voyager
asked in Data Quality & Governance ⚙️
How long should it take to kill a job in the DPM?

I’ve recently noticed that some jobs I’m trying to kill, jobs which happen to running for much too long (~2 days), take a long time to kill. Is that a bug in the kill function or maybe related to the jobs themselves taking so long to run?

50
E
Data Voyager
5 days ago
K
Ataccamer
KaterinaAtaccamer
published in Data Quality & Governance ⚙️
Enhancing Data Quality Evaluation with Component Rules in ONE Web

Speaking about data quality rules, we usually refer to DQ rule configuration in ONE web application. However, when it comes to referencing external sources or performing preliminary work or combining different sources, we may find that the capabilities within ONE Web are insufficient for our needs. For example, consider a scenario where we need to validate whether our data matches an external data source, such as ABN Lookup on https://abr.business.gov.au/. This is where component rules come in handy.The Task at Hand: Validate ABN Information.To implement this data quality (DQ) check, we require a component rule.Here's how you can configure the ABN validation component and use it for DQ evaluation: ​​​​​​​Configure a placeholder for the component Configuring Logic for the Component using ONE Desktop application Adding Changes to Runtime Configuration in ONE Web Application Configuring the DQ component rule Applying the Rule​​​​​​​Configure a placeholder for the componentNavigate to DQ s

2305
anna.spakova
Ataccamer
9 days ago
C
Data Voyager
ChrisAbdelmalekData Voyager
asked in Data Quality & Governance ⚙️
Exporting Monitoring Projects

Hello everyone, We recently upgraded to version 13.9.3, and gained the ability to export DQ rules from one Ataccama environment and import them into another. This works great, but I need to do the same thing for our monitoring projects. I would like to be able to export them from DEV and import them into a higher environment.  Unfortunately, there doesn’t seem to be a plan available in my version to export monitoring projects. Has anybody figured out a way to migrate monitoring projects without recreating them in each environment? Alternately, I am considering trying to write my own plan. Is there any documentation out there on how to write the JSON for export plans? Thanks!

1955
Cansu
Community Manager
12 days ago
M
Data Enthusiast
manikanta vData Enthusiast
asked in Data Quality & Governance ⚙️
How to Implement Logic in Ataccama to Process One Record and Flag Another as Error for Duplicate Correlation ID and Version Number

Two records as same correation id and same version number one should process another one should error out how to implement this logic in ataccama 

15012
M
Data Enthusiast
12 days ago
R
Data Voyager
ritguptaData Voyager
asked in Data Quality & Governance ⚙️
Create DQ Rule that checks multiple tables

Hi,I want to create a DQ aggregation rule that checks fields from two different tables for the group by clause. Is there a way to do this directly in Ataccama One Web UI.Thanks!

455
R
Data Voyager
12 days ago
R
Data Voyager
ritguptaData Voyager
asked in Data Quality & Governance ⚙️
Access to Transformation Plan

Hi,I am not able to access the transformation plan feature in v15.3 from both the Transformation Plan tab and from the monitoring projects tab. Can someone explain why is there a restriction on creating transformation plans and how can I get access to it?Thanks

412
Cansu
Community Manager
15 days ago
M
Universe Traveller
MelUniverse Traveller
asked in Data Quality & Governance ⚙️
Join a ONE Data table with a Catalog Item in OneWeb

I have a One Data table created by importing data from a .csv file.  I need to use some of the data in this table by joining it with an existing Catalog Item by creating a SQL Catalog Item so that I can implement some rules on it.  Where you select the source when starting to create the SQL Catalog Item I do not get an option to select oneData.  Is it even possible to do this in OneWeb?

364
M
Universe Traveller
15 days ago
S
Universe Traveller
Szymon OlejniczakUniverse Traveller
asked in Data Quality & Governance ⚙️
Debugging - setting breakpoints

Debugging - is this possible to i.e. setup some breakpoint and let the application run all the previous steps before selected one?I like using debug function on steps (CTRL+Shift+D). To debug below Step i need to first debug all the previous steps and then i am able to debug the current one.However, if I happen to update logic on some step from the top i then would need to run & debug all the previous steps before i could debug i.e. Multiplicative Regex Matching.Is this possible to automate the manual work here? I would like to create a breakpoint on Multiplicative Regex Matching and each time i could just click a button and it would self-debug from the start to my breakpoint. I looked in your documentation but didn’t found any solution for this. I found some debugging view in ONE Desktop but it doesn’t allow me to add breakpoints.Do i need to configure my debugger somehow to achieve the above? Szymek

344
Cansu
Community Manager
16 days ago
D
Data Voyager
Devi
asked in Data Quality & Governance ⚙️
Runtime error in JDBC reader: Error creating record

Encountering with Runtime error in JDBC reader, while running the basic profiling for a table from SQL Server. Full Error details -Runtime error in JDBC reader: Error creating record: column type: 'STRING', read value: '<null value>', read object type: '<null value type>', cause: Connection reset Looking for ideas troubleshooting this error.

131
T
Data Pioneer L2
21 days ago
Albert de Ruiter
Star Blazer L2
Albert de RuiterStar Blazer L2
published in Data Quality & Governance ⚙️
Business glossary – computed content (part 4)

Hi Community,The first posts about the business glossary dealt about applying more term types (attribute and entity terms), and how they get context by adding relationship types. The previous post was about implementing multilingualism. The post ended with a remark about computed content, being the topic of this post.To refresh your memory, at the end of the article I showed this screen print, depicting how the translated terms show in the term’s detail page. So like this.Notice that the translation of the term name is preceded by the language code, so it is a concatenation in other words. I will describe how you can define this. Actually to accomplish this I made use of a document that Ataccama provided and that I will add to this article. The document also describes another use-case, so make sure to have a look! Anyhow, this is how the concatenation of the language code and translated term name is done.The calculation is defined in a new metadata model entity. As naming convention yo

650
Cansu
Community Manager
28 days ago
V
Universe Traveller
Vanja GilbertUniverse Traveller
asked in Data Quality & Governance ⚙️
Issue Viewing Rule Logic in Sectioned Report View

Hi,I have a question about the rule logic display when a section is applied. In the report view, I can see the rule logic/implementation when I select a rule. However, once I select a section and do the same, I can only see the valid and invalid results, not the rule logic. Is there a reason behind it? Thank you!

323
S
Ataccamer
28 days ago

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings