Data Quality & Data Governance ⚙️

Discuss, learn, ask, suggest, share all things DQ and DG 🔍

  • 275 Topics
  • 533 Replies

275 Topics

Albert de Ruiter
Albert de RuiterRocket Pioneer II
asked in Data Quality & Data Governance ⚙️

Large button on UI for embedded array

Hi community,In the metamodel, when defining a relation to another antity of type embedded object array, in the user interface it appears as a very large white area containing one button. We feel that this feature is not in line wth proper user interface design. Therefore I am looking for a way to have this button shown in a way more minimal format.Does anyone knows if this is possible? A trait or a specification in the page lay-out definition or…?Thanks in advance!Kind regards, Albert

4
Albert de Ruiter
5 hours ago
M
Marnix WisselaarRocket Pioneer II
asked in Data Quality & Data Governance ⚙️

Working nested GraphQL results in a JSON Call step in desktop

Hi,There is a lot about GraphQL and JSON-call. But I have question on how to process nested GraphQL in a JSON and maintain integrity when using (nested) datastreams) in the JSON Call step in Ataccama (v13.x).Below an example. There are profiles with path ($.data.catalogItem.publishedVersion.profilingConfigurationInstances.edges.node.publishedVersion.profiles.edges.node.@). And then a nested data-stream, which continus from this path (publishedVersion.attributeProfiles.edges.node). Each stream will give an output for the JSON-caller step. Now how do we maintain integrity between the streams/output files? So how can I refer from a sub-stream to a parent key, so I can later join the JSON-call output streams?    

2
A
9 hours ago
S
sgillaRocket Pioneer
asked in Data Quality & Data Governance ⚙️

DQS - camel case

Is there a function or custom ways to convert strings into camel case on DQS - Data Quality services?

4
A
21 hours ago
L
lsmall028Space Explorer
asked in Data Quality & Data Governance ⚙️

Exporting Profiling Results from Web App

I am working on version 14.3 and I am trying to export the profiling results from the Web App, but when I go thru the process I only receive the attributes with records. I am wanting to export the profiling and not the records since they hold PHI. Is this possible in the web app or only in desktop?

0
L
2 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

Monitoring Project Configuration Part 1️⃣Best Practice

Happy Tuesday Community! We started this week by sharing best practices on Monitoring Projects, and today we’ll be continuing with how to configure Monitoring Projects. If you have missed the first post you can find it here:Let's explore the best practices for configuring Monitoring Projects to ensure robust data quality checks and anomaly detection.Select Catalog ItemsGo to the Configuration & Results tab. To add catalog items: For a new project, select Add catalog items.   To add more items to an existing project, select +Add under Items to Monitor.   To remove an item, use the more options icon and select Delete. The listed catalog items will display a summary of: Structure: Aggregated results for structure checks on attributes. Anomalies: Detected anomalies on attributes. DQ Checks: Results of data quality checks assigned to attributes. Overall: Aggregated quality based on assigned DQ checks contributing to overall quality. Individual DQ dimensions, listed with the

301
R
2 days ago
F
FCB developerRocket Pioneer
asked in Data Quality & Data Governance ⚙️

To find attributes that are not linked to business terms.

Нello, everyoneI need to find attributes that are not linked business terms specifically with the name of tables and attributes. Using the "Filter" tab, I wrote such a query schema is 'cli_dwh' and attributes.any(termInstances.count() == 0) using step ONE Metadata Reader, but it seems it takes only the schema but skips the business term, gives a list of attributes that are already linked to the business term. Can you give advice on this problem?I will be very grateful for your help🤗Thank you!

0
F
3 days ago
jhoppesch
jhoppeschSpace Explorer
asked in Data Quality & Data Governance ⚙️

Pass column from DQC plan to workflow

I have a workflow which does an HTTP GET to download an HTML page. I then use a DQC plan to filter and alter the data in the HTML file to get a “new” URL. I now have this “new” URL in a column at the end of the plan. I’m wondering, is it possible to use this “new” URL column from the DQC in the same workflow to perform another HTTP GET of the “new” URL? 

2
M
3 days ago
M
Marnix WisselaarRocket Pioneer II
asked in Data Quality & Data Governance ⚙️

Filter graphql output

Hi, we need to see if for terms (or other entities) certain roles have viewer rights. (...and if not set them)Below is a GraphQL, but I need to filter only the term.assignedIdenties.capability.OperationSet where the name is "viewer”. How would I go about doing this? (outside filtering in ONE Desktop of course)query GetCapabilities_term($gid: GID!) {  term(gid: $gid) {    gid    assignedIdentities {      grantingStrategy      capability {        name        id        operationSets {          name          displayName          id        }      }      identity {        ... on Role {          name          __typename          ... on GroupRole {            name            numberOfUsers            gid          }        }      }    }  }}

4
M
3 days ago
F
FCB developerRocket Pioneer
asked in Data Quality & Data Governance ⚙️

How to find the completeness of the descriptions for all the tables and their attributes

Good day! I would like to find the completeness of the descriptions for all the tables and their attributes in Ataccama Web. Is it possible to do this in AtaccamaWeb using Rules? There are approximately 700 tables in the system.  I will be very grateful for your helpThank you!

8
F
4 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Data Quality Results 🧑‍🔬

Hi Community! Are you looking to retrieve Data Quality (DQ) results from monitoring projects or catalogs using Ataccama's ONE Desktop? We have created a set of easy-to-follow steps to help you achieve this efficiently. These steps will mimic the API calls created for the same purpose and return DQ results. Let’s get into it 🔮 PrerequisitesBefore diving into the process, make sure you have the following prerequisites in place: Active Server Connection: Ensure you are connected to the ONE Web Application. Input Data: You need the data for which you want to retrieve results. You can use either the appropriate Input step or the MMM Reader step. You can add input data by dragging and dropping files onto the canvas or by configuring the steps manually. If multiple input steps are required, use the Join step and create connections accordingly.   Note: Depending on the step you wish to use to retrieve results, different parameters will be required in the input. Refer to the individual step

281
S
4 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

Collibra Integration in Ataccama ONE Desktop 🏹

Hi everyone, Today’s post is specifically on Collibra! So if you work with Collibra and are curious about how to integrate it with Ataccama ONE Desktop read on. This post will cover how you can harness the power of the Collibra Reader and Collibra Writer steps to effortlessly manage Data Quality (DQ) Rules. PrerequisitesBefore we dive in, ensure you have the following essentials:Ataccama ONE Desktop: Make sure it's installed on your machine with a valid license. Collibra: You need an active Collibra instance, complete with valid credentials for accessing and managing assets.Setting Up a Collibra Server ConnectionLet's establish a connection to Collibra from ONE Desktop:In ONE Desktop, right-click on Servers, and choose New > Server.   Fill in the details: Implementation: Choose Generic. Name: Give your connection a meaningful name. URL: Provide the URL of your Collibra instance. Authentication: Select Basic. Username: Enter your Collibra username. Password: Input your Collibra

100
Cansu
6 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Remote Plan Execution 🦾

Happy Monday community! Today we are going to talk about Remote Plans. Let’s start with the basics, what are remote plan executions? The remote plan execution lets you run any ONE plan on the Ataccama ONE Platform. Plans are typically developed and tested on small data sets or on dedicated environments. This helps you validate the plans and assess how well they perform so that you can make the necessary adjustments before applying them on your actual data. Once the plan works as expected, you can run it remotely on production data in ONE.Jobs are first sent to the Data Processing Manager (DPM) module, which forwards the job information to a suitable Data Processing Engine (DPE). Actual data processing takes place only in DPE instances. How to configure Remote Plan ExecutionRemote plan execution leverages existing controls and concepts of ONE Desktop. Once you connect to the Ataccama ONE Platform, you can choose between the following launch options:Local: A suitable DPE is selected auto

80
Cansu
6 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Nested Components 🪹

Hi everyone! This week’s last best practice is also the final post on components! Check out yesterday’s post to catch on if you have missed the series so far:  Nested components are like puzzle pieces that fit together to create sophisticated data transformations within a single entity. In this guide, we'll explore how to work with nested components in Ataccama ONE Desktop, focusing on their creation, import, configuration, and use cases. Understanding Nested ComponentsImagine having a main (parent) component that uses multiple component files to achieve complex data transformations. Nested components enable you to do just that. This approach allows you to maintain the original data while applying intricate logic to your datasets. For instance, you can use predefined components provided by Ataccama ONE Desktop to cleanse, filter, or standardize specific data attributes, all while preserving the integrity of your original data. Configuring a Nested Component in ONE DesktopIn Ataccama ON

141
M
6 days ago
S
ShruthiSpace Explorer
asked in Data Quality & Data Governance ⚙️

Anomaly detection based on more than one column

Is there a way I can get the anomaly detection configured and notified when there is a anomaly in the counts in more than one attribute?eg:SELECT CALENDARDAY, COMPETITOR, COUNT(*) as count FROM TablenameGROUP BY CALENDARDAY, COMPETITORA notification whenever there is drop or increase in the count based on CALENDARDAY, COMPETITOR

0
S
7 days ago
M
mahesh ArRocket Pioneer
asked in Data Quality & Data Governance ⚙️

How do we connect Workday objects  from Onedesktop by using CData drivers

We successfully connected Workday from Atacccama one web by using CData drivers.Now we are trying to do the same from One Desktop.How do we connect Workday objects  from Onedesktop by using CData drivers?We were able to connect Workday  through  the API's  but our requirement demands CDATA drivers connectivity.Thanks,Mahesh Arigela

1
Cansu
11 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Component Steps Part 2️⃣Best Practice

Happy Wednesday Community! We are halfway into the week and ready with Part 2 of Component Steps. If you have missed Part 1 don’t worry, check it out on the link below 🔽 In the second part of our journey into Component Steps, we will delve deeper into their creation, configuration, parameter mapping, running from the command line, and working with nested components. Creating a ComponentBuilding a component step is akin to constructing a plan, but certain rules dictate its placement within a plan: At the Beginning: A component at the start of a plan should commence with an input step connected to actual data, not Integration Input. In the Middle: For components in the middle of a plan, they must have at least one input endpoint with Integration Input and one output endpoint with Integration Output. At the End: Components concluding a plan must start with an input endpoint with Integration Input and conclude with an output step that isn't Integration Output (ensuring data is written

60
Cansu
11 days ago
S
SaranyaTSpace Explorer II
asked in Data Quality & Data Governance ⚙️

Time series analysis in V13.9

Any one who has worked on Time series analysis option in V13.9 and successfully got the required results

2
S
12 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Component Steps Part 1️⃣

 Happy Tuesday community 🙋‍♀️ This week we have a series of best practices on all things components on ONE Desktop. Component steps, often referred to as simply "components," are instrumental tools that come with unique capabilities. Here, we'll explore what makes components special and how to make the most of them in your data projects. What Sets Components ApartComponents are akin to plans but with distinctive attributes: Reusable: Components can be employed repeatedly, functioning like steps within other plan files. Parameterization: Components can have properties within their constituent steps mapped as parameters, allowing external plans to interact with them (learn more in the section on "Mapping Properties as Parameters"). Console Launch: Components can be executed directly from the console, with parameters for added flexibility. Example: The Component in ActionComponent files are identifiable by their ".comp" extension and can play pivotal roles within your data workflows.

80
Cansu
12 days ago
B
BhumiStarblazer
asked in Data Quality & Data Governance ⚙️

Is event based trigger possible in Ataccama Thin client

Hi Community,Is event based trigger possible in Ataccama Thin client (web version)? If so, how to implement it?Thanks!

0
B
13 days ago
F
ferozSpace Explorer
asked in Data Quality & Data Governance ⚙️

S3 based data Catalog in Ataccama oneWeb

How I can use create sql catalog on s3 files (csv, parquet) in Ataccama One Web Knowledge Catalog.Do Ataccama use AWS S3 CData drivers

0
F
16 days ago
C
CharishmaSpace Explorer
asked in Data Quality & Data Governance ⚙️

Json writer

Hi ,I want to know what was the input to be given in root object for JSON writer  

1
Cansu
17 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Frequently Used Steps and Expressions 🪜

Hi community, happy Tuesday! Today, we are covering one of the essential functions of ONE Desktop - Expressions & Steps ✨Steps and functions enable you to craft plan files tailored to your project's unique needs. Here, we provide a comprehensive overview of the frequently used steps and functions, guiding you towards best practices.Frequently Used Steps:Flow Control StepsData Parsing StepsAnalysis StepsMatch and Merge StepsUsing FunctionsWithin these steps, you can leverage various functions to further enhance your data processing capabilities. Here are some commonly used functions:matches: Performs a full match of input data with a regular expression, returning true or false. find: Conducts a partial match of a regular expression within the input string, returning true or false. substr: Extracts a substring from the input string, starting from zero.Using Regular ExpressionsAtaccama's products support regular expressions for powerful pattern matching. Here are some fundamental regu

200
Cansu
20 days ago
F
FionaFSpace Explorer
asked in Data Quality & Data Governance ⚙️

Integrate Date function into a string datatype while creating rule in ONE Web

Needed to know how to create a rule having an attribute of STRING datatype for extracting current date and past date in One Web. Tried using date functions in string using advanced expression (today() is ATTRIBUTE) ..showing error in testcase. But when tried using same functions for DATETIME datatype working fine. Can anyone help?

0
F
20 days ago
Maxim Kim
Maxim KimAtaccamer
posted in Data Quality & Data Governance ⚙️

Export Data Quality results

Eventually, you might need all your data quality results evaluated in One monitoring projects or within regular catalog items (via business terms and related DQ rules) exported to some reporting system or into excel. And it would be nice if it looked like this:  TLDR; There are 2 components attached you can use to export DQ results, one for monitoring projects, another one for regular catalog items. Monitoring ProjectsThe idea is to read all monitoring projects metadata, join it with their check results as well as rules names, catalog item names and voilà, we can save it to excel/csv or to some database table.It sounds like something very simple, and indeed there is no black magic, but nonetheless component looks a bit more elaborated: Catalog Items DQ (based on Term evaluations)The idea is to find all catalog items that have aggregated overall data quality and join them with the DQ rules results:This component has a bit simpler implementation:  Of course, the output of the components

6
M
20 days ago
Cansu
CansuCommunity Manager
published in Data Quality & Data Governance ⚙️

ONE Desktop: Data Quality Scoring 🧮

Happy Monday Community! We are continuing our ONE Desktop best practices series this week as well! Today on the menu is - DQ Scoring!Data quality scoring is a vital process in Ataccama Desktop, enabling you to assess data quality based on specific rules and assign a numerical score. This score helps differentiate between high-quality and low-quality records, making it an essential step in data processing.Here are the best practices for data quality scoring in Ataccama Desktop:1. Categorize Scoring into Four LevelsTo effectively score data quality, categorize your cleansing scores into four quantification levels:Scoring Level Scoring Description Scoring Result 0 No modification was done to the input value. VALID < 10,000 Small modification was done to the input value. VALID/CORRECTED < 10,000,000 Huge modification was done to the input value. UNSAFE/UNKNOWN > 10,000,000 Input value or pre-cleansed value is null. NOT VALID Each cleansing operation, as de

220
Cansu
21 days ago

Badge winners

  • Start engines
    hamkhan.chas earned the badge Start engines
  • Liftoff
    Shruthihas earned the badge Liftoff
  • Liftoff
    ferozhas earned the badge Liftoff
  • Crew member bronze
    Ojaswinihas earned the badge Crew member bronze
  • Liftoff
    FionaFhas earned the badge Liftoff
Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Terms & ConditionsCookie settings