Skip to main content

Learn & Share

Discuss data, share best practices, learn, and grow with Ataccama's Community.

876 Topics
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Overwrite the data using Jdbc Writer

Hello,everyone. When you load data into the database using Jdbc Writer and execute the plan every time, it keeps adding the same data from above. How can I overwrite the data? For instance, I have a table like this.I saved it in the database, made some changes to the table, and when I re-executed the plan, it added the changes on top. Duplicates appeared, to put it bluntly.I will be very grateful for your help Thank you!

694
F
Data Pioneer
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
ONE Desktop: Projects Part 2️⃣Best Practice

Hi community! We’re happy to share the second part of our ONE Desktop Projects guide. In this section, we'll delve into essential techniques for project creation, workspace management, and resource configuration. If you have missed the first part you can find it here: Creating a New ProjectBefore you embark on data transformations and analyses, a structured project is your starting point. A project is a collection of plans, components, and data files, organized logically into folders. To initiate a new project:Navigate to the File Explorer. Click on New > Project.   In the ensuing dialog, specify the project's Name and type. Regular projects encompass various categories: Empty project: A blank slate where you define your structure. Project from template: Predefined layout, useful for component projects. Tutorials project: Generates predefined tutorial projects.   Click Finish, and your project will appear in the File Explorer. See the figure below for a template project based

1520
Cansu
Community Manager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Monitoring Projects - Lifecycles 🔁Best Practice

Hi community!We are almost halfway into our Monitoring Project Best Practice series, today we are going to talk about Lifecycles 🎢Let’s dive in! Introduction to Monitoring Project LifecyclesThe introduction of Monitoring Project Lifecycles allows you to simulate a multi-environment process (Configure > Test > Send for approval > Make public, and repeat) within a single physical environment. While you can create two physical environments, the non-production environment should primarily be used for specific purposes, such as testing new sources, integration plans, model changes, and service re-configurations. It's important to note that non-production environments are not intended for developing data quality monitoring projects and migrating between environments. For the best results, we recommend having one production environment for all metadata and a non-production environment for model changes, API work, and testing. This setup differs from other modules like RDM and MDM, w

1062
Cansu
Community Manager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Working with Time Series Data 📈Best Practice

Hello Community! As part of our ongoing Data Quality Best Practice series, today we'll dive into the world of time series data analysis.Time series data analysis is a powerful tool that enables you to unlock valuable insights from your transaction data by identifying trends and detecting anomalies. To make the most of this feature, here are some best practices to follow: Understanding Transaction Data and Time Series AnalysisTransaction data encompasses internal and external events or transactions within a business, often related to financial and logistical records. Common examples include sales orders, invoices, purchase orders, shipping documents, payments, and insurance claims.The benefits of time series analysis include:Visualizing time series data. AI-driven anomaly detection.   Keep in mind that time series analysis supports columns with timestamp information, such as DATETIME, DATE, or DAY data types. Configure and Run Time Series AnalysisTo get started with time series analysis

1252
Cansu
Community Manager
B
Data Voyager
BhumiData Voyager
asked in Data Quality & Catalog ⚙️
Can ‘SQL Select’ stage only be used to connect with SQL Server DB

Can ‘SQL Select’ stage only be used to connect with SQL Server DB ?

753
anna.spakova
Ataccamer
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
ONE Desktop: Projects Part 1️⃣Best Practice

Happy Wednesday community! The Power of Projects: A PrimerBefore you dive into the exciting world of data transformations and analysis, let's lay down the foundation: projects. Think of projects as your organized toolkit – a collection of plans, components, data files, workflows, and custom scripts, all neatly arranged within logical folders. These projects live within a workspace folder, ensuring everything is in its right place. Structuring ProjectsImagine the harmony of a well-structured project. Here's how it can look:A Glance at a Well-Structured Projectbat: Home to batch or shell scripts for triggering plans. bin: Houses DQC plans for data cleansing and deduplication. components: Stores the goldmine of component plans for main plans. data: Nurtures your data directories. ext: Contains external lookup files and replacement lists. bat: Serves batch or shell scripts to trigger plans for lookup files. build: Holds DQC plans for crafting lookups. lkp: The indexed version of looku

790
Cansu
Community Manager
F
Data Pioneer
FCB developerData Pioneer
asked in Data Quality & Catalog ⚙️
Сhanging stewards for all business terms

Hi everyone!How can we avoid manually changing stewards for all business terms and catalog items in Ataccama One? I will be very grateful for your helpThank you!

571
Albert de Ruiter
Star Blazer L2
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
ONE Desktop: Environments 🌍Best Practice

Hi everyone! I hope you are all excited to take one more step into ONE Desktop today 👣Today on the menu we have…..Environments!Imagine effortlessly crafting and testing your solutions across different operational landscapes like DEV, TEST, or PROD. That's what Environments are all about! These versatile tools offer a seamless platform for:Running plans on diverse databases without building separate plans. Testing plans, workflows, and web services across testing and production servers. Crafting and exporting runtime configurations for various environments (perfect for those snazzy folder shortcuts).How Environments Work: A Guided TourEnvironments are your game-changers. Here's a sneak peek at how they operate:Local Launch EnvironmentWhen you're running your plans or workflows on your local machine through ONE Desktop, this environment kicks in. It's your go-to for efficient local testing and development.ONE Platform Launch EnvironmentFor those times when your plans need to be executed

1010
Cansu
Community Manager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
#1 Adding Data SourcesGet certified with Ataccama

👋 Welcome to our new article series, where we will go through the individual topics that are part of the Ataccama Data Quality & Governance Certified Professional: Foundation Level Ataccama ONE v13 certification exam.We will focus on practicing and pointing out the most common mistakes. Our first topic is Adding Data Sources. 🔎 LET’S TEST YOUR SKILLSUsing ONE Desktop Application, create a database connection to your database. ​​Using the ONE Desktop Application, upload the 'bloodtypes.xlsx' into your database. Create the database table <your_name>_bloodtypes, and upload the contents of the 'bloodtypes.xlsx' file. 💡 How to do it?Open Ataccama ONE Desktop Application and create a new projectIn your newly created project, create a plan and name it loaddata.plan Add (drag and drop from the palette on the left side) steps Excel File Reader and Jdbc WriterUpload bloodtypes.xlsx (attached to this article) into your project Open Excel File Reader and fill in File Name, Sheet Name

3741
F
Data Pioneer
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
Greyed out fields appear to change with Enrichment... but they do not (thankfully)!

Howdy Ataccama Community:If you are needing to know how to grey out fields for edit, please navigate here: Grey out fields for edit | Community (ataccama.com)Problem: After greying out fields, I noticed that fields an Enrichment acts on appeared to change in the User Interface, even if greyed out.Solution: Working with our Ataccama professional support person, it was found that although the fields change in the UI, when saved the greyed out fields are not updated in the record being edited. This is great news, as the purpose (at least for our team) of greying out a field, is so that it cannot be changed once created.So that’s it, probably the shortest post I will put on here, but an important one for record reliability.Thank you,Jason R.S.

170
J
Universe Traveller
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Tasks 📃✔️Best Practice

Happy Wednesday everyone!Today’s topic is Tasks! We will cover how to optimize your task management on Ataccama to effectively organize and track your work. Let's dive in! A Glimpse into Task ManagementAt the heart of the Ataccama ONE Platform💜 lies the Task management system. This tool brings together your tasks into a Kanban board, offering an intuitive visual representation of your workload and its progress. From prioritizing tasks to altering their status and engaging in collaborative discussions, this system is your gateway to productivity. Unlocking the Potential of TasksWithin the Task management system, you can accomplish a multitude of actions: Task Creation and Management: Create tasks and their subtasks, assign them to team members, and even solicit assistance from colleagues. Change Management: Request changes like entity creation or access adjustments seamlessly. Workflow Task Management: Efficiently handle tasks related to access requests, approval workflows, and mor

620
Cansu
Community Manager
A
Data Voyager
anagha.kData Voyager
asked in Data Quality & Catalog ⚙️
Where do we get Monitoring project ID?

Hi Team,I want to use the DQ monitoring project results component. From where do we get the  Monitoring project Id for the component?Thanks in AdvanceAnagha 

905
anna.spakova
Ataccamer
alexAguilarMx
Data Pioneer
alexAguilarMxData Pioneer
asked in Master Data Management & Reference Data Management 🛰️
Multi-load operation :: Why data acquisition is slow?

Hi Every one, Why the data acquisition in a multi-load operation could be so slow?Checking the database, the query to get the data is taking less than 10 minutes. Do you have any guidance about it?

462
alexAguilarMx
Data Pioneer
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
Errors in LKP file with multiple columns

Howdy Ataccama community:Today I am creating an LKP file with multiple columns for the first time, and came across a couple rookie mistakes I was making, and wanted to share the warnings and errors I am getting.The file I am making is a simple mapping file going from the key to a field “PH1”Rookie mistake number 1: The file being used to make the lkp is a csv, and I made the assumption the Text File Reader in the lookup builder plan would recognize this from the file type. It did not, it had the separation of columns as \t (tab) still. This created the warning [EHS:SHORT_LINE][step:Text File Reader][line:1] with a row for ever row in my file. To fix this one was simply setting the “Line Separator” field in the Text File Reader to a comma instead of \t.Rookie mistake number 2: After correcting my first mistake, I was now getting the following error “There were logged 1 runtime warnings and 1 runtime errors.” With the details being “Error sending response” with no info on why. I got frus

852
Cansu
Community Manager
B
Data Voyager
babamarData Voyager
asked in Data Quality & Catalog ⚙️
Snowflake key pair authentication

Hello,I’m setting a new connection for Snowflake datasource, and I need to use key pair authentication for this.In the credentials drop-down menu it doesn’t look like this option to be present:  What do you advise?

4581
Cansu
Community Manager
Marcel-Jan
Star Blazer L2
Marcel-JanStar Blazer L2
posted in Data Quality & Catalog ⚙️
Ataccama source monitoring

At our site we have Ataccama on premise and several sources from different data systems. It happened a couple of times that profiling on one of the sources was no longer possible and that we found that out more than a week later. After which it was difficult to track back why this happened.We do have monitoring of all kinds of Ataccama components with Prometheus. But that didn't seem to detect these issues. I was looking for a way to monitor profiling functionality.So what I came up with was a monitoring project that profiles a small table from each source, every hour.Since you can schedule a monitoring project every hour at least, and since you can add notification in case of failure, this gives us warning at most near 1 hours after something breaks.Just adding a catalog item of a table to the MP turned out not to be sufficient though. Because when the monitoring project ran, it did only a structure check, not a profile. To get it profiling, it needed an anomaly check.When you do an a

602
Marcel-Jan
Star Blazer L2
M
Star Blazer L2
Marnix WisselaarStar Blazer L2
asked in Data Quality & Catalog ⚙️
Hiding the Global Settings or at least users for all viewers

We have a question. The basic viewer can view the global settings and with that also a list of users. We'd rather not share this info with the basic viewer. How can we change this?

919
anna.spakova
Ataccamer
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
Grey out fields for edit

Howdy Ataccama Community!Have you ever wanted to have a user see a field while in edit mode, but want to have it greyed out for editing? This problem arose while we were working with a Primary Key that needed to be seen in edit mode but never edited. The only way for a field to be seen in edit mode, is to set it to Modify in the permissions, but we did not want the user to modify. Problem: Show all fields in Edit mode, but only have certain ones editable.Solution: Begin by deploying your project and populating any tables with initial loads as needed. Next go back to OneDesktop IDE, and open the table you are wanting to make some fields non-editable. For the fields you want to make non-editable select Generated. While this sounds like it will auto generate the field, I assure you it does not.Now generate and deploy your project again (remember to generate at the App Configuration level).Note: You might need to stop and restart services for the change to take effect. While I was testing

331
Cansu
Community Manager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Mastering Anomaly Detection in Your Data Catalog📔Best Practice

Hi Community!  Yesterday we took a step into Anomaly Detection, what is it, and how to use it. If you’ve missed that post, go check it out here:Today, we will have a short summary of what we covered yesterday with a focus on the Data Catalog.Spotting the Unseen: Anomaly Detection DemystifiedAnomaly Detection is like having a data detective, tirelessly examining your catalog items for any unusual activities. It operates on two levels:Catalog Item Level: Analyzing key metrics like the number of records and summary statistics. Attribute Level: Delving into value distribution, standard deviation, numeric sum, and more.Whenever a metric's value behaves abnormally, it raises a flag. But wait, there's more! Our AI-driven Anomaly Detection system, fondly known as "ONE," is a quick learner. With every confirmed or dismissed alert, it refines its understanding of your data's unique quirks. This means more precise anomaly detection with each profiling session.Customizing Anomaly Detection to Fit

2100
Cansu
Community Manager
L
Data Enthusiast
lsmall028Data Enthusiast
asked in Data Quality & Catalog ⚙️
AQL for Purpose Tags

Easy question: I used to be able to use this AQL search to look up tables with specific purpose tags, but it’s no longer working with the upgrade to 14.3. Does anyone have a suggestion? Here is the AQL search I used: dataInstances.any(displayName = '[Purpose Tag]')

382
L
Data Enthusiast
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Mastering Anomaly Detection: Unveiling Data Inconsistencies 🔎

Hi everyone!A new week means new best practices! This week we will focus on Anomaly Detection to our new Data Dictionary so if you haven’t started to follow our forums - click on Subscribe now to not miss out on any news 🔔 Easiest way to follow us is by selecting all forums you are interested to follow from our homepage and hit subscribe. Now let’s dive in! Anomaly detection is a pivotal tool for ensuring data accuracy and quality by identifying irregularities and inconsistencies within datasets.Why Anomaly Detection MattersAnomaly detection is your safeguard against data discrepancies and plays a crucial role in maintaining data quality and relevance. With this service, you gain the following advantages:Swift Issue Resolution: Users are promptly notified of potential data discrepancies or pipeline variations, enabling timely problem resolution. Enhanced Data Quality Monitoring: Monitor your dataset's quality to ensure its alignment with actual data conditions. This, in turn, improves

850
Cansu
Community Manager
M
Data Voyager
Mayank VermaData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Looking for a training on Ataccama MDM tool

Hi All,I want to learn the implementation of ataccama MDM. Is there any learning module or instructor who can provide training is available in India.If yes, please do let me know. Thanks,Mayank Verma

371
Cansu
Community Manager
K
Data Pioneer L2
KundanData Pioneer L2
asked in Data Quality & Catalog ⚙️
JSON Parsing

How to extract the root of JSON result which I got from GET api call from JSON CALL step .?I am able to fetch the attributes of root but I am unable to fetch the root itself .

1143
A
Ataccamer
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
Screen flashing when user tries to log in

Howdy all:Earlier this week we started having an issue when end users were trying to log in to our DEV environment. When they would navigate to the URL, their browser would show Ataccama splash screen, then instantly (and continuously) refresh. When asked to try to ping the URL via the command line, the correct IP was being hit, and there was a low latency reply from the server.Note: This issue occurred using Azure AD.Problem: User navigates to a known good URL for Ataccama RDM Web Application. When the application is reached, Ataccama splash screen shows in browser, then browser starts refreshing every 2 to 3 seconds.Solution: It was found that someone accidentally change the setting in KeyCloak called “First Login Flow.” This should have been set to “First Broker Login,” but was switched to “Browser.” Switching back to “First Broker Login” fixed the issue for our users.I am unsure if this is specific to Azure AD, but it is worth a try, if you are seeing a similar issue.Thank you,Jaso

120
J
Universe Traveller
L
Data Voyager
LianeData Voyager
asked in Data Quality & Catalog ⚙️
Which steps do you take to add a DQ rule?

I’d like to check if I am listing the correct steps to add a DQ rule in an existing Monitoring Project, and the order in which to take them. I am quite new to Ataccama so please excuse my ignorance. Can anyone please confirm or adjust them? Many thanks in advance! How to add a DQ ruleCheck if the rule you want to apply, already exists out of the box. For example, if you want to say a certain field cannot be empty, you can apply the predefined rule: String completeness. Other standard checks include validation of email addresses, and URLs. Go to Monitoring Project, tab Configuration & Results Choose the file you want to check (either a VCI or a Parquet File – the latter starts with PF). Choose which attribute you want to check In the 5th column, add: applied DQ checks Doublecheck whether the scoring of the rule is correct (some of the standard rules have a scoring of 0 – you might want to change that) In the upper right hand corner, choose “Publish” Run monitoring. This will take so

962
L
Data Voyager

Badge winners

  • First idea 💡
    sumishahas earned the badge First idea 💡
  • First idea 💡
    BarbaraBhas earned the badge First idea 💡
  • First idea 💡
    jbinghas earned the badge First idea 💡
  • First idea 💡
    samsTmobilehas earned the badge First idea 💡
  • First idea 💡
    ViLashas earned the badge First idea 💡
Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings