Skip to main content

Learn & Share

Discuss data, share best practices, learn, and grow with Ataccama's Community.

877 Topics
L
Data Voyager
LianeData Voyager
asked in Data Quality & Catalog ⚙️
Which steps do you take to add a DQ rule?

I’d like to check if I am listing the correct steps to add a DQ rule in an existing Monitoring Project, and the order in which to take them. I am quite new to Ataccama so please excuse my ignorance. Can anyone please confirm or adjust them? Many thanks in advance! How to add a DQ ruleCheck if the rule you want to apply, already exists out of the box. For example, if you want to say a certain field cannot be empty, you can apply the predefined rule: String completeness. Other standard checks include validation of email addresses, and URLs. Go to Monitoring Project, tab Configuration & Results Choose the file you want to check (either a VCI or a Parquet File – the latter starts with PF). Choose which attribute you want to check In the 5th column, add: applied DQ checks Doublecheck whether the scoring of the rule is correct (some of the standard rules have a scoring of 0 – you might want to change that) In the upper right hand corner, choose “Publish” Run monitoring. This will take so

962
L
Data Voyager
Q
Data Voyager
qingyuData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Supported Data Sources

Hi Team, I want to know if ataccama support the connection to Apache Druid and TigerGraph? If yes, is there any Docs for that?

221
Cansu
Community Manager
A
Data Voyager
anagha.kData Voyager
posted in Data Quality & Catalog ⚙️
DQ Timeliness rule

How to implement Timeliness rule in web application 

682
A
Data Voyager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Monitoring Project - Results Retention 🦾Best Practice

Hi everyone! This week’s last best practice post is on Monitoring Project Results Retention! Managing data retention is crucial for maintenance and security purposes. Whether you want to keep results for a specific duration or limit data storage based on the number of runs, Ataccama's Monitoring Project Results Retention feature has got you covered. Flexible Retention SettingsWith Monitoring Project Results Retention, you have the flexibility to choose how long you want to store the results of your monitoring projects. This can be defined in two ways: Per project configuration: Keep the last X runs. Remove runs after X days. Global configuration (available to power-users only) that takes precedence over project configuration: Keep the last X runs. Remove runs after X days. When global settings are configured, you'll receive notifications in the web application.For more details on configuring global settings, check out Monitoring Projects Global Retention.When Data Will Be DeletedMo

890
Cansu
Community Manager
A
Data Voyager
anagha.kData Voyager
asked in Data Quality & Catalog ⚙️
DQ Accuracy Rule for Date datatype

Hi Team,I have tried to create Accuracy rule for Date datatype to check whether Date is in the standard format. Accurate rule should be as follows:Standard date format is ‘MM-DD-YYYY’. If date is entered in any format other than the above mentioned (‘DD-MM-YYYY’ or ‘YYYY-MM-DD’ or ‘YYYY-DD-MM’) then it should be considered as inaccurate.When I tried using regex expression, I am getting error. Any ideas?

3501
A
Ataccamer
B
Data Voyager
BhumiData Voyager
asked in Master Data Management & Reference Data Management 🛰️
DQI SK stage sample demo

Is there a doc/demo available on how to use DQI SK stage (Analyze category) in Thick client please?

391
A
Ataccamer
M
Data Pioneer
mahesh ArData Pioneer
asked in Master Data Management & Reference Data Management 🛰️
parse multi arrays response value from JSON call

I have to parse the below JSON GET response to get “Headers” and "Name” as columns in Json Call step with Data Streams, How can I perform this? [    {        "Headers": [            “Id”,            “Name”,            “Email”        ],        "ID": “test_111”,        "Rows": [            [                "100”,                “John”,                “John@gmail.com”            ]}]

882
M
Data Pioneer
M
Data Pioneer
mahesh ArData Pioneer
asked in Master Data Management & Reference Data Management 🛰️
How can we integrate python script with Ataccama plan?

My requirement is that I want to read a file in Atacama plan  which is an output of a python script .I would like to integrate python script then plan to run sequentially.

1851
Lisa Kovalskaia
Ataccamer
T
Data Voyager
ThangarajData Voyager
asked in Data Quality & Catalog ⚙️
Thin Client - Catalogue item based access Request

Is there any way that we restrict accessing the data that are imported to Ataccama Thin client ?, as we do role based access , we are able to see all others catalogue items imported into Thin Client.Such catalogue item view breach the data classification compliance which is managed in main data factory where we store the data.Anyone suggest here your thoughts to how restrict the data by giving right access / restriction while we importing the catalogue item or any measure we can take to restrict in viewing the catalogue items within Thin client w.r.t catalogue item import owner?  

441
Lisa Kovalskaia
Ataccamer
S
Data Voyager
sandipilagData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Login issue in ataccama web URL (http://localhost:8050/)

Hi Team,When trying to login in ataccama web URL, receiving below error:Could you please help me here?As all compeonts like DB, leyloack, MDM server, MDM are up and running successfuly.-----------------------------------------------------------Error in MDM server ------------------------------------------------------------------WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by retrofit2.Platform (file:/D:/Ataccama/mdm/lib/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)WARNING: Please consider reporting this to the maintainers of retrofit2.PlatformWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release2023-08-07 15:26:23,797 ←[34mINFO ←[0;39m [][][restartedMain] ←[36mn.r.s.b.l.a.LogbackAccessContext:149←[0;39m - Configured the Logback-access: context=[default], config=[classpat

861
Cansu
Community Manager
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
How can we implement concatenation of the last 10 rows in a column?

How can we implement concatenation of the last 10 rows in a column?I am looking for a way of storing previous values in a variable and adding the current row value to the variable like that.

331
Cansu
Community Manager
R
Data Enthusiast
rupakdeData Enthusiast
asked in Data Quality & Catalog ⚙️
Pass parameter as filter condition in JDBC reader sql

As per requirement I want to read a marker value from a file and pass that value as parameter in select SQL of a JDBC reader to pull only delta records. Is this possible in 13.9.1 version? Please advise with documentation link if available.

1361
A
Ataccamer
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Monitoring Projects - Reports 📝Best Practice

Happy Tuesday community! Today we are continuing with the Monitoring Projects functionalities. What’s on the menu? Reports!We'll walk you through the essential features and insights you can gain from DQ (Data Quality) Reports in this post, helping you keep a close eye on the health of your data. Overview of DQ ReportsDQ Reports display the DQ results and detected anomalies over time for all monitoring runs. They offer a comprehensive view of issues and statistics, allowing you to track changes in DQ across the entire monitoring project, catalog items, specific DQ dimensions, and checks assigned to attributes.As a DQ Steward, you'll be notified whenever a new report is available and have the ability to explore DQ issues and anomalies in greater detail. Viewing the ReportOnce monitoring is complete, you can access the Data Quality results in the Report tab of your project. The report provides different levels of granularity:Overall: A high-level summary of the entire monitoring project.

1180
Cansu
Community Manager
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
How do I view Enrichment Online XML

Howdy all!I said in another post today that it was going to be short. It turned a little longer than I expected. This one might be a little shorter.Question: How do I find the XML that makes up my “.online” files for Enrichments?Steps:Start all your services, and ensure you have deployed your Configuration in the Admin consol (localhost:8060/admin) Go to localhost:8061 Under Server, select Applications Click on the Application “.online” file you are wanting to see the XML for Click the service you want to view the XML for in the next screen Then select the link for the XML SchemaThe data in here can help to find errors in the setup of the service! Enjoy!Jason R.S.

101
Cansu
Community Manager
S
Data Voyager
sandipilagData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Ataccama login issue

Hi All,Receving below error when trying to login in ataccama URL: Need help on this? WARNING: An illegal reflective access operation has occurredWARNING: Illegal reflective access by retrofit2.Platform (file:/D:/Ataccama/mdm/lib/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)WARNING: Please consider reporting this to the maintainers of retrofit2.PlatformWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future release2023-08-07 15:26:23,797 ←[34mINFO ←[0;39m [][][restartedMain] ←[36mn.r.s.b.l.a.LogbackAccessContext:149←[0;39m - Configured the Logback-access: context=[default], config=[classpath:logback-access.xml]2023-08-07 15:26:25,274 ←[34mINFO ←[0;39m [][][restartedMain] ←[36mc.a.l.s.s.e.KeyProvidersSpringBootAutoConfiguration:41←[0;39m - messageId=createDefaultKeyProvider encryption=INTERNAL, reason=the 'internal.encryption.keyst

291
Cansu
Community Manager
B
Data Voyager
BhumiData Voyager
asked in Data Quality & Catalog ⚙️
How to implement a DQ rule to check if an attribute's value is present in another attribute (for eg. Attribute1 'Product' value is present in Attribute2 'Quantity' field) using Thin client

How to implement a DQ rule to check if an attribute's value is present in another attribute (for eg. Attribute1 'Product' value is present in Attribute2 'Quantity' field) using Thin client

341
B
Data Voyager
S
Data Voyager
sandipilagData Voyager
asked in Master Data Management & Reference Data Management 🛰️
.metadata folder not visible

Hi All,Require quick help on ataccama IDE, I am not able to see .metadata folder in IDE. What is issue here?Please find attached screen shot ?Please help me here? 

252
S
Data Voyager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Import Monitoring Project Configurations Part 3️⃣Best Practice

Hi everyone! In the last part of Monitoring Projects Configurations, we will talk about how to easily import configurations from one monitoring project to another, making your data quality monitoring more efficient. If you’ve missed Part 1&2 you can find them below:Import the Configuration Instead of creating a new project from scratch, you can copy the configuration of one monitoring project and apply it to another. This is particularly useful when you want to replicate settings from a testing environment to a production environment. The copied configuration includes all checks, invalid samples, reports structure, and DQ filters.To import the configuration:Navigate to the project where you want to apply the configuration.   Expand the more options menu and select Import configuration. Choose the monitoring project from which you want to copy the configuration, from the list provided. Check if the preselected catalog items are correct. You can modify the list by deleting items or

691
ivan.kozlov
Ataccamer
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
Pivot every 13 rows into one row in target.

The source csv file has splitted into 13 rows for each actual row in target.By using Ataccama one desktop we would like to pivot the data by sending source 13 rows into one row in target. I am trying to implement the logic below in the Text file reader step but it's not working . v_RowNum =  iif(v_RowNum IS  NULL,0,v_RowNum+1)                    -------- Initialize v_RowNum to 0 if NULL; otherwise, increment it by 1 for each row.v_CurrentNumber =  iif(v_CurrentNumber%13= 0, v_CurrentNumber+1,  v_CurrentNumber  ).    -- Increment v_CurrentNumber by 1 for every 10 rows.GENERATED_NUMBER= v_CurrentNumber GENERATED_NUMBER INTEGER   Please  help us  to make it work in Ataccama one desktop .

642
M
Data Pioneer
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Monitoring Project Configuration Part 2️⃣Best Practice

Hi everyone! In this continuation of our best practices post on Monitoring Project Configuration, we will explore additional settings and actions to ensure seamless data quality monitoring.Check out the part one of the series if you’ve missed it:Disable Rule SuggestionsPower users can disable rule suggestions in monitoring projects. For more details, refer to the link provided in the text. Set Partition DefaultsFor partitioned tables in monitoring projects, identify them with the provided icon. Define default analysis for partitioned tables using the more options icon and selecting either Last partition analysis as default or Full analysis as default. These settings will be applied when running the monitoring project. See the link provided for more information. Anomaly DetectionWhen running anomaly detection during monitoring projects, choose between the time-dependent model or time-independent model. Configure the sensitivity of the detection as well. For further details, check out th

910
Cansu
Community Manager
P
Data Pioneer L2
pdanpoonkijData Pioneer L2
asked in Data Quality & Catalog ⚙️
Questions for Ataccama One Power BI connector

Hello community,I’m creating a Power BI source on Ataccama One and have a few queries:Is the built-in Power BI connection allow One to retrieve a Power BI datasets? Or this is strictly for only bringing the Power BI reports to One? The imported catalog items from this connection shows “0” number of attributes and “-” number of records (screenshots below). This is also presents in the the screenshot from the Ataccama support page (link). Is this behave as intended or am I missing anything here?Thank you in advance.Ben

1762
P
Data Pioneer L2
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
Pivoting all rows into columns in target.

My input file is in the below structure and looking to pivot all rows into columns. No name 1 value1 2 value2 3 value3 4 value4 5 value5    Expecting output should be like  Col1 Col2 Col3 Col4 Col5 value1 value2 value3 value4 value5   What will be the best approach in Ataccama One desktop? Looking for support and help..

1744
A
Ataccamer
M
Data Pioneer
mahesh ArData Pioneer
asked in Data Quality & Catalog ⚙️
I have to write below Multi array list in to a file through JSON call.How can I perform this?

Json : [   {      "Headers":[         "Empid",         "Name",         "dept",         "EMAIL"      ],      "ID":"Table_112",      "Rows":[         [            "100",            "John",            "Finance",            "John@gmail.com"         ],         [            "101",            "Smith",            "HR",            "Smith@gmail.com"         ],         [            "102",            "Derek",            "Marketing",            "Derek@gmail.com"         ]      ]   }] Expecting the output in below format:Empid Name dept EMAIL ID 100 John Finance John@gmail.com Table_112 101 Smith HR Smith@gmail.com   102 Derek Marketing Derek@gmail.com      I have created a plan with below details:After running a plan with JSON CALL step below response file got created.   Based on response file i have given below columns in Data Streams    I am not getting the out put what i expected. Appreciate your help and support . 

1961
Lisa Kovalskaia
Ataccamer
J
Universe Traveller
Jason.SupticUniverse Traveller
posted in Master Data Management & Reference Data Management 🛰️
EXTRA_DATA error or warning

The EXTRA_DATA error/warning was a thorn in my side for the last couple of days. I added a column to a dataset I was loading to a model from a Text File Reader, and I could not figure out for the life of me what I was doing different from before the column add. It turned out, where I was trying to unselect data was too early.Problem: Suppose you are bringing in a delimited file to use in a Text File Reader that has an extra field at the beginning, which you don't want to have in your output. Finding the correct place to disable this data can difficult. If you disable it before your Text File Reader, or simply don't bring it into your Text File Reader, then you run the risk of getting an error or warning that states EXTRA_DATA for every row you try to bring in, as seen below.I have seen the EXTRA_DATA error in 3 scenariosDeselecting the “Use” box in the Metadata definition on columns you don’t want. Selecting all columns in the Metadata definition, but not bringing the column into your

321
Cansu
Community Manager
Cansu
Community Manager
CansuCommunity Manager
published in Data Quality & Catalog ⚙️
Getting Started with Monitoring Projects 🎢Best Practice

Hi Community!In the next few days, we will discuss Monitoring Projects, a powerful tool to evaluate and monitor data quality over time for selected catalog items. Monitoring projects use DQ checks, automatic anomaly detection, and structure checks to maintain data quality. We also have a short video introduction to Monitoring Projects, go check it out here 👇 If you are ready, let's dive in 🤿 Accessing Monitoring ProjectsTo access Monitoring Projects:Go to the left-hand menu and select Data Quality. Choose Monitoring Projects.     In this section, you'll find all configured projects. If you want to start a new project, click Create.Each project listed will show the following information:Name: The project's name. Terms: The glossary terms associated with the project. Items: The number of items monitored within the project. Latest run: The date of the project's last evaluation.   The current state of a project is indicated through row highlighting, showing if it's published, recently ad

1420
Cansu
Community Manager

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings