Skip to main content

Learn & Share

Discuss data, share best practices, learn, and grow with Ataccama's Community.

1077 Topics
konstantinos.patikas
Ataccamer
konstantinos.patikasAtaccamer
published in Master Data Management & Reference Data Management 🛰️
MDM model Pros/Cons of splitting Entity based on TypeBest Practice

1 entity Party→ 2 entities (Person/Organization)or 1 entity Contact → 2 entities (Phone/Email) etc Pros Clean Data Model (on attribute level)In general, we have only columns that are needed for a particular entity (Person) or (Organization). This means that, for example we do see the company_name attribute on the Person records, etc. So in general we have fewer columns in the tables. This could be especially important for the generated MDM native services format (WSDL in SOAP, or all attributes used in the REST API). The consumers of the web services only see attributes per definition of the given model entity and don’t see any irrelevant attributes (similar to the data hiding/encapsulation principle in the Object-Oriented approach).This can also help minimize the area of impact in case of a required model change (the impact to online API change can be countered with custom .online services exposed as wrappers to the MDM native services keeping the API stable with a fixed set of attrib

6881
S
Data Enthusiast
konstantinos.patikas
Ataccamer
konstantinos.patikasAtaccamer
published in Master Data Management & Reference Data Management 🛰️
Setup/Tuning Kafka streaming in MDMBest Practice

IntroductionWe did an internal performance PoC for Streaming. We are reading messages (“msg”) from Kafka or Java messaging service. Parameters In order to tune the performance of the streaming we need to modify 2 main parameters on MDM side:settings.count = how many messages we need to consume in one single batch settings.timeout_s = what is the maximum timeout to wait for this number of messages Average duration of the task vs Records per second We are not aiming for the throughput (records per second) but we are aiming for the average duration of the task. The goal is to run it under 1 sec (1000ms). In various scenarios we have to increase the settings.count parameter starting from 1, 20, 30 etc. Scenarios Below are the different scenarios of using different settings for various tasks/transactions in MDM within 10 mins (600sec)For example of timeout of 1 sec and settings.count = 1000 (scenario , that means that we try to consume 1000 msg at once within this 1 sec. We will consume wha

2890
konstantinos.patikas
Ataccamer
Martin Danek
Ataccamer
Martin DanekAtaccamer
published in Master Data Management & Reference Data Management 🛰️
MDM Implementation StylesBest Practice

Multiple ways to implement your MDM solutionThere are several ways to handle and manage data when deploying a Master Data Management solution. Depending on the approach, we recognize the following styles that can be implemented: Overview of MDM implementation styles REGISTRY StyleIn this style, the various source systems publish their customer data and the subscribing hub stores only the source system IDs, the Foreign Keys (record IDs) and the key data values (needed for matching). The hub runs the cleansing & matching algorithms and assigns unique global identifiers to the matched records, but it does not send any data back to the source systems.Master data distribution is performed downstream by federating the source data. The transformation and consolidation process for building the golden data is performed on demand.This approach is completely non-intrusive and maintains data exclusively in the source applications. This style allows read-only access to the golden data via feder

17071
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
Connecting Oracle Database to ONE DesktopBest Practice

If the ONE Desktop application hangs when connecting to an Oracle database.Adding the following two Java Runtime Parameters to the one-deskop.ini file in the root of your ONE Desktop installation folder.-Djdbc.oracle.useDriverQueryForSchemas=true-Djdbc.oracle.disableListingProcedures=truePossible causes of the delayed, hang or errored Oracle Database connection.When the ONE Desktop application first connects to the Oracle Database we try to retrieve all the metadata beforehand, so we don’t have to connect to the database later (every time). If the Oracle Database contains many Procedures this retrieval process can be extremely slow thus the ONE Desktop gets stuck on this request.Adding the -Djdbc.oracle.disableListingProcedures=true Java Runtime Parameter disables the retrieval of the procedures during the initial connection and you can now connect to the database quickly.In addition adding the -Djdbc.oracle.useDriverQueryForSchemas=true Java Runtime Parameter instructs the connection

1590
DannyRyan
Ataccamer
A
Data Voyager
Akash_SinghData Voyager
asked in Master Data Management & Reference Data Management 🛰️
Group and calculation

Hi,I am creating a plan under IDE to group the records on the basis of two column values and then to calculate the sum of all common records having sales value column. I was using Group aggregator step but no luck. Could you please suggest any help. Thanks 

2351
M
Ataccamer
S
Data Voyager
ShreeData Voyager
published in Master Data Management & Reference Data Management 🛰️
Instant trigger at database tables, if inserts, updates or deletes in RDM data.Reference Data Management

Hello all,I am looking for the live reflection of changes at rdm data and its corresponding tables in the database. As per my understanding from documentation the rdm acts as --    While installation RDM it configures RDM database as well, when ever the new table or any changes to the existing tables done in the data logical model it automatically reflects in database as well, for this process we define the database names while creating a table in data logical model.    RDM can synchronize reference data in its repository with any other defined databases. For this process it required to create separate plans/components, the workflow and need to be scheduled for regular runs by the Task Scheduler. In-fact it seems no mechanism for live updates to the table which is located in external data base. the red color shows the data is published in RDM and waiting to run the scheduling process to affect the changes in the database. The green color confirms the changes after running the schedulin

1352
M
Ataccamer
B
Data Voyager
bazData Voyager
posted in Master Data Management & Reference Data Management 🛰️
Multi Edit problem and work around

Hi,   I’ve noticed an issue within RDM when attempting to update a foreign key values for multiple rows using RDM’s Multi edit functionality.If I attempt to enter a value the “Apply” button never enables and the usual selection list fails to appear If I click on the search magnifying glass  the Lookup window fails to list any values to select from The work around is to add anther non-key field to the multi-edit.. enter a value within it, then enter a new foreign key value and then delete the added non-key field before clicking Save to perform the update. Non-key field added Foreign Key field is now updatable Now delete the non-key field.. it works!! 

983
D
Ataccamer
A
Data Voyager
Akash_SinghData Voyager
published in Master Data Management & Reference Data Management 🛰️
Could you please let me know how we can create a connection to Hadoop environment in IDEONE Desktop

I am not able to open the documentation for the linkhttps://support.ataccama.com/home/docs/aip/13.2.0/big-data-engine.It showing error. Could you please share the link

521
M
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
How to monitor the current execution of a plan?Tutorial

DQA offers a way to monitor the processing visually, providing counts of processed records, percentages on completion of the different operations (steps) and visual cues to the flow of data to identify bottlenecks. This can be achieved by opening a special view of any plan being executed, which will mirror the original plan, showing the processing of records and other aforementioned visual features.To open the monitoring, click the Show progress button, which appears in the Console when the plan is being executed.You can also configure this monitoring window to be opened automatically whenever a plan is executed. The default configuration for the monitoring can be changed by:Open Window > Preferences Navigate to Ataccama DQ Analyzer > Launching > Progress Viewer. Change Automatically open progress viewer for launched plans to Always. You can also change how the Progress Viewer should behave once the execution finishes. 

380
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
What is the maximum number of rows and columns provided in one input to the profiler with drill through on?F.A.Q.

There is no limit on the number of rows written to the drill-through table. If not specified, all rows will be written.The maximum number of columns is not a specific number and depends on the DB. See this Wikipedia page for detailed information by DB type. 

390
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
Is it possible to automate table profile creation?F.A.Q.

Table profile creation can be automated after buying the more advanced Data Quality Center (DQC) software.Schedule a call | Ataccama On the other hand, with Data Quality Analyzer (DQA, free software), the automation is not possible.  

370
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
How to connect to an SQL Server using Windows Authentication?F.A.Q.

There are two ways to do this, using two different driversRight-click Databases > New Database Connection. Connect via the JDBC driver:select MS SQL in the Database type fieldadd integratedSecurity=true to the end of the connection string.For example jdbc:sqlserver://localhost:1433;databaseName=test1;integratedSecurity=true  Connect via the jTDS driver:select jTDS - MS SQL in the Database type fieldspecify a connection string in the following formatjdbc:jtds:sqlserver://localhost:9876/test1;domain=companyfill in Username and Password 

6000
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
How to configure the temporary folder used by DQA (permissions error)?F.A.Q.

You might experience problems when installing DQA on a computer with limited permissions or high restrictions. Even if the installation is done in a folder where you have rights for installation, DQA might fail to execute plans. The reason is that DQA makes use of a temporary folder (usually in the path to the user cache folder) for placing instances of executed plans. This folder is the default Java temporary folder. The execution might throw an exception due to being unable to create a temp plan file.You can override the path to this folder by passing its chosen value as part of the special environment variable JAVA_OPTS , which serves as the Java Virtual Machine arguments. This should be created as follows:Open for editing the dqa.ini file placed in the root folder of the DQA installation, using an editor sensitive to End Of Lines (EOLs) in Windows, WordPad is an option, Notepad++ is another option. Add the variable java.io.tempdir . The variable must be passed using the -D notation

420
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
How to reserve more memory for the DQA IDE?Best Practice

If you experience out of memory errors when processing a large dataset. You can increase the amount of system memory that is available to the Ataccama DQA application.Open dqa.ini located in the root folder of the DQA installation. This file contains the JAVA_OPTS that will be used by the virtual machine on start.Edit Xmx and substitute the number (256 by default) by a higher number for reserving more memory, e.g. -Xmx1024 

480
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
How to reserve more memory for executing plans?Best Practice

DQA plans store data while processing it. This means the more input data there and the more the plan logic multiplies this data to produce desired outputs, the more memory needs to be reserved it.Some of the most memory-intensive steps are Profiling and JDBC Reader.There steps to change the default settings are:Open Window > Preferences > Ataccama DQ Analyzer > Launching. Change the value of the Default memory for launch configuration (MB) to 1024, which is recommended for modern for modern machines. 

550
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
DQ Analyzer for Windows can help you gain valuable insights into your data.Tutorial

Did you know our DQ Analyser for Windows (32 or 64bit) can help you to gain valuable insights into your data, and the application is free to use.Visit https://www.ataccama.com/download/dq-analyzer Some of the features include;Duplicate count Pattern & mask analysis Business rule analysis Numeric statistics Ability to drill down into the data on a per record basis Foreign key analysis Dependency analysis Primary key analysis Concerned about duplicate values in your datasets?Frequency Analysis can tell us that we have a value ‘SYSTEM MIGRATION’ in the customer name column 4551 times. It is clear that these customer names were not transferred into the current system correct - supposedly during previous migration activity.One of my personal favorites for the identification of duplicate records is Group Frequency analysis.Here we can see the same value of 4551 as a group size, with a group count of 1. We can interpret the group analysis as there is 1 value that appears 4551 times.Or usi

1470
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
Ataccama ONE Free ProfilerTutorial

Disruption starts with good data. We believe good data empowers your company to innovate and grow. It’s easy to start—offer a suite of free software for the data people.Ataccama ONE ProfilerProfile a whole database schema with a few clicks or simply drag and drop a file. Profile data directly from your browser.Start profiling data We also have a collection of example data sets and a PostgreSQL database (loaded with example data) for you to explorer in the free profiler.https://app.ataccama.com/welcome/example-datasetsWe have example data sets for various industries including;Automotive Banking Biotechnology Consulting Education Government Healthcare Insurance IT Services Retail / E-commerce Transportation Telecommunications.Lets try the PostgresSQL database as a connection source from the Education example data set.After clicking the Connect to the database button, we are directed to the Data Sources section of the free profiler. Here we can see I can the education_datasource in my lis

1700
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
Data Quality Analyzer Tutorial - 4. Email AnalysisTutorial

Learn how to standardize and extract email addresses from a customer data set. Used functions are regex matching, conditions to separate data flows, and merging data flows. 

480
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
Data Quality Analyzer Tutorial - 3. Advanced ProfilingTutorial

Learn how to configure custom mask analysis, enable and configure drill-through, check your data against business rules, and configure primary key analysis. 

410
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
Data Quality Analyzer Tutorial - 2. Understanding Profiling ResultsTutorial

Understand and interpret the information generated in profiling results with the free Ataccama DQ Analyzer (DQA). 

350
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Data Quality & Catalog ⚙️
Data Quality Analyzer Tutorial - 1. Creating a profileTutorial

Get started with Ataccama DQ Analyzer (DQA) and learn how to create a data profile. 

530
DannyRyan
Ataccamer
M
Ataccamer
Maksim ZhelyazkovAtaccamer
published in Master Data Management & Reference Data Management 🛰️
Configuring MSSQL Windows Integrated Security in GEN1 and Gen2Tutorial

Installing KerberosIn Linux MSSQL with Windows integrated security requires the installation of the Kerberos client. In terminal run the following to install Kerberos on your workstation: yum install krb5-server krb5-libs krb5-workstation Create a krb5.conf file. Below is an example, but be noted that yours may differ: [libdefaults]forwardable = trueticket_lifetime = 24hdns_lookup_realm = falsedns_lookup_kdc = falsedefault_ccache_name = /tmp/krb5cc_%{uid}#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5[domain_realm][logging]default = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.logkdc = FILE:/var/log/krb5kdc.log[realms][EXAMPLE.NET](http://example.net/) = {admin_server = 123.123.123.123kdc = [abc.example.net](http://abc.example.net/)} In terminal run:  kinit -V -c krb5cc firstname.lastname@EXAMPLE.NETEnter your password and wait for the message: Authenticated to Kerberos Authenticate to the SQL server Se

2180
M
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
How to count the number of source systems per master id?Tutorial

If we had the following Instance Data containing the source system and master id groups for each of the instance records and we would like to calculate the count of source per master id. Step by step guide. Create our input data as a text file. Create a new plan and add the following steps to the plan; 1x Text File Reader 1x Representative Creator 2x Text File Writer Connect the steps as follows Configure the Text File Reader to read from the input data we created in step 1. and add 2 shadow columns. These shadow columns could alternatively be added via an Alter Format step. The columns will hold our count per source system (A | B)   Double click on the Representative Creator step and click on the Group By tab. select the master_id as the column to group by.     Add a Rule to the Representative Creator step by clicking on the Add button or double clicking on the asterisk.  Double click on the newly created Attribute rule and then on the Representative Attributes tab

1640
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
ONE Desktop Keyboard ShortcutsF.A.Q.

Below are Ataccama ONE Desktop shortcuts organized into categories by particular viewer or editor.A full list of keyboard shortcuts can be found and called by pressing Ctrl+Shift+L at any time.To edit shortcuts, go to Window > Preferences > General > Keys.Data Viewer SQL Editor Profile Viewer Plan Editor Search and Navigate Edit Layout Run Progress Viewer Online Service Editor Common ShortcutsData ViewerCtrl+F - Find - Opens Find dialog. Ctrl+L - Show Columns - Opens dialog where user selects which columns should be visible and which columns should be hidden. Ctrl+Home - Selects and reveals the top left cell. Ctrl+End - Selects and reveals the bottom right cell.SQL EditorCtrl+F - Find - Opens Find/Replace dialog if SQL query field has focus (from v8). Ctrl+Enter - Execute - Executes current query. Ctrl+R - Read Rest - Reads the rest of the query execution. Ctrl+B - Next Batch - Reads the next batch of the query execution. Esc - Stop - Stops the query execution.Profile View

1060
DannyRyan
Ataccamer
DannyRyan
Ataccamer
DannyRyanHead of Technical Training
published in Master Data Management & Reference Data Management 🛰️
How to calculate Standard Deviation in ONE Desktop?Tutorial

The standard deviation measures the spread of the data about the mean value. It is useful in comparing sets of data which may have the same mean but a different range.For example, the mean of the following two is the same:18 19 20 21 22 = 18+19+20+21+22 / 5 = 2010 15 20 25 30 = 10+15+20+25+30 / 5 = 20However, the second is clearly more spread out. If a set has a low standard deviation, the values are not spread out too much. Lets do the calculations outside of ONE Desktop, so we know our expected result.ExampleFind the standard deviation of 4, 9, 11, 12, 17, 5, 8, 12, 14?First work out the mean: 4+9+11+12+17+5+8+12+14 / 9 = 10.222Now, subtract the mean individually from each of the numbers given and square the result. This is equivalent to the (x - )² step. x refers to the values given in the question.x 4 9 11 12 17 5 8 12 14 (x -  )2 38.7 1.49 0.60 3.16 45.9 27.3 4.94 3.16 14.3 Now add up these results (this is the 'sigma' in the formula): 38.7+1.

910
DannyRyan
Ataccamer

Badge winners

Show all badges

Let's shape the future of Ataccama together

Join User Research at Ataccama!
See new features and prototypes before everyone else, and have an opportunity to shape the future of new products.

Read more
Community Terms of Use & ConditionsCookie settings