Learn & Share

If you experience out of memory errors when processing a large dataset. You can increase the amount of system memory that is available to the Ataccama DQA application.Open dqa.ini located in the root folder of the DQA installation. This file contains the JAVA_OPTS that will be used by the virtual machine on start.Edit Xmx and substitute the number (256 by default) by a higher number for reserving more memory, e.g. -Xmx1024 

DQA plans store data while processing it. This means the more input data there and the more the plan logic multiplies this data to produce desired outputs, the more memory needs to be reserved it.Some of the most memory-intensive steps are Profiling and JDBC Reader.There steps to change the default settings are:Open Window > Preferences > Ataccama DQ Analyzer > Launching. Change the value of the Default memory for launch configuration (MB) to 1024, which is recommended for modern for modern machines. 

Did you know our DQ Analyser for Windows (32 or 64bit) can help you to gain valuable insights into your data, and the application is free to use.Visit https://www.ataccama.com/download/dq-analyzer Some of the features include;Duplicate count Pattern & mask analysis Business rule analysis Numeric statistics Ability to drill down into the data on a per record basis Foreign key analysis Dependency analysis Primary key analysis Concerned about duplicate values in your datasets?Frequency Analysis can tell us that we have a value ‘SYSTEM MIGRATION’ in the customer name column 4551 times. It is clear that these customer names were not transferred into the current system correct - supposedly during previous migration activity.One of my personal favorites for the identification of duplicate records is Group Frequency analysis.Here we can see the same value of 4551 as a group size, with a group count of 1. We can interpret the group analysis as there is 1 value that appears 4551 times.Or usi

Disruption starts with good data. We believe good data empowers your company to innovate and grow. It’s easy to start—offer a suite of free software for the data people.Ataccama ONE ProfilerProfile a whole database schema with a few clicks or simply drag and drop a file. Profile data directly from your browser.Start profiling data We also have a collection of example data sets and a PostgreSQL database (loaded with example data) for you to explorer in the free profiler.https://app.ataccama.com/welcome/example-datasetsWe have example data sets for various industries including;Automotive Banking Biotechnology Consulting Education Government Healthcare Insurance IT Services Retail / E-commerce Transportation Telecommunications.Lets try the PostgresSQL database as a connection source from the Education example data set.After clicking the Connect to the database button, we are directed to the Data Sources section of the free profiler. Here we can see I can the education_datasource in my lis

Learn how to standardize and extract email addresses from a customer data set. Used functions are regex matching, conditions to separate data flows, and merging data flows. 

Learn how to configure custom mask analysis, enable and configure drill-through, check your data against business rules, and configure primary key analysis. 

Understand and interpret the information generated in profiling results with the free Ataccama DQ Analyzer (DQA). 

Get started with Ataccama DQ Analyzer (DQA) and learn how to create a data profile. 

Installing KerberosIn Linux MSSQL with Windows integrated security requires the installation of the Kerberos client. In terminal run the following to install Kerberos on your workstation: yum install krb5-server krb5-libs krb5-workstation Create a krb5.conf file. Below is an example, but be noted that yours may differ: [libdefaults]forwardable = trueticket_lifetime = 24hdns_lookup_realm = falsedns_lookup_kdc = falsedefault_ccache_name = /tmp/krb5cc_%{uid}#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5[domain_realm][logging]default = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.logkdc = FILE:/var/log/krb5kdc.log[realms][EXAMPLE.NET](http://example.net/) = {admin_server = 123.123.123.123kdc = [abc.example.net](http://abc.example.net/)} In terminal run:  kinit -V -c krb5cc firstname.lastname@EXAMPLE.NETEnter your password and wait for the message: Authenticated to Kerberos Authenticate to the SQL server Se

If we had the following Instance Data containing the source system and master id groups for each of the instance records and we would like to calculate the count of source per master id. Step by step guide. Create our input data as a text file. Create a new plan and add the following steps to the plan; 1x Text File Reader 1x Representative Creator 2x Text File Writer Connect the steps as follows Configure the Text File Reader to read from the input data we created in step 1. and add 2 shadow columns. These shadow columns could alternatively be added via an Alter Format step. The columns will hold our count per source system (A | B)   Double click on the Representative Creator step and click on the Group By tab. select the master_id as the column to group by.     Add a Rule to the Representative Creator step by clicking on the Add button or double clicking on the asterisk.  Double click on the newly created Attribute rule and then on the Representative Attributes tab

The standard deviation measures the spread of the data about the mean value. It is useful in comparing sets of data which may have the same mean but a different range.For example, the mean of the following two is the same:18 19 20 21 22 = 18+19+20+21+22 / 5 = 2010 15 20 25 30 = 10+15+20+25+30 / 5 = 20However, the second is clearly more spread out. If a set has a low standard deviation, the values are not spread out too much. Lets do the calculations outside of ONE Desktop, so we know our expected result.ExampleFind the standard deviation of 4, 9, 11, 12, 17, 5, 8, 12, 14?First work out the mean: 4+9+11+12+17+5+8+12+14 / 9 = 10.222Now, subtract the mean individually from each of the numbers given and square the result. This is equivalent to the (x - )² step. x refers to the values given in the question.x 4 9 11 12 17 5 8 12 14 (x -  )2 38.7 1.49 0.60 3.16 45.9 27.3 4.94 3.16 14.3 Now add up these results (this is the 'sigma' in the formula): 38.7+1.

Did you know that you can setup email notifications when certain events occur in your DQ Monitoring Projects? There are 4 types of notifications that can be assigned to events; Notification + Email sends both an in-app notification and an email to the recipients in case of the event occurrence. Email sends an email to the recipients in case of the event occurrence. Notification sends an in-app notification to the recipients in case of the event occurrence. Leave the events notification options blank if you do not want to be notified of a particular event occurrence. Process Steps Configure the MMM Module for generating links in emails.Ensure the MMM Module has been configured for SMTP (There is a link to the documentation at the end of this post)Underneath your existing configuration for plugin.email.ataccama.one.addressAdd a new configuration line for plugin.email.ataccama.one.base.url which is the base URL for generating links in emails. The default value is http://localhost:80

In this Tips & Tricks post, I will show you how to configure a plan to evaluate your data and make a decision on whether to continue processing your data as planned or to stop processing data and exit the plan safely without the plan throwing an error.In this scenario we will generate a data set consisting of 1 column and 100 records. The column will have either a true or false value. We’ll have a business rule that checks the values of the 100 records and stops processing the data if at least one false value is found in the data stream.  Let’s prepare the data pipe initial so that we have input and output data flows.Starting with a blank .plan file we will add a Text File Reader step to store our results and a Random Record Generator step to generate the 100 records.We’ll configure the Random Record Generate step to create 1 column named value, and to create the values true 99 in 100 frequency and false 1 in 100 frequency. This configuration will ensure that when we run the plan 1