Discuss, learn, ask, suggest, share all things DQ&C π
Recently active
When it comes to data quality (DQ) management, monitoring projects provide us with ability to export DQ results and maintain a close eye on data quality. However, the initial export contains attributes organized in columns, alongside additional columns housing information regarding the DQ results for these attributes. Analysing this data can be challenging because DQ rule results and explanations are presented as a single string.Despite the fact, that extracting DQ results in a customized format might require a more detailed approach, it might be better aligning with the client's specific needs. In this guide, you will find information how to export DQ results tailored to specific requirements.The Task at Hand:Our objective is to ensure a separate line for each attribute in an export file, with the following columns related to DQ results:the name of the DQ rule; an explanation for the check; a list of terms assigned to that attribute; the attribute value.We have three main steps:Attrib
HiΒ everyone, I'm Sathya, Solutions Consultant at Ataccama. In this post I am going to discuss about adding Data Classification and relationship types.Β Step 1:Β Go to Global setting and you will see Metadata Model, click on Create EntityΒ Β Β Step 2:Β Once Create Entity page opens, fill the Name of the Entity: Data ClassificationΒ Step 3:Β Click Add Property and fill in the property needed for the Entity: Name and Description with its data typeΒ and hit saveΒ Step 4:Β Goto Metadata Model under Global settings and search for metadataΒ Step 5:Β Click Add property: Fill in name, type: Embedded Object Array and Object type:dataClassificationΒ Step 6:Β Goto Metadata Model under Global settings and search for termΒ Step 7:Β Click add property and Fill in data Classification type as Referenced Object, PII and CDE as BooleanΒ Β Β Step 8:Β Hit Apply changes and the System will be Switching to Maintenance modeΒ Step 9:Β Click DB backup and the application restarts.Β Β Step 10:Β Click on Run changes again and all the chan
Hi, is it possible to have aΒ single rule that checks many condition. example is belowΒ Β Β i want a single rule named--Name Validationi want this rule to checkΒ and validate name that are in -Title Case --first letter upper case follow by all lower caseΒ Accented--- Accepts accented characterΒ Accepts special charactersΒ Could have Hyphen -Β the rule should should invalidate any other names that does not meet these conditions . or is there a better way to do this ? preferred to break it down so its will be easy to know the reason why the names are not valid whenever its does not matchesΒ
Hello,I have a runtime-serverΒ online service that I am trying to call using Azure Synapse using OAuth authentication. I created a REST Linked Service in Azure that connects successfully to the runtime-server online service. IΒ am usingΒ a copy activity in Azure to do a POST call on the online service. When running the copy activity I get the following error:{ "errorCode": "2200","message": "Failure happened on 'Source' side. ErrorCode=UserErrorCharSetInvalidOrNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The CharSet '\"UTF-8\"' is invalid or not supported from Response.Headers.Content-Type.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentException,Message='\"UTF-8\"' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method.\r\nParameter name: name,Source=mscorlib,'", "failureType": "UserError", "target": "CopyDataCatalogToRaw","details": [] }T
Hi Community,Would there be a way in Ataccama Desktop to select records from for instance entity term that have been inserted/updated/deleted from a certain datetime, like the Published on datetime in the History tab? My intention is to use the metadatareader to read the details from term and then use this mutation datetime in the filter in the metadatareader.Kind regards,Albert
Hello, Iβm trying to mask certain attributes in a table and be able to export them while exporting the table as a file. Is it possible to mask them as part of the JSON code while creating an export plan or create masking rules and apply it toΒ the attributes and then export them. (using ataccama ONE Web). Has anyone tried doing this before or has any idea about this? Please let me know!Thank you,PrithikaΒ Β
Hi Everyone,We are in the process of transitioning to Ataccama as our DQ solution. In our current setup, we perform various data controls such as:Empty File Check Duplicate File Check Header or Trailer Validation (e.g., validating trailer record count or checksum)We are keen to understand if these types of data controls can also be implemented using Ataccama. Our goal is to consolidate all data quality processes into a single tool for better efficiency and management.Thank you!
Hi everyone, I am Sathya, Solutions Consultant at Ataccama. In this post I am going to discuss about how to create Relationship types(Source and Target reference).Β Step 1: To add new relationship type, go to Global settings on the left navigation bar.Β Step 2:Β Under Application Settings, choose Relationship typesΒ Β Step 3:Β Available Relationship Types will be listed.Β Β Step 4:Β Click on create button on the top right corner of the screen layout.Β Β Step 5:Β Fill in the name, Source rel name and Target rel name of the Relationship TypeΒ Β Step 6:Β DomainOf relationship type is created, with its source rel name and Target rel name with its ID.Β Β Step 7:Β All the relationship Type is listed along with the newly created DomainOf.Β Β I hope you found this article useful, please let us know if you have any questions or thoughts in the comments below.Β Β
Hello, I am trying to create a rule for validity on Names. The logic and condition is that the rule matches names that areΒ Title Case --first letter upper case follow by all lower caseΒ Accented Accepts special charactersΒ Mandatory Could have Hyphen -Β I have this regexΒ ^[a-zA-Z '-]+$Β ^[A-Z][a-z]*(?:[ '-][A-Z][a-z]*)*$(?<=^|\s)[a-z]*[A-Z][a-z]*[A-Z]*[a-z]*Β above regex capture and validate names likeΒ John DoeMary Anne which are okay.Β but also validate name in all lower case for example john doe which is not okay. i want to know what i need to do to allow the regex not validate names in all lower case .Β
Hello everyone ! Β Iβm Srija ,Solutions Consultant here at Ataccama. Today, Iβd like to talk about how to trigger profilingΒ while running a monitoring project.While running the monitoring project, the profiling process is getting triggered without any schedule or event created on the catalog item to initiate profiling. This unexpected profiling of over 1000+ records significantly slows down the system. The likely cause for this issue is the anomaly detection feature being enabled on the catalog item.To address this issue, you need to check if anomaly detection is enabled on the catalog item.Β Β Β Β Anomaly Detection in Ataccama DQWhat Anomaly Detection Does:Anomaly detection in Ataccama Data Quality (DQ) automatically identifies outliers and unusual patterns in your data. This feature is designed to ensure data quality by detecting anomalies such as unexpected values, missing data, or patterns that do not conform to the expected behavior.Implications of Enabling Anomaly Detection:Automatic
Hi everyone, I'm Sathya, Solutions Consultant at Ataccama. In this post I am going to talk about creating Data Domain , relationship usingΒ AQL expression.Β Step 1:Β On the organization tab, go to Global settingsΒ Step 2:Β Click Navigation IconΒ Step 3:Β List of entities will be listed, under Business glossary we will have Data DomainsΒ Step 4:Β Click on the 3 dots and click Add Child option to add a child to the list.Β Step 5:Β Pop up window appears, fill in the details as follows:Navigation item type: Listing Listing Node: Term Icon: Folders Text Medium Name: Fill the name of the Data Domain Save and publish.Β Β Β Β Step 6:Β All the terms will be added to the Data domain. To filter out we have to update the AQL expression.Β Step 7:Click on the 3 dots on the right corner of the terms page where the data domain is displayed and choose Edit page.Β Step 8: In the Filter part we must fill in the AQL ExpressionΒ Step 9:Β Syntax for AQL expression:name is "(name of the dataDomain)" OR @termRelationship(target)
Hi all, I have a group of PPP that need to be renamed.Β Is it possible to update name once published? How? Thanks
I want to configure one general email notification that will be used for all of my monitoring programs. I would prefer to receive a single email notification that compiles the findings from various monitoring projects rather than the numerous emails I usually receive for each monitoring project. I'm aware that I should use One Desktop, but I'm not sure how to proceed precisely. Well, I know that I have to call monitoringProjectProcessing which allows me to Β retreive the finish time with processing ID. And then I have to complie to get the ID link of notification but the problem is here that I dont know the step exactly. It would be great if you could help.Thanks in advance.
Letβs say we have some properties placed at the table attribute level. Is it possible to display those table attribute properties in the data structure page of the catalog itemΒ as shown below? Thanks in advance.Β Β
Hello, Iβm trying to figure out why the export plan section in Ataccama ONE desktop keeps showing βNo Records Foundβ although I have created a list of export plans. Once I create an export plan, I save and publish it, but it does not show up as part of the export plan section instead it says, βJSON errorβ. Does anyone have any idea on how to fix this issue ?Thank you,PrithikaΒ Β
Hi,There are certain records that contain βwhite spacesβ at the beginning and/or at the end. Ex: βΒ Β Β 100137069Β Β Β Β β, βΒ Β Β 100017129Β Β Β Β β, etc. We don't allow the use of white spaces and want to build a DQ rule to check the invalid records. Hence the advanced expression:Β Β length(String_ATTRIBUTE_1) <> length(trim(String_ATTRIBUTE_1)). When I test the DQ rule it works as intended. I copy the above results and paste them in the test environment and it shows me the record is invalid. When I remove the white spaces it will considered valid. However, when I run the Monitoring Project based on the same records and DQ rule it gives me 100% validity. How is this possible? And maybe more importantly, how can I create a rule that will show the invalid records shown earlier?Β Β
Hi Community,I have usedΒ this community tutorialΒ (thanks @anna.spakovaΒ !) andΒ this part of the documentation portalΒ to create a custom relationship (and an existing custom relationship as example). So that should be enough info, but still I encounter an issue. As follows.The relationship is between an extension of term (entity term) and custom entity application. The new relationship has been defined in entityΒ entityTermSourceOfRecordRel as followsThis new entity has also been defined in entity metadata as array.A new relationship type has been defined for this purpose:In the entity term screen I have defined the realtionship as follows:{"_type":Β "relationships","relationshipPropertyPath":Β "/entityTermSourceOfRecordRel"}On the entity term screen I would now expect to be able a new relationship like this (from the example we already had implemented):Instead I only have the option to define a new relationship type, like:What am I overlooking? We are on version 13.9.Thanks and kind regard
Hey Everyone!We have a use case involving datastage and curious if anyone has worked with this in the past. We have a team who has datastage jobs that are triggered by a table being pushed information downstream. We are wondering if there is a way either in the web or desktop, if we can use the trigger files that are created by datastage,Β somehow plug into the datastage flow, or another wayΒ that would trigger a monitoring project simultaneously to run data quality checks against the load? if certain data is not loaded in correctly, it will break the connection for the two tables, and in turn break the job,Β and they want to see the population of what is breaking the join. Hope this makes sense!Β Thanks,Thomas
Hi everyone! Iβm wondering if its possible to trigger or use theΒ Review workflow from one desktop? For instance, im currently updating a large amount of attribute descriptions and am publishing them with one desktop. Whenever I use the One Metadata writer it will apply the descriptions automatically without further approval.I would like to know if its possible to write the descriptions and then have them sent to the tasks board for the data owner to review and approve or deny.Example of what im trying to do:Use one desktop to write descriptions to attributes Review workflow is triggered, andΒ new descriptions are assigned to data owner for review (preferably in batches of 15-20 if possible) Owner either approves changes, causing descriptions to be updated or denies them causing no change to descriptionsThank you for any help!
Hello everyone!In this post, Iβll walk you through how to extract the description of all the columns and tables using the one desktop plan. Providing detailed descriptions and comments is a fundamental best practice in data management and data standards. This approach significantly enhances theunderstanding and usability of data.The summary of all the tables, attributes and their descriptions can be extracted using the below plan.1. Open a plan and add Attribute and Catalog Item metadata reader steps.2. The Attribute meta data reader helps to extract the attribute names, comments, description. Open the meta data reader and provide the column names as below and provide the Parent Id Column name as βCatalogItemIDβ.3. The Catalog Items meta data reader helps to extract the catalog Item name based on the catalog Item Identifier. Open the meta data reader and provide the column names as below and provide the Id column name as βcatalogItemIDβ.4. Use the Join Step and join the meta data reade
Β Hi everyone! In this post, Iβm going to walk you through how to ensure that changes made in the post-processing plan are visible across different user accounts.Issue DescriptionIn version 14.5 of ONE Desktop, there is an issue where if you create a post-processing plan for a monitoring project from one account, the changes are not reflected when another user logs into the same environment.SolutionTo fix this issue, you need to download the remote file every time before making any changes to the Post Processing Component to ensure you are editing the latest file.Steps to Fix the Issue1. Login to ONE Desktop:Connect to your server by navigating to Server -> New -> ONE Platform2. Create a Post-Processing Plan:Go to the Monitoring Project -> New -> Post-Processing Plan. Β 3. Download the Latest Changes:To get the latest changes from one account:Right-click on the Component. Select Download. Choose Download Remote File Β By downloading the remote file, you will get the latest c
Hi Team,Β I am working on task to check the number of rows in a column. I am using count of ATTRIBUTE_1 but not able to achieve it . can you please suggest a better approach. let me know if i miss anything.Β
Hello Ataccama teamΒ Checking ataccama admin center on resource section there is mentioned the JVM Heap Memory,I would like to know more about the Ataccama JVM Heap Memory configuration , as I understand Ataccama manage Java APIs (mmm comment, dpe,...) , I am curious to know if Ataccama has a configuration file to define the limits of JAVA memory to use on each ataccama api, or if Ataccama has a configuration file to manage the whole java resources.
Β Β Hi I'm Srija, Solutions Consultant here at Ataccama.In this post, we can see a workaround for changing the stewardship for out-of-the-box rules and terms along with the project related business terms and rules. This workaround helps to transfer the stewardship of business rules and terms to βOrganizationβ without causing the API error.How to perform the transfer of stewardship?ONE web applicationΒ By creating a custom plan to change stewardship for multiple business terms and rules.Β How to create a custom plan to bulk change stewardship for business terms and rules?In ONE desktop, you can use metadata reader steps from term or rule, and you can join with group metadata reader using stewardship name and name. In the JSON step, use the below GraphQL query to change the stewardship for terms or rules.{"query":"mutation TransferStewardship_rule($gid: GID!, $newStewardId: GID, $accessLevelForPreviousSteward: String) {Β Β ruleTransferStewardship(Β Β Β Β gid: $gidΒ Β Β Β newStewardId: $newStewardIdΒ Β
I am working on the usecase of comparing two different columns in two different csv files. I need to compare these column, remove any duplicates and merge them into single column. I am trying to use join component but not sure about this approach will work or any better solution is available to achieve this use case - If anyone can suggest a better apprach it will be helpful. - Thanks.
Already have an account? Login
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.