Discuss data, share best practices, learn, and grow with Ataccama's Community.
Recently active
Hi Everyone, I have postman REST API call that retrieves CSV as output. The API call uses Bearer token, I am not able to use JSON call (output is CSV format) , Https GET (doesnt accept bearer token) . Is there any other way to use the REST API → CSV(output file)?
Good day! I would like to find the completeness of the descriptions for all the tables and their attributes in Ataccama Web. Is it possible to do this in AtaccamaWeb using Rules? There are approximately 700 tables in the system. I will be very grateful for your helpThank you!
Hi all,I have created a new entity in the metamodel, let's say with properties A, B, C and D (in that order).In the user interface the overview screen of this entity is a list-overview, showing the properties in the same order as defined in the entity.I would like to be able to adjust this lay-out, like D, B, C and not showing A. Refer to the example in the attachment for more clarity.How can I accomplish this?In addition, how can I define a filter on one of the property columns?Kind regards,Albert de Ruiter
Hi community,In the metamodel, when defining a relation to another antity of type embedded object array, in the user interface it appears as a very large white area containing one button. We feel that this feature is not in line wth proper user interface design. Therefore I am looking for a way to have this button shown in a way more minimal format.Does anyone knows if this is possible? A trait or a specification in the page lay-out definition or…?Thanks in advance!Kind regards, Albert
I have a workflow which does an HTTP GET to download an HTML page. I then use a DQC plan to filter and alter the data in the HTML file to get a “new” URL. I now have this “new” URL in a column at the end of the plan. I’m wondering, is it possible to use this “new” URL column from the DQC in the same workflow to perform another HTTP GET of the “new” URL?
Hi,There is a lot about GraphQL and JSON-call. But I have question on how to process nested GraphQL in a JSON and maintain integrity when using (nested) datastreams) in the JSON Call step in Ataccama (v13.x).Below an example. There are profiles with path ($.data.catalogItem.publishedVersion.profilingConfigurationInstances.edges.node.publishedVersion.profiles.edges.node.@). And then a nested data-stream, which continus from this path (publishedVersion.attributeProfiles.edges.node). Each stream will give an output for the JSON-caller step. Now how do we maintain integrity between the streams/output files? So how can I refer from a sub-stream to a parent key, so I can later join the JSON-call output streams?
Is there a function or custom ways to convert strings into camel case on DQS - Data Quality services?
Happy Tuesday Community! We started this week by sharing best practices on Monitoring Projects, and today we’ll be continuing with how to configure Monitoring Projects. If you have missed the first post you can find it here:Let's explore the best practices for configuring Monitoring Projects to ensure robust data quality checks and anomaly detection.Select Catalog ItemsGo to the Configuration & Results tab. To add catalog items: For a new project, select Add catalog items. To add more items to an existing project, select +Add under Items to Monitor. To remove an item, use the more options icon and select Delete. The listed catalog items will display a summary of: Structure: Aggregated results for structure checks on attributes. Anomalies: Detected anomalies on attributes. DQ Checks: Results of data quality checks assigned to attributes. Overall: Aggregated quality based on assigned DQ checks contributing to overall quality. Individual DQ dimensions, listed with the
Happy Monday community! Today we are going to talk about Remote Plans. Let’s start with the basics, what are remote plan executions? The remote plan execution lets you run any ONE plan on the Ataccama ONE Platform. Plans are typically developed and tested on small data sets or on dedicated environments. This helps you validate the plans and assess how well they perform so that you can make the necessary adjustments before applying them on your actual data. Once the plan works as expected, you can run it remotely on production data in ONE.Jobs are first sent to the Data Processing Manager (DPM) module, which forwards the job information to a suitable Data Processing Engine (DPE). Actual data processing takes place only in DPE instances. How to configure Remote Plan ExecutionRemote plan execution leverages existing controls and concepts of ONE Desktop. Once you connect to the Ataccama ONE Platform, you can choose between the following launch options:Local: A suitable DPE is selected auto
We successfully connected Workday from Atacccama one web by using CData drivers.Now we are trying to do the same from One Desktop.How do we connect Workday objects from Onedesktop by using CData drivers?We were able to connect Workday through the API's but our requirement demands CDATA drivers connectivity.Thanks,Mahesh Arigela
Happy Wednesday Community! We are halfway into the week and ready with Part 2 of Component Steps. If you have missed Part 1 don’t worry, check it out on the link below 🔽 In the second part of our journey into Component Steps, we will delve deeper into their creation, configuration, parameter mapping, running from the command line, and working with nested components. Creating a ComponentBuilding a component step is akin to constructing a plan, but certain rules dictate its placement within a plan: At the Beginning: A component at the start of a plan should commence with an input step connected to actual data, not Integration Input. In the Middle: For components in the middle of a plan, they must have at least one input endpoint with Integration Input and one output endpoint with Integration Output. At the End: Components concluding a plan must start with an input endpoint with Integration Input and conclude with an output step that isn't Integration Output (ensuring data is written
Any one who has worked on Time series analysis option in V13.9 and successfully got the required results
Hello,I have a requirement to append the records in parquet file in the event of RDM on-publish synchronization. Does anyone know, how can we append data into Parquet file in RDM on-publish event?I believe parquet file writer will write data in the file, but if the new record created in the RDM, will it append the data in the same file or will replace the file completely and create new one.Thanks for any help.-Ojaswini
Hi ,I want to know what was the input to be given in root object for JSON writer
Hello Ataccamer,I am dealing with Parquet file writing in on-publish mode. Could you please suggest how to write data to Parquet file when it is a record deletion event on RDM UI (in on-publish) . As in case of Parquet file it is always first delete the file and then created the file again.How the CHANGE_TYPE is in {'DELETED'} will work in this case. Any suggestions, please share.Thanks and regards,Ojaswini
Hi community, happy Tuesday! Today, we are covering one of the essential functions of ONE Desktop - Expressions & Steps ✨Steps and functions enable you to craft plan files tailored to your project's unique needs. Here, we provide a comprehensive overview of the frequently used steps and functions, guiding you towards best practices.Frequently Used Steps:Flow Control StepsData Parsing StepsAnalysis StepsMatch and Merge StepsUsing FunctionsWithin these steps, you can leverage various functions to further enhance your data processing capabilities. Here are some commonly used functions:matches: Performs a full match of input data with a regular expression, returning true or false. find: Conducts a partial match of a regular expression within the input string, returning true or false. substr: Extracts a substring from the input string, starting from zero.Using Regular ExpressionsAtaccama's products support regular expressions for powerful pattern matching. Here are some fundamental regu
Eventually, you might need all your data quality results evaluated in One monitoring projects or within regular catalog items (via business terms and related DQ rules) exported to some reporting system or into excel. And it would be nice if it looked like this: TLDR; There are 2 components attached you can use to export DQ results, one for monitoring projects, another one for regular catalog items. Monitoring ProjectsThe idea is to read all monitoring projects metadata, join it with their check results as well as rules names, catalog item names and voilà, we can save it to excel/csv or to some database table.It sounds like something very simple, and indeed there is no black magic, but nonetheless component looks a bit more elaborated: Catalog Items DQ (based on Term evaluations)The idea is to find all catalog items that have aggregated overall data quality and join them with the DQ rules results:This component has a bit simpler implementation: Of course, the output of the components
Happy Monday Community! We are continuing our ONE Desktop best practices series this week as well! Today on the menu is - DQ Scoring!Data quality scoring is a vital process in Ataccama Desktop, enabling you to assess data quality based on specific rules and assign a numerical score. This score helps differentiate between high-quality and low-quality records, making it an essential step in data processing.Here are the best practices for data quality scoring in Ataccama Desktop:1. Categorize Scoring into Four LevelsTo effectively score data quality, categorize your cleansing scores into four quantification levels:Scoring Level Scoring Description Scoring Result 0 No modification was done to the input value. VALID < 10,000 Small modification was done to the input value. VALID/CORRECTED < 10,000,000 Huge modification was done to the input value. UNSAFE/UNKNOWN > 10,000,000 Input value or pre-cleansed value is null. NOT VALID Each cleansing operation, as de
Hi everyone!Today, we have a quick best practice on covering the basics of column naming and debugging in ONE Desktop plans.When working with ONE Desktop, maintaining a consistent and meaningful naming convention for columns in your output files is crucial. This practice ensures clarity, organization, and seamless data processing. In this guide, we'll provide recommendations for naming conventions and debugging your expressions and steps for efficient plan execution. Column Naming ConventionsOutput files in ONE Desktop can contain numerous columns, especially after processing input data in DQC. To keep your data well-organized, we recommend grouping columns by content. Below are the key prefixes and suffixes used in our naming convention: PrefixesAttribute Prefix Explanation src_xxx Source input values (read-only attribute) meta_xxx Source input metadata dec_xxx Decoded source input values dic_xxx Translated master value storage (src_xxx is the source) pur_xxx Pre-clea
Hi,I am wondering if the following is possible. On the glossary's term screen not all properties/metadata will always be filled. For instance in most occasions there will not be an abbreviation for a term.Is there a way, on the view screen only, to hide this abbreviation field when it contains no value?Thanks for your time!Kind regards,Albert de Ruiter
Hi Community!We are back with part 2 of One Desktop Plans! If you have missed our first post on how to build plans you can find it by clicking on the link below 👇 When it comes to executing plans within ONE Desktop, understanding the various methods and choosing the right approach is essential for a seamless experience. In this guide, we'll explore the best practices for running plans effectively. Understanding Execution OptionsONE users have multiple ways to run a plan: Local Launch: This is done using Ataccama ONE runtime (formerly known as DQC) from the ONE Desktop interface. ONE Platform Launch: You can employ DPE or Spark DPE for remote execution via the ONE Platform. Command Line Execution: Utilize the runcif.sh|.bat script via command line when employing Ataccama ONE runtime. DPM CLI Tool: Employ the DPM CLI tool for remote execution using DPE or Spark DPE. This article primarily focuses on executing plans. For information on running plans within workflows, please refer t
Hello Ataccama Community,I'm currently using Ataccama One Desktop version 13.9.1, and I'm encountering an error while trying to execute a plan that involves a Kafka Reader component. I have configured a Kafka server(kafka integration) with the following properties:ssl.truststore.location=LOCATION_TO_TRUSTSTOREssl.truststore.password=TRUSTSTORE_PASSWORDssl.protocol=TLSsecurity.protocol=SASL_SSLsasl.mechanism=GSSAPIsasl.kerberos.service.name=kafkasasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \ useKeyTab=true \ storeKey=true \ keyTab="LOCATION_OF_THE_KEYTAB" \ principal="USERNAME_HERE@EUROPE.XYZXYZ.CORP"; The error message I'm seeing is: "Property with name 'KafkaResources' doesn't exist in com.ataccama.dqc.streaming.KafkaProviderComponent!"I've reviewed my configuration, but I'm still unable to resolve this issue. Can someone please provide guidance on how to troubleshoot and resolve this error? Any insights or suggestions would be gre
Hi, I have below use case:A party can have multiple phones related to it, when any of them change, we need to update a flag on every related phones (same party_id) I am having problems to get those related phones during the merge process, what would be the better way to do it? I have tried it using Mdc Read Step, but I'm not able to start the server due below errorjava.lang.IllegalStateException: Cannot find INmeEngine: Failed to create step 'Mdc Read' (com.ataccama.nme.dqc.steps.MdcRead) (Cannot find INmeEngine) at com.ataccama.nme.internal.dqc.PlanExecutor.createRunner(PlanExecutor.java:137) at com.ataccama.nme.internal.dqc.PlanExecutor.<init>(PlanExecutor.java:83) at com.ataccama.nme.internal.dqc.PlanExecutor.<init>(PlanExecutor.java:69) at com.ataccama.nme.internal.dqc.pool.PlanExecutorCreator.createResource(PlanExecutorCreator.java:34) at com.ataccama.nme.internal.dqc.pool.PlanExecutorCreator.createResource(PlanExecutorCreator.java:17) at com.ataccama.dqc.commons.reso
Hi team!I'm trying to implement a simple maker-checker process for Business Glossary Management: there are different Data Domains, each with its own responsible: Data Steward and Data Owner. So, for each Data Domain, the corresponding Data Steward creates the term --> the corresponding Data Owner approves it. Therefore, it's essential to consider the separation of permissions by data domains.Please give me some ideas on how I can implement it in the current product version. As far as I could investigate, grouping terms by different folders/categories in the Glossary doesn't help since I can't assign any roles to a particular folder/category: I can't assign Steward1 and Owner1 to the BusinessTerms group and Steward2 and Owner2 to the SecurityTerms, for example.How should it be implemented, then? Many thanks for considering my request!
Hi,We've loaded 2000+ terms and assigned those with desktop as a migration from a data dictionary. When we now have new catalog items, Ataccama suggests 20+ terms for a CI Attribute. This is not usable.How we tweak it to:Limit to only top 5 Stop when a term is already assigned.And here's an idea: If the column header is the same or can be matched using a simple fuzzy match, that is a better indicator than most AI.Another: Please skip some of the datatypes, a boolean … probably not a good idea.Any idea's?We are also wondering if we can turn it off.Thank you in advance!Marnix
Already have an account? Login
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.