Skip to main content

Global runtime configuration (GRC) is the answer to the following questions:

  • How to upload DQ results from my monitoring project into a database/other data source instead of CSV file?
  • How to send my DQ results into PowerBI to create some nice reports? 
  • How to connect to my data source from a VCI if this data source is not supported from the Catalog?

These three questions are the most frequent use cases for using the Global runtime configuration. It allows us to define additional connections to databases, servers (e.g. MiniO or S3), APIs etc. and perform different kinds of operations with it - like save our DQ results in a post-processing component to a database or upload data from additional data sources using the VCI (Virtual Catalog Item).

How to create a Global runtime configuration

The GRC is attached as a parameter plugin.dqc-support.ataccama.one.dqc.runtime-config.default in the DPM application.properties (or using the Configuration Service). What is very important, the GRC is an XML configuration similar to the runtime configuration used in an Orchestration server, but is encoded using Base64 before inserting into the application.properties.

Let’s give some examples. We may want to

  • access some lookups we store in MiniO to use them in our VCI or post-processing component, 
  • upload DQ results from our monitoring project into our MS SQL Server database instead of creating CSV files.

To accomplish these two tasks, we need to create a Global runtime configuration with definiton of these two resources: Minio and MS SQL Server database.

Our GRC would look like this:

<?xml version='1.0' encoding='UTF-8'?>
<runtimeconfig>
  <contributedConfigs>
    <config class="com.ataccama.dqc.s3.config.S3Contributor">
      <s3Connections>
        <s3Connection bucket="lookups" s3Compatible="MINIO" endpoint="http://localhost:9000/" secretKey="crypted:AES:KogsCeZBG2bBAgUVo5mnBlvgFKDOprxHpvlDzAItQj2T6ZhMvqDI7ugA7Azbs5EB" accessKey="minio" name="minio" sseType="NONE"/>
      </s3Connections>
    </config>
  </contributedConfigs>
  <dataSources>
    <dataSource password="crypted:AES:<password>" name="MSOUT" driverClass="com.microsoft.sqlserver.jdbc.SQLServerDriver" user="one" url="jdbc:sqlserver://ec2-54-246-154-116.eu-west-1.compute.amazonaws.com:1433;databaseName=master"/>
  </dataSources>
</runtimeconfig>

We can see that the definition is identical to the standard runtimeConfig used in Orchestration server. How to set up different kinds of intergrations can therefore be found in the corresponding documentation (see below under Resources)

There is couple of important notes to keep in mind:

  • When defining a data source, you must include the driverClass attribute instead of driverName. The driverClass needs to be specified in plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.cp.runtime property of DPE (see below under Resources).
  • You have to make sure the driver class is present in the destination specified in the DPE.
  • Postprocessing components or VCIs referencing the data source must use the same name of the data source as specified in the runtime configuration.

Before inserting the GRC into DPM application.properties, we have to encode the XML file using Base64. You can use for example this web page: https://www.base64encode.org/

The result in the DPM application.properties looks like this:

Global runtime configuration in DPM​

After changing the properties (either directly in the application.properties file or in the Configuration Service), you need to restart the DPM module:

sudo systemctl restart <name-of-the-DPM-service>

After restarting you are ready to use the new resources.

Resources

I found a way to encode base64 in Notepad++!

 

For those of you not running Windows PCs or just prefer Sublime:

Install the package CoolBase64 and you can do the same thing that May found in Notepad++.

https://github.com/kamrynsite/CoolBase64-sublime

 

Reply


Community Terms of Use & ConditionsCookie settings