Ataccama Admin Centre: ServerConfig and RuntimeConfig for beginners (Self-managed installations) 2/2

First part of the tutorial - Ataccama Admin Centre: ServerConfig and RuntimeConfig for beginners (Self-managed installations) 1/2

Use cases

Now let’s consider some use cases and see how we can set up the Admin Centre to meet our needs.

• Security on the Admin Centre UI
• Online service endpoint
• Schedulers, workflows and plans / components

Admin Centre Security

Security on the Admin Centre UI itself is taken care of by the Http dispatcher Component. The one-executor.serverConfig and one-executor.runtimeConfig files provide an example of how the files could work.

(default.serverConfig and default.runtimeConfig , for local testing purposes, is set to NoSecurity, which is not recommended for production use)

1. ServerConfig

👉See attachment: 1_ServerConfig

Notes about this config:

• the configName of keycloak-local and clientId of one-admin-centre need to be referenced in the runtimeConfig file. See below.
• interceptUrls is where the various nodes of the Admin Centre UI can be secured. At the moment, the configuration above shows pattern /** (everywhere) should have isAuthenticated() mode; meaning everyone who tries to hit any Admin Centre UI must have a valid keycloak user with correct credentials entered.

2. RuntimeConfig

Local:

👉See attachment: 2_RuntimeConfig_local

The server version is very similar, just with the different location for the keycloak url. For example, if my Keycloak is installed at https://abc.ataccama.com/auth, then it would look like:

👉See attachment: 2_RuntimeConfig_server

Online service endpoint

My project requires an endpoint to allow data exports from RDM. I have prepared the product_export.online and product_export_export.plan files, and placed them in my project folder (see below).

The product_export.online contents are as follows, in xml format:

👉See attachment: Online_service_endpoint

NOTE:

• The product_export.online file above references the product_export_export.plan configFile with a relative file path (where the file path starts with pathvar://)
• Having the method location at /rdm/product_export_export​​​​

The product_export_export.plan file opened in One Desktop, looks like this:

Within product_export_export.plan, the Rdm Extended Reader step itself references a server connection to rdmapp, where the RDM application is running.

1. ServerConfig for OnlineServicesComponent

To enable online service for Admin Centre, we need to set up online service component. In default.serverConfig, we have this:

<component class="com.ataccama.dqc.online.OnlineServicesComponent">
				<serviceLookupFolders>
					<configFolder>../services</configFolder>
				</serviceLookupFolders>
			</component>

Relative paths and absolute paths are both supported.

In our case, in case the serverConfig file itself is not stored within the project, the ../services path might not exist. Therefore let’s now adjust the folder path for our online services to absolute path. Note the path is where I’ve put my files - you’ll need to adjust it according to where you’ve put your files.

	<component class="com.ataccama.dqc.online.OnlineServicesComponent">
				<serviceLookupFolders>
					<configFolder>C:\Ataccama\workspace\workspace_1380_preGA\projects\rdm\onlineServices</configFolder>
				</serviceLookupFolders>
			</component>

This path will be good for local testing. More than likely your application server for dev/test/prod environment, the folder path might look more like this instead:

<component class="com.ataccama.dqc.online.OnlineServicesComponent">
				<serviceLookupFolders>
					<configFolder>/opt/ataccama/projects/rdm/onlineServices</configFolder>
				</serviceLookupFolders>
			</component>

2. RuntimeConfig for pathvar and rdmapp

Our .online file references a path variable (aka pathvar) for rdm_plans, and our .plan file references a server connection (rdmapp). These are (alias) connections set up on ONE Desktop:

After generating the runtimeConfig file (Follow this document to do this using One Desktop) , I can prune it to only contain what I need: not forgetting the keycloak section we configured before. For local:

👉See attachment: 2_RuntimeConfig_pathvar_rdmapp_local

For server:

👉See attachment: 2_RuntimeConfig_pathvar_rdmapp_server

3. ServerConfig for Http dispatcher

We need to secure the online services endpoint of location /rdm. Let’s say a decision has been made that the data extraction endpoints are to be publicly available, so no authentication is required. The configuration would look like this:

👉See attachment: 3_ServerConfig

The ordering of the interceptUrl tags are significant. Tags are evaluated top to bottom, so always make sure the catch-all case (access="isAuthenticated()" pattern="/**") is last.

For example Admin Centre URL is https://localhost:8888. For any URL that start with https://localhost:8888/rdm, no credentials are required.

Anything else will require valid credentials.

Workflow and Scheduler

Plans or components need to be part of an online service (see above), or be referenced by a workflow to run. When configured, Admin Centre will allow the user to run schedulers and workflows. In the example below I enable the Admin Centre to run scheduler, workflow and plans.

I now want to enable the Admin Centre to run scheduler, workflow and plans.

Admin Centre will allow the user to run schedulers and workflows, and plans or components would always need to be referenced by a workflow (or as part of online service, see above).

ServerConfig for Scheduler component configuration

Local:

👉See attachment: 3_ServerConfig_scheduler_local

Server:

👉See attachment: 3_ServerConfig_scheduler_server

ServerConfig for Workflow component configuration

Local:

👉See attachment: 3_ServerConfig_workflow_local

Server:

👉See attachment: 3_ServerConfig_workflow_server

RuntimeConfig for workflow and components

Local:

👉See attachment: RuntimeConfig_workflow_local

Server:

👉See attachment: RuntimeConfig_workflow_server

Security for workflow by role

Let’s say you only want users with keycloak role DEVOPS to access schedulers on the Admin Centre. This is how you would add to the interceptUrls on the serverConfig:

👉See attachment: security_for_workflow_by_role

Deploy your files and test them locally

As with working with any configuration files or change processes:

• Keep a backup of your files before change in case you need to revert to them
• Add comments to places you have changed so people who read the config file in the future can understand what’s been changed and why
• Especially after config files are changed, check the logs on start up to ensure server starts correctly. You’ll be looking for a message of “Server Startup Complete”

Once you completed all your changes, start your server through your usual means and the new components should be activated.

• example server log file

Using java at: "C:\Ataccama\13.8.0_preGA\jre\bin\java.exe"
Using DQC  at: "C:\Ataccama\13.8.0_preGA\runtime"
2022-06-14 18:12:18 [INFO]    com.ataccama.dqc.server.bin.OnlineCtl messageId=generic_msg message=Starting server on localhost
2022-06-14 18:12:18 [INFO]    com.ataccama.dqc.commons.license.LicenseManager messageId=generic_msg message=Using following licenses:
2022-06-14 18:12:18 [INFO]    com.ataccama.dqc.commons.license.LicenseManager messageId=generic_msg message=  C:\Users\may.kwok\emp_may.kwok_13.plf
2022-06-14 18:12:19 [INFO]    com.ataccama.lib.securityutil.javaencryption.DefaultEncryptionEngine messageId=createDefaultKeyProvider reason=the 'properties.encryption.keystore' property not specified
2022-06-14 18:12:19 [INFO]    com.ataccama.lib.securityutil.javaencryption.DefaultEncryptionEngine messageId=createDefaultKeyProvider reason=the 'internal.encryption.keystore' property not specified
2022-06-14 18:12:20 [INFO]    com.ataccama.dqc.web.logging.LoggingComponentInstance messageId=generic_msg message=Switching to logging setup from logging.xml
2022-06-14 18:12:20 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'HttpDispatcher' 
2022-06-14 18:12:20 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'LoggingComponent' 
2022-06-14 18:12:20 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'HealthStateProviders' 
2022-06-14 18:12:22 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'WebConsoleComponent' 
2022-06-14 18:12:22 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'HealthStateWebConsole' 
2022-06-14 18:12:22 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'ManagementService' 
2022-06-14 18:12:22 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'OnlineServicesComponent' 
2022-06-14 18:12:22 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'WorkflowServerComponent' 
2022-06-14 18:12:22 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'SchedulerServerComponent' 
2022-06-14 18:12:24 [INFO]    com.ataccama.dqc.web.jetty.JettyListener$1 messageId=listening message=Listening for HTTP requests, port=8888
2022-06-14 18:12:24 [INFO]    com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Server startup complete.

Check the UI and Server Health Status

• On start up of Admin Centre, open a web browser and navigate to the web url (https://localhost:8888) in our example, to make sure it’s started correctly.
• Check the UI. It should look something like this:

• Note the applications (where online services sit), workflow, and scheduler sections. These sections highlighted above will only show if added to the configuration.
• If the server does not start up correctly, the logs will show errors. Fix the errors in the configuration file(s), then start the server again. Repeat until it starts correctly.
• Some frequent errors include:
  • runtimeConfig is invalid
    • The full message could look something like this: Unable to parse runtime configuration file; nested exception is: com.ataccama.dqc.processor.engine.InvalidRuntimeConfigException: Runtime configuration 'C:\\Ataccama\\13.8.0_preGA\\runtime\\server\\etc\\default.runtimeConfig' is invalid
    • Check the xml structure of the runtimeConfig and make sure the structure is valid. Check further down the error message it could provide hints as to what exactly is wrong with the syntax.
  • serverConfig is invalid
    • The full message is: Server Configuration is not valid
    • Check the structure of the serverConfig and make sure the structure is valid. Check the full log as it could provide hints as to what exactly is wrong with the syntax. As an example: Folder 'C:\\Ataccama\\workspace\\workspace_1380_preGA\\projects\\rdm\\workflow' must exist. [offending property path=/serverComponents[6]/sources[0]/path]
    • This points me to the 6th server component, a source path defined doesn’t exist. Indeed upon checking, there was a typo in the path and so the server configuration could not find the folder.
• Check Server Health Status to see if all connections are happy:

As we can see above, my database connection failed. This means the structure of the connection configuration seems correct (otherwise the server won’t start), but the connection is failing. The next step in the above example would be to check if the database credentials are correct, or if the database itself is up, and continue investigating from there.

Also showing above, the online services has a grey question mark so the .online file and its associated plan / component should also be checked.

Test User Permissions

In our example above, we restricted access to the scheduler to the DEVOPS role. Have a go and try accessing the Scheduler node. If you don’t have the DEVOPS role given, you should see this screen:

Otherwise if it is all green ticks, then your set up is complete.

Deploy to server

If you are satisfied with your server on your local workstation, you are ready to deploy the files to the server! Transfer the files to the server, and start up the Admin Centre. Go through the same checks as you would start up on local workstation. If there are no errors, well done!