First part of the tutorial - Ataccama Admin Centre: ServerConfig and RuntimeConfig for beginners (Self-managed installations) 1/2
Now let’s consider some use cases and see how we can set up the Admin Centre to meet our needs.
- Security on the Admin Centre UI
- Online service endpoint
- Schedulers, workflows and plans / components
Admin Centre Security
Security on the Admin Centre UI itself is taken care of by the Http dispatcher Component. The
one-executor.runtimeConfig files provide an example of how the files could work.
default.runtimeConfig , for local testing purposes, is set to
NoSecurity, which is not recommended for production use)
👉See attachment: 1_ServerConfig
Notes about this config:
one-admin-centreneed to be referenced in the runtimeConfig file. See below.
interceptUrlsis where the various nodes of the Admin Centre UI can be secured. At the moment, the configuration above shows pattern
/**(everywhere) should have
isAuthenticated()mode; meaning everyone who tries to hit any Admin Centre UI must have a valid keycloak user with correct credentials entered.
👉See attachment: 2_RuntimeConfig_local
The server version is very similar, just with the different location for the keycloak url. For example, if my Keycloak is installed at
https://abc.ataccama.com/auth, then it would look like:
👉See attachment: 2_RuntimeConfig_server
Online service endpoint
My project requires an endpoint to allow data exports from RDM. I have prepared the
product_export_export.plan files, and placed them in my project folder (see below).
product_export.online contents are as follows, in xml format:
👉See attachment: Online_service_endpoint
product_export.onlinefile above references the
product_export_export.planconfigFile with a relative file path (where the file path starts with
- Having the method location at
product_export_export.plan file opened in One Desktop, looks like this:
product_export_export.plan, the Rdm Extended Reader step itself references a server connection to
rdmapp, where the RDM application is running.
1. ServerConfig for OnlineServicesComponent
To enable online service for Admin Centre, we need to set up online service component. In
default.serverConfig, we have this:
Relative paths and absolute paths are both supported.
In our case, in case the serverConfig file itself is not stored within the project, the
../services path might not exist. Therefore let’s now adjust the folder path for our online services to absolute path. Note the path is where I’ve put my files - you’ll need to adjust it according to where you’ve put your files.
This path will be good for local testing. More than likely your application server for dev/test/prod environment, the folder path might look more like this instead:
2. RuntimeConfig for pathvar and rdmapp
Our .online file references a path variable (aka pathvar) for rdm_plans, and our .plan file references a server connection (rdmapp). These are (alias) connections set up on ONE Desktop:
After generating the runtimeConfig file (Follow this document to do this using One Desktop) , I can prune it to only contain what I need: not forgetting the keycloak section we configured before. For local:
👉See attachment: 2_RuntimeConfig_pathvar_rdmapp_local
👉See attachment: 2_RuntimeConfig_pathvar_rdmapp_server
3. ServerConfig for Http dispatcher
We need to secure the online services endpoint of location
/rdm. Let’s say a decision has been made that the data extraction endpoints are to be publicly available, so no authentication is required. The configuration would look like this:
👉See attachment: 3_ServerConfig
The ordering of the
interceptUrl tags are significant. Tags are evaluated top to bottom, so always make sure the catch-all case (
access="isAuthenticated()" pattern="/**") is last.
For example Admin Centre URL is
https://localhost:8888. For any URL that start with
https://localhost:8888/rdm, no credentials are required.
Anything else will require valid credentials.
Workflow and Scheduler
Plans or components need to be part of an online service (see above), or be referenced by a workflow to run. When configured, Admin Centre will allow the user to run schedulers and workflows. In the example below I enable the Admin Centre to run scheduler, workflow and plans.
I now want to enable the Admin Centre to run scheduler, workflow and plans.
Admin Centre will allow the user to run schedulers and workflows, and plans or components would always need to be referenced by a workflow (or as part of online service, see above).
ServerConfig for Scheduler component configuration
👉See attachment: 3_ServerConfig_scheduler_local
👉See attachment: 3_ServerConfig_scheduler_server
ServerConfig for Workflow component configuration
👉See attachment: 3_ServerConfig_workflow_local
👉See attachment: 3_ServerConfig_workflow_server
RuntimeConfig for workflow and components
👉See attachment: RuntimeConfig_workflow_local
👉See attachment: RuntimeConfig_workflow_server
Security for workflow by role
Let’s say you only want users with keycloak role
DEVOPS to access schedulers on the Admin Centre. This is how you would add to the
interceptUrls on the serverConfig:
👉See attachment: security_for_workflow_by_role
Deploy your files and test them locally
As with working with any configuration files or change processes:
- Keep a backup of your files before change in case you need to revert to them
- Add comments to places you have changed so people who read the config file in the future can understand what’s been changed and why
- Especially after config files are changed, check the logs on start up to ensure server starts correctly. You’ll be looking for a message of “Server Startup Complete”
Once you completed all your changes, start your server through your usual means and the new components should be activated.
- example server log file
Using java at: "C:\Ataccama\13.8.0_preGA\jre\bin\java.exe"
Using DQC at: "C:\Ataccama\13.8.0_preGA\runtime"
2022-06-14 18:12:18 [INFO] com.ataccama.dqc.server.bin.OnlineCtl messageId=generic_msg message=Starting server on localhost
2022-06-14 18:12:18 [INFO] com.ataccama.dqc.commons.license.LicenseManager messageId=generic_msg message=Using following licenses:
2022-06-14 18:12:18 [INFO] com.ataccama.dqc.commons.license.LicenseManager messageId=generic_msg message= C:\Users\may.kwok\emp_may.kwok_13.plf
2022-06-14 18:12:19 [INFO] com.ataccama.lib.securityutil.javaencryption.DefaultEncryptionEngine messageId=createDefaultKeyProvider reason=the 'properties.encryption.keystore' property not specified
2022-06-14 18:12:19 [INFO] com.ataccama.lib.securityutil.javaencryption.DefaultEncryptionEngine messageId=createDefaultKeyProvider reason=the 'internal.encryption.keystore' property not specified
2022-06-14 18:12:20 [INFO] com.ataccama.dqc.web.logging.LoggingComponentInstance messageId=generic_msg message=Switching to logging setup from logging.xml
2022-06-14 18:12:20 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'HttpDispatcher'
2022-06-14 18:12:20 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'LoggingComponent'
2022-06-14 18:12:20 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'HealthStateProviders'
2022-06-14 18:12:22 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'WebConsoleComponent'
2022-06-14 18:12:22 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'HealthStateWebConsole'
2022-06-14 18:12:22 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'ManagementService'
2022-06-14 18:12:22 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'OnlineServicesComponent'
2022-06-14 18:12:22 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'WorkflowServerComponent'
2022-06-14 18:12:22 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Starting 'SchedulerServerComponent'
2022-06-14 18:12:24 [INFO] com.ataccama.dqc.web.jetty.JettyListener$1 messageId=listening message=Listening for HTTP requests, port=8888
2022-06-14 18:12:24 [INFO] com.ataccama.dqc.commons.logging.appenders.Slf4jLogAppender messageId= Server startup complete.
Check the UI and Server Health Status
- On start up of Admin Centre, open a web browser and navigate to the web url (
https://localhost:8888) in our example, to make sure it’s started correctly.
- Check the UI. It should look something like this:
- Note the applications (where online services sit), workflow, and scheduler sections. These sections highlighted above will only show if added to the configuration.
- If the server does not start up correctly, the logs will show errors. Fix the errors in the configuration file(s), then start the server again. Repeat until it starts correctly.
- Some frequent errors include:
- runtimeConfig is invalid
- The full message could look something like this:
Unable to parse runtime configuration file; nested exception is: com.ataccama.dqc.processor.engine.InvalidRuntimeConfigException: Runtime configuration 'C:\\Ataccama\\13.8.0_preGA\\runtime\\server\\etc\\default.runtimeConfig' is invalid
- Check the xml structure of the runtimeConfig and make sure the structure is valid. Check further down the error message it could provide hints as to what exactly is wrong with the syntax.
- The full message could look something like this:
- serverConfig is invalid
- The full message is:
Server Configuration is not valid
- Check the structure of the serverConfig and make sure the structure is valid. Check the full log as it could provide hints as to what exactly is wrong with the syntax. As an example:
Folder 'C:\\Ataccama\\workspace\\workspace_1380_preGA\\projects\\rdm\\workflow' must exist. [offending property path=/serverComponents/sources/path]
- This points me to the 6th server component, a source path defined doesn’t exist. Indeed upon checking, there was a typo in the path and so the server configuration could not find the folder.
- The full message is:
- runtimeConfig is invalid
- Check Server Health Status to see if all connections are happy:
As we can see above, my database connection failed. This means the structure of the connection configuration seems correct (otherwise the server won’t start), but the connection is failing. The next step in the above example would be to check if the database credentials are correct, or if the database itself is up, and continue investigating from there.
Also showing above, the online services has a grey question mark so the .online file and its associated plan / component should also be checked.
Test User Permissions
In our example above, we restricted access to the scheduler to the
DEVOPS role. Have a go and try accessing the Scheduler node. If you don’t have the
DEVOPS role given, you should see this screen:
Otherwise if it is all green ticks, then your set up is complete.
Deploy to server
If you are satisfied with your server on your local workstation, you are ready to deploy the files to the server! Transfer the files to the server, and start up the Admin Centre. Go through the same checks as you would start up on local workstation. If there are no errors, well done!