Skip to main content

Hi everyone!
 

In the last part of Monitoring Projects Configurations, we will talk about how to easily import configurations from one monitoring project to another, making your data quality monitoring more efficient. If you’ve missed Part 1&2 you can find them below:


Import the Configuration
 

Instead of creating a new project from scratch, you can copy the configuration of one monitoring project and apply it to another. This is particularly useful when you want to replicate settings from a testing environment to a production environment. The copied configuration includes all checks, invalid samples, reports structure, and DQ filters.

To import the configuration:

  1. Navigate to the project where you want to apply the configuration.

     

  2. Expand the more options menu and select Import configuration.
  3. Choose the monitoring project from which you want to copy the configuration, from the list provided.
  4. Check if the preselected catalog items are correct. You can modify the list by deleting items or selecting different ones from the catalog.
  5. Make sure the structure of the catalog items matches to proceed with the import.
  6. Click Start import at the bottom of the page. The configuration will be copied to the new monitoring project instantly.

Set a Purpose

Setting a purpose for catalog items is optional but helpful. It helps preselect catalog items for a monitoring project without manual assignment. To link a project with catalog items, ensure that both the project and items have the same purpose.

To set the purpose:

  • For the project, go to the Overview tab, click Add purpose in the Purpose widget, and choose from the preconfigured list.
  • For catalog items, go to the Overview tab of the item, click Add purpose in the Purpose widget, and select from the preconfigured list.

The list of purposes can be managed by power users via the List of Values in the Organization tab.

Example Use Case

Let's consider a 'development' and a 'production' environment:

  1. Users can test projects on the development environment and later apply the same configuration to the production environment.
  2. Users can comfortably test the report without affecting the production dashboard.

To achieve this:

  1. Add both a DEV and a PROD purpose in List of Values.

     

  2. Configure a monitoring project for the development environment and assign the DEV purpose.

     

  3. Create a monitoring project for the production environment and assign the PROD purpose.
  4. Copy the monitoring project configuration from development to production.
  5. Link the tables that correspond to development and production environments.

     

  6. Click Start import to import the configuration to the production monitoring project.

     

Try out these steps to start using the configuration import feature! If you have any questions please share them in the comments 👇

I’d like to add that based on my personal project experience this feature is in general a huge timesaver when you need to work with large monitoring projects that would have tenth of catalog items and hundreds of rules. Large international companies might have a need to deploy a multiple large scale monitoring project with identical configuration (for example country specific projects which would have almost identical content, require the same set of DQ checks and so on). Manual configuration of a large monitoring project with, for example, 20+ CI’s and let’s say 300+ DQ checks applied to the project can easily take a day of work. However, if you already have one such project that you can use as a reference configured and would like multiple new projects be configured the same way, you would need to spend just 10 minutes maximum to configure import mapping and run the procedure to create a new project with identical configuration but based on different CI’s.

It’s important to mention some of the existing limitations:
1) It is currently not possible to skip one of the CI’s in the import mapping. You always have to pick a mapping target CI for each of the CI’s that exist in the source projects. If you don’t pick a target, then the same CI from source project would be imported into new project which you would then have to delete.
2) There’s currently no ability to skip import of some of the DQ checks, so you would always import the whole configuration.
3) There’s no ability to resolve conflicts at the moment. For example, if the structures of source and target CI are different (some of the attributes to which DQ checks are applied in the source table do not exist in the target table) it will not be possible to import configuration to this particular CI and it will have to be configured inside of the Monitoring Project. All the CI’s which are mapped properly and have the same structure will be imported without problems.

In future releases we should see some improvements that will address current limitations and make this feature more flexible.

Reply


Community Terms of Use & ConditionsCookie settings