Hi everyone,

In this post, we continue our discussion on Data Observability configurations today! Let's dive into some advanced settings, terms, and rules that will enhance your data observability experience.
 

Observing Data with Selected Terms

1. Choose your preferred method to select the data you want to observe.
   • If you select Terms:
     1. Click on Add terms and select terms associated with important data from the available list.
     2. Once you've selected the terms, all associated catalog items will be chosen for monitoring. If necessary, you can exclude individual catalog items by selecting Edit selection.
        • After choosing Edit selection, you will see all catalog items associated with the selected terms. Use the more options icon and select Exclude to exclude any of these items from monitoring.
        • When you're ready, click Save.

• If you select AQL:
  1. Use Ataccama Query Language to define the data you want to observe. Once you've selected the terms, all associated catalog items will be chosen for monitoring. If necessary, you can exclude individual catalog items by selecting Preview.
     • After selecting Preview, you will see the data selected based on your AQL query. If needed, use the more options icon and select Exclude to exclude any of these items from monitoring.
     • When you're ready, click Save.

2. Configure the following settings under Processing settings:
   1. Specify whether Overall data quality should be calculated. Enabling this option allows data quality monitoring based on rules associated with the selected terms.
      • Specify whether the Manual threshold is active. If enabled, specify the required threshold (%). You will receive alerts when changes in data quality exceed this threshold.
      • Specify whether the Adaptive threshold is active. Enabling this option will trigger alerts when the AI detects significant relative changes in percentage.
        • You need to select the desired sensitivity level: Very Low, Low, Medium, High, or Very High. Higher sensitivity may detect more changes but can lead to false positives, while lower sensitivity reduces the number of detected changes but can result in false negatives.
           

          

           

   2. Specify whether Anomaly detection and record volume checks are active. When enabled, AI-driven anomaly detection is activated, and you will be alerted when anomalous values are detected or if the number of records in each table deviates from expectations.
      • You need to select the desired sensitivity level: Very Low, Low, Medium, High, or Very High. Higher sensitivity may detect more anomalous values but can result in false positives, while lower sensitivity reduces the number of detected anomalies but can result in false negatives.
   3. Specify whether Business term detection is active. When enabled, terms are automatically assigned based on AI and detection rules.
      • If business term detection is active, you can also enable alerts for the following instances:
        • Selected term is detected: Receive an alert when selected terms from the glossary are detected in the source for the first time. Use Add Term to select the terms you want to be alerted about.
        • Any new term is detected: Receive an alert when any term is detected in the source for the first time.
        • For term alerts, you will be notified only the first time a term is detected.
        • Selected occurrence is detected: Receive an alert when selected terms from the glossary are detected on catalog items in the source. Use Add Term to select the terms you want to be alerted about.
        • Any new term occurrence is detected: Receive an alert when any term is newly detected on catalog items in the source.
        • For term occurrence alerts, you will be alerted every time the term is detected on catalog items or attributes.

          

      • Additionally, select Include manually added terms to receive notifications when a term is manually applied to a catalog item or attribute.

3. Schedule monitoring using either basic or advanced settings:
   • For Basic settings:
     1. Choose the intervals at which you want the monitoring to run under Repeat.
     2. Select the desired time, day, and date. The available options depend on the selected interval.
     3. Choose the appropriate time zone.
     4. Use the checkbox to queue the checks if the system is unavailable at the scheduled time.
   • For Advanced settings:
     1. Define the monitoring schedule using cron expression syntax.
     2. Select the desired time zone.
     3. Use the checkbox to queue the checks if the system is unavailable at the scheduled time.

Observing Data without Selected Terms

Customize processing settings for additional checks on unselected catalog items.

1. Specify whether Business term detection is active. When enabled, terms can be assigned to new catalog items if appropriate. If the terms selected for monitoring are assigned to new items, the system will perform additional checks on these items.
   • If business term detection is active, you can also enable notifications. You can either use the same notification settings as for the selected data (by selecting the checkbox) or define new notification settings as described below:
     1. Select Send me notifications. You can receive alerts for the following instances:
        • Selected term is detected: Receive an alert when selected terms from the glossary are detected in this source for the first time. Use Add Term to select the terms you want to be alerted about.
        • Any new term is detected: Receive an alert when any term is detected in the source for the first time.
        • For term alerts, you will be notified only the first time a term is detected.
        • Selected occurrence is detected: Receive an alert when selected terms from the glossary are detected on catalog items in the source. Use Add Term to select the terms you want to be alerted about.
        • Any new term occurrence is detected: Receive an alert when any term is newly detected on catalog items in the source.
        • For term occurrence alerts, you will be alerted every time the term is detected on catalog items or attributes.
     2. Select Include manually added terms to receive notifications when a term is manually applied to a catalog item or attribute.
2. Schedule monitoring using either basic or advanced settings, following the same steps as mentioned earlier.

Monitoring Schema Changes

Stay informed about any changes in the structure of metadata, such as missing columns or data type changes. Please note that VCIs and files are not included in schema change checks.

1. Select Set schemas to monitor changes in additional schemas that have not been imported into the catalog.
2. Specify whether Schema changes should be actively monitored. When enabled, you will receive alerts about changes in the data structure based on the settings below.
   • Please note that the options not selected here will not trigger notifications or appear as issues in the data observability dashboard. However, they will count towards the overall schema change statistics.
   • Table:
     • New: Receive alerts for newly detected tables.
     • Missing: Receive alerts for missing tables.
   • Column:
     • New: Receive alerts for newly detected columns.
     • Missing: Receive alerts for missing columns.
     • Data type: Receive alerts for changes in column data types.
3. Schedule monitoring using either basic or advanced settings, following the same steps as mentioned earlier.
    

   

    

Configuring Notifications

Choose how you would like to be alerted. You can select multiple options and enable/disable each option using the checkboxes.

• In-app: Select groups or roles within the application to receive alerts.
• Email: Receive alerts via email.
• Slack: Enter the channel name to receive alerts on Slack.
  • If Ataccama ONE is not connected to Slack, you can connect it to your workspace in the general settings.
• Microsoft Teams: Receive alerts on Microsoft Teams.

Any questions, comments let us know!