Hello everyone !

 I’m Srija ,Solutions Consultant here at Ataccama. Today, I’d like to talk about how to trigger profiling while running a monitoring project.

While running the monitoring project, the profiling process is getting triggered without any schedule or event created on the catalog item to initiate profiling. This unexpected profiling of over 1000+ records significantly slows down the system. The likely cause for this issue is the anomaly detection feature being enabled on the catalog item.

To address this issue, you need to check if anomaly detection is enabled on the catalog item.

Anomaly Detection in Ataccama DQ

What Anomaly Detection Does:

Anomaly detection in Ataccama Data Quality (DQ) automatically identifies outliers and unusual patterns in your data. This feature is designed to ensure data quality by detecting anomalies such as unexpected values, missing data, or patterns that do not conform to the expected behavior.

Implications of Enabling Anomaly Detection:

1. Automatic Profiling: When anomaly detection is enabled, Ataccama may automatically trigger profiling to gather necessary statistics and identify anomalies. This can happen without explicit scheduling or manual intervention.
2. System Performance: Enabling anomaly detection, especially on large datasets, can consume significant system resources and potentially slow down other processes, such as monitoring projects, due to the additional computational load required for profiling.
3. Data Quality Insights: Despite the performance impact, anomaly detection provides valuable insights into the quality of your data by highlighting data issues that might otherwise go unnoticed.

Recommended Actions

1. Check Anomaly Detection Settings:
   • Verify if anomaly detection is enabled on the catalog item associated with the monitoring project.
   • If enabled, consider the necessity of this feature based on your data quality requirements.
2. Disable Anomaly Detection (if necessary):
   • If the automatic profiling triggered by anomaly detection is causing significant performance issues, consider disabling this feature temporarily or configuring it to run at off-peak times to minimize impact.
3. Optimize Performance:
   • If anomaly detection is essential, explore ways to optimize system performance. This might include increasing system resources, optimizing data storage, or refining the scope of anomaly detection to target specific areas of interest.

Steps to Disable Anomaly Detection

1. Open Ataccama ONE Desktop:
   • Navigate to the catalog item.
2. Access Anomaly Detection Settings:
   • Go to the settings or configuration section for the catalog item.
3. Disable Anomaly Detection:
   • Locate the anomaly detection feature and disable it, or adjust its configuration to reduce its impact on system performance.
4. Save Changes:
   • Ensure you save any changes made to the settings.

Please find the document regarding the anomaly detection - https://docs.ataccama.com/one/latest/anomaly-detection/anomaly-detection-in-catalog-items.html

By following these steps and recommendations, you can manage the impact of profiling on system performance and ensure that your monitoring projects run smoothly. If you have further questions or need assistance, please let us know!