Skip to main content
Solved

Query the monitoring report results at item level

  • November 8, 2024
  • 2 replies
  • 66 views

Iโ€™m hoping anyone has some good GraphQL examples to share. For us, it is necessary to integrate Ataccama into our standard reporting. To do so Iโ€™m trying to get a GraphQL working that returns the scores for each item in a monitoring project. Ideally, listing out all monitoring projects and items with their scores - for the latest processing (although Iโ€™ll take all at this point and filter the old ones later). This is all about the statistics (# records passed/failed etc. not the individual results - I am aware I can create post-processing plans to export data, but this is not applicable here).

Itโ€™s basically the overview from the report tab in the monitoring project, at item level:

I can get everything I need, including the total row count except the passed or failed figure. I have tried and looked at all GraphQL examples everywhere, including favicon-16x16.pngONE API :: Ataccama ONE, and have pieced this together based on the structures visible in the desktop app. Things like DqAggregationResultOverTime donโ€™t really cut it, because it already aggregates across dq dimensions, and Iโ€™m interested in the per-item information.

How can I report on passed/failed for each item in a monitoring project, ideally without explicitly listing the catalog item (because we have hundreds)?

The query below is close, all itโ€™s missing is the passed and/or failed count.

query listMonitoringProject {
    monitoringProject(gid: "<gid>") {
        gid
        publishedVersion {
            name
            processings {
                edges {
                    node {
                        gid
                        publishedVersion {
                            __typename
                            startedAt
                            state
                            result
                            dqResult {
                                publishedVersion {
                                    __typename
                                    startedCount
                                    statistics {
                                        publishedVersion {
                                            successCount
                                        }
                                    }
                                }
                            }
                            items {
                                edges {
                                    node {
                                        publishedVersion {
                                            __typename
                                            catalogItem {
                                                publishedVersion {
                                                    name
                                                }
                                            }
                                            result
                                            dqResult {
                                                __typename
                                                gid
                                                publishedVersion {
                                                    result
                                                    dqResult {
                                                        storedVersion {
                                                            __typename
                                                            ruleCount
                                                            recordCount

                                                        }
                                                    }                  
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

 

Best answer by RoelantVos

Hi all,

In case others attempt something similar, this is not directly possible and requires multiple data sets to be retrieved and combined.

Ataccama support sent a helpful component for the desktop application, and that works. Iโ€™ve also refactored this into GraphQL calls, and itโ€™s possible to piece this overview together by combining the following:

Get the passed/failed indicator for each dimension pathway.

query listDataQualityDimensions {
dqDimensionResults(
    versionSelector: { publishedVersion: true }
  ) {
    edges {
      node {
        gid
        publishedVersion {
          _displayName
          passedOverallResult
        }
      }
    }
  }
}

Get the monitoring projects and catalog items following the configuration details. 

query listMonitoringProject 
            {
            monitoringProjects (versionSelector: { draftVersion: false }) {
                edges {
                    node {    
                        gid
                        publishedVersion {
                            __typename
                            name  
                            configuration {                
                                __typename
                                publishedVersion {
                                    items {
                                        edges {
                                            node {
                                                gid
                                                publishedVersion {                                    
                                                    displayName
                                                    dqChecks {
                                                        edges {
                                                            node {
                                                                __typename
                                                                gid
                                                                publishedVersion
                                                                {
                                                                    __typename
                                                                    displayName
                                                                    mappings {
                                                                        edges {
                                                                            node {
                                                                                __typename
                                                                                gid
                                                                                publishedVersion
                                                                                {
                                                                                    catalogItemAttribute 
                                                                                    {
                                                                                        publishedVersion 
                                                                                        {
                                                                                            name
                                                                                        }
                                                                                    }
                                                                                }                                                     
                                                                            }
                                                                        }
                                                                    }
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            }          
                        }
                    }
                }
            }
        }

Use these details to collect the results per project / item / check using the dqCheckResultsOverTime query:

query CheckResults {
          monitoringProject(gid: "<monitoring_project_gid>") {
            publishedVersion {
              dqCheckResultOverTime(
                selector: { limit: 1 }
                catalogItemId: "<item_gid>"
                checkIds: ["<dq_check_gid>"]
              ) {
                processingId
                startedAt
                results {
                  dqCheckId
                  results {
                    id
                    count
                    name
                  }
                }
              }
            }
          }
        }

Combining these three sets can re-create this overview.

The desktop app component uses the same building blocks.

View original

  • Data Voyager
  • November 24, 2024

Hi all,

In case others attempt something similar, this is not directly possible and requires multiple data sets to be retrieved and combined.

Ataccama support sent a helpful component for the desktop application, and that works. Iโ€™ve also refactored this into GraphQL calls, and itโ€™s possible to piece this overview together by combining the following:

Get the passed/failed indicator for each dimension pathway.

query listDataQualityDimensions {
dqDimensionResults(
    versionSelector: { publishedVersion: true }
  ) {
    edges {
      node {
        gid
        publishedVersion {
          _displayName
          passedOverallResult
        }
      }
    }
  }
}

Get the monitoring projects and catalog items following the configuration details. 

query listMonitoringProject 
            {
            monitoringProjects (versionSelector: { draftVersion: false }) {
                edges {
                    node {    
                        gid
                        publishedVersion {
                            __typename
                            name  
                            configuration {                
                                __typename
                                publishedVersion {
                                    items {
                                        edges {
                                            node {
                                                gid
                                                publishedVersion {                                    
                                                    displayName
                                                    dqChecks {
                                                        edges {
                                                            node {
                                                                __typename
                                                                gid
                                                                publishedVersion
                                                                {
                                                                    __typename
                                                                    displayName
                                                                    mappings {
                                                                        edges {
                                                                            node {
                                                                                __typename
                                                                                gid
                                                                                publishedVersion
                                                                                {
                                                                                    catalogItemAttribute 
                                                                                    {
                                                                                        publishedVersion 
                                                                                        {
                                                                                            name
                                                                                        }
                                                                                    }
                                                                                }                                                     
                                                                            }
                                                                        }
                                                                    }
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            }          
                        }
                    }
                }
            }
        }

Use these details to collect the results per project / item / check using the dqCheckResultsOverTime query:

query CheckResults {
          monitoringProject(gid: "<monitoring_project_gid>") {
            publishedVersion {
              dqCheckResultOverTime(
                selector: { limit: 1 }
                catalogItemId: "<item_gid>"
                checkIds: ["<dq_check_gid>"]
              ) {
                processingId
                startedAt
                results {
                  dqCheckId
                  results {
                    id
                    count
                    name
                  }
                }
              }
            }
          }
        }

Combining these three sets can re-create this overview.

The desktop app component uses the same building blocks.


Cansu
Community Manager
Forum|alt.badge.img+3
  • Community Manager
  • December 4, 2024

Hi โ€‹@RoelantVos, thank you for coming back and sharing the solution here - really helpful for anyone looking to implement a similar reporting ๐Ÿ™Œ


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings