Skip to main content
Solved

Query the monitoring report results at item level

  • November 8, 2024
  • 2 replies
  • 58 views

I’m hoping anyone has some good GraphQL examples to share. For us, it is necessary to integrate Ataccama into our standard reporting. To do so I’m trying to get a GraphQL working that returns the scores for each item in a monitoring project. Ideally, listing out all monitoring projects and items with their scores - for the latest processing (although I’ll take all at this point and filter the old ones later). This is all about the statistics (# records passed/failed etc. not the individual results - I am aware I can create post-processing plans to export data, but this is not applicable here).

It’s basically the overview from the report tab in the monitoring project, at item level:

I can get everything I need, including the total row count except the passed or failed figure. I have tried and looked at all GraphQL examples everywhere, including favicon-16x16.pngONE API :: Ataccama ONE, and have pieced this together based on the structures visible in the desktop app. Things like DqAggregationResultOverTime don’t really cut it, because it already aggregates across dq dimensions, and I’m interested in the per-item information.

How can I report on passed/failed for each item in a monitoring project, ideally without explicitly listing the catalog item (because we have hundreds)?

The query below is close, all it’s missing is the passed and/or failed count.

query listMonitoringProject {
    monitoringProject(gid: "<gid>") {
        gid
        publishedVersion {
            name
            processings {
                edges {
                    node {
                        gid
                        publishedVersion {
                            __typename
                            startedAt
                            state
                            result
                            dqResult {
                                publishedVersion {
                                    __typename
                                    startedCount
                                    statistics {
                                        publishedVersion {
                                            successCount
                                        }
                                    }
                                }
                            }
                            items {
                                edges {
                                    node {
                                        publishedVersion {
                                            __typename
                                            catalogItem {
                                                publishedVersion {
                                                    name
                                                }
                                            }
                                            result
                                            dqResult {
                                                __typename
                                                gid
                                                publishedVersion {
                                                    result
                                                    dqResult {
                                                        storedVersion {
                                                            __typename
                                                            ruleCount
                                                            recordCount

                                                        }
                                                    }                  
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}

 

Best answer by RoelantVos

Hi all,

In case others attempt something similar, this is not directly possible and requires multiple data sets to be retrieved and combined.

Ataccama support sent a helpful component for the desktop application, and that works. I’ve also refactored this into GraphQL calls, and it’s possible to piece this overview together by combining the following:

Get the passed/failed indicator for each dimension pathway.

query listDataQualityDimensions {
dqDimensionResults(
    versionSelector: { publishedVersion: true }
  ) {
    edges {
      node {
        gid
        publishedVersion {
          _displayName
          passedOverallResult
        }
      }
    }
  }
}

Get the monitoring projects and catalog items following the configuration details. 

query listMonitoringProject 
            {
            monitoringProjects (versionSelector: { draftVersion: false }) {
                edges {
                    node {    
                        gid
                        publishedVersion {
                            __typename
                            name  
                            configuration {                
                                __typename
                                publishedVersion {
                                    items {
                                        edges {
                                            node {
                                                gid
                                                publishedVersion {                                    
                                                    displayName
                                                    dqChecks {
                                                        edges {
                                                            node {
                                                                __typename
                                                                gid
                                                                publishedVersion
                                                                {
                                                                    __typename
                                                                    displayName
                                                                    mappings {
                                                                        edges {
                                                                            node {
                                                                                __typename
                                                                                gid
                                                                                publishedVersion
                                                                                {
                                                                                    catalogItemAttribute 
                                                                                    {
                                                                                        publishedVersion 
                                                                                        {
                                                                                            name
                                                                                        }
                                                                                    }
                                                                                }                                                     
                                                                            }
                                                                        }
                                                                    }
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            }          
                        }
                    }
                }
            }
        }

Use these details to collect the results per project / item / check using the dqCheckResultsOverTime query:

query CheckResults {
          monitoringProject(gid: "<monitoring_project_gid>") {
            publishedVersion {
              dqCheckResultOverTime(
                selector: { limit: 1 }
                catalogItemId: "<item_gid>"
                checkIds: ["<dq_check_gid>"]
              ) {
                processingId
                startedAt
                results {
                  dqCheckId
                  results {
                    id
                    count
                    name
                  }
                }
              }
            }
          }
        }

Combining these three sets can re-create this overview.

The desktop app component uses the same building blocks.

View original
Did this topic help you find an answer to your question?

2 replies

  • Author
  • Data Voyager
  • 3 replies
  • Answer
  • November 24, 2024

Hi all,

In case others attempt something similar, this is not directly possible and requires multiple data sets to be retrieved and combined.

Ataccama support sent a helpful component for the desktop application, and that works. I’ve also refactored this into GraphQL calls, and it’s possible to piece this overview together by combining the following:

Get the passed/failed indicator for each dimension pathway.

query listDataQualityDimensions {
dqDimensionResults(
    versionSelector: { publishedVersion: true }
  ) {
    edges {
      node {
        gid
        publishedVersion {
          _displayName
          passedOverallResult
        }
      }
    }
  }
}

Get the monitoring projects and catalog items following the configuration details. 

query listMonitoringProject 
            {
            monitoringProjects (versionSelector: { draftVersion: false }) {
                edges {
                    node {    
                        gid
                        publishedVersion {
                            __typename
                            name  
                            configuration {                
                                __typename
                                publishedVersion {
                                    items {
                                        edges {
                                            node {
                                                gid
                                                publishedVersion {                                    
                                                    displayName
                                                    dqChecks {
                                                        edges {
                                                            node {
                                                                __typename
                                                                gid
                                                                publishedVersion
                                                                {
                                                                    __typename
                                                                    displayName
                                                                    mappings {
                                                                        edges {
                                                                            node {
                                                                                __typename
                                                                                gid
                                                                                publishedVersion
                                                                                {
                                                                                    catalogItemAttribute 
                                                                                    {
                                                                                        publishedVersion 
                                                                                        {
                                                                                            name
                                                                                        }
                                                                                    }
                                                                                }                                                     
                                                                            }
                                                                        }
                                                                    }
                                                                }
                                                            }
                                                        }
                                                    }
                                                }
                                            }
                                        }
                                    }
                                }
                            }          
                        }
                    }
                }
            }
        }

Use these details to collect the results per project / item / check using the dqCheckResultsOverTime query:

query CheckResults {
          monitoringProject(gid: "<monitoring_project_gid>") {
            publishedVersion {
              dqCheckResultOverTime(
                selector: { limit: 1 }
                catalogItemId: "<item_gid>"
                checkIds: ["<dq_check_gid>"]
              ) {
                processingId
                startedAt
                results {
                  dqCheckId
                  results {
                    id
                    count
                    name
                  }
                }
              }
            }
          }
        }

Combining these three sets can re-create this overview.

The desktop app component uses the same building blocks.


Cansu
Community Manager
Forum|alt.badge.img+3
  • Community Manager
  • 625 replies
  • December 4, 2024

Hi ​@RoelantVos, thank you for coming back and sharing the solution here - really helpful for anyone looking to implement a similar reporting πŸ™Œ


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings