Get Invalid Data into DB

Hello ​@kpillai ,

can you please provide the graphql query that you are trying to parse? The invalid samples are stored in Minio, so AFAIK they are not available under API call (but I’ve never tried). This will give you only samples. To get full invalid results, you need to use the post-processing plan/post-processing transformation.

Kind regards,

Anna

Yes ​@anna.spakova  that makes sense. As of now I managed to get the values, column name and explanation. But now the problem is I am getting all values instead of invalid samples. I am wondering how to apply a filter to only get invalid samples. Also the invalid explanation is appearing for all values in export via ONE Desktop Plan.

Here is the graphql:

query ProjectInvalidSamples($processingId: GID!, $resultItemFilter: [Filter!], $page: Int!, $pageSize: Int!, $rules: [GID!]!, $aggregations: [GID!]!, $dqFilters: [values!]) {
  monitoringProjectProcessing(gid: $processingId) {
    publishedVersion {
      items(filter: $resultItemFilter) {
        edges {
          node {
            __typename
            storedVersion {
              dqResult {
                storedVersion {
                  dqResult {
                    storedVersion {
                      drillthrough(
                        page: $page
                        pageSize: $pageSize
                        
                        rules: $rules
                        aggregations: $aggregations
                        filter: $dqFilters
                      ) {
                        totalCount
                        headers {
                          attributeId
                          dataPolicy
                          __typename
                        }
                        rows {
                          values
                          recordId
                          invalidRuleInstances {
                            
                            ruleInstanceId
                            explanation
                            __typename
                          }
                          __typename
                        }
                        __typename
                      }
                      __typename
                    }
                    __typename
                  }
                  __typename
                }
                __typename
              }
              __typename
            }
            __typename
          }
          __typename
        }
        __typename
      }
      __typename
    }
    __typename
  }
}

Hi ​@kpillai ,

just to confirm - are you certain you are getting some valid samples? Meaning records where there is no invalid value? Because the invalid samples (how the name suggests) offer a sample of invalid records, so there is always at least one rule that the record failed (AFAIK at least :) ). If you filter in the plan after retrieving the values using the invalidRuleInstances column, there shouldn’t be any record with an empty field.

If this is not what you meant, can you clarify what do you mean by getting all values?

Btw I used the same query in my testing env. and I am getting the explanation only for the failed rules- For example, on this screenshot, there 2 two rules the record failed (so two invalidRulesInstances)

What is NOT available in the response is what the attributes are where those rules were applied. But you can use the ruleInstanceId to query the ruleInstances (dqChecks) in your project to get that information (through dqCheckMappings I believe).

And since the sample contains only invalid records, there should always be at least one invalidRuleInstance for that row.

Kind regards,

Anna