Extracting data quality results- deatiled failed records

+1

Interesting let us know how this unfolds. We have a similar challenge.

We have Gen3 and monitoring projects are really nice, so maybe migrate first to Gen2. We use a monitoring project for each project - may contain several catalog items. I’ve created a template (component) for the post processing to a data quality repository - as we report and process further from there. Applying this component will store the info from the monitoring project in a database.

We have a similar use case for storing DQ results into a backend database also that we are considering.  We are using v13.9 and looking to migrate to V14 in the near future.  We would like the aggregated metrics and well as the detailed invalid or exception records. 

Any advice would be more than welcome on the best practice or best methods for doing this.

Thanks

Hi folks, I second @Marnix Wisselaar’s suggestion. To follow up on that, you could also try to use the post-processing plan (per single table) and filter the rows with invalid rule (there's a column for that).

From that point, you can continue with the action you’d like to take, i.e. putting it into SQL DB.

The graphql APIs can’t return per-record info/data; the full data doesn't leave the DPE processing (except for a small sample of invalid results), and we do not store it in detail, only the aggregated results. 

Getting data in CSV format is not a recommended practice since the file can be extremely large. Therefore, we recommend using the plan approach as opposed to the graphql APIs (stream-like processing). Hope this helps!