Hello @bobparry!
This is a very good question. Unfortunately due to product limitations, you are exactly correct - if you need a rule involving multiple tables you must create a new Catalog Item (either with SQL, ONE Desktop Components, or the Transformation Editor in v15+), and attach your rule to it. It then has no impact on your original table’s Data Quality.
We are, however, actively tracking this exact use case, as it is a very common one. Could you please create an idea for this on the community? Thank you!
@bobparry Hi! To add to what @OGordon100 wrote, for the time being you might make use of catalog item relationships to help users discover SQL CIs related to source CIs. With a bit of training users can get used to checking those relationships whenever they are browsing the catalog. Of course if you want to build some automations based on DQ scores, you'd need some logic to translate DQ issues from the SQL CI to the source CI - that's something you could potentially achieve with the metadata model customizations and an automation workflow that would export DQ results from an SQL CI asset and import it to the source CI asset. Let me know if you are interested and have questions.
That said, please do raise a feature idea here on the community too! Hearing from our customers helps our product team tremendously to prioritize and refine their plans. Thank you!
Thanks Gordon and Lisa, I’ve created the idea here: Enable Data Quality Rules based on Joins | Community
@Lisa Kovalskaia, can you elaborate on what you mean about the automations based on DQ scores? Do you mean having a metadata field, and populating that with the Data Quality score from the SQL catalog item, and then have users look at that field rather than the Overall score?
To your point about the item relationships, is there any way to automate the creation of those? Especially for SQL Catalog items it feels like the Source catalog items they are related to should automatically appear in the Relationships pane, given the environment knows exactly how they are related. It’s a great idea to encourage users to check the relationships in cases like this, it would be great if it was a little easier to have those relationships populated.
@bobparry regarding automations, yes, what you’re describing is close to what I had in mind - although I think this would be more useful for triggering some automatic actions on the source item, such as DQ alerts or data remediation tasks or DQ reporting. I wouldn’t necessarily highlight that new field in the UI. I feel like for end users that would add more confusion than benefit, since the metrics in the additional DQ metadata field and the standard Data Quality tab would not always align.
Regarding relationships, I agree it would be a nice improvement to see them created automatically based on the SQL CI query. Currently they have to be created manually. Depending on the size of your catalog you may also consider configuring an orchestration workflow to automate that, e.g. parsing the SQL CI query, looking for the CI with the same name and path, and creating the relationship. Then again, if there’s say one new SQL CI per week or per month, it might be easier for the data stewards to keep setting these relationships. What do you think?
Hi @bobparry, I’m closing this thread for now, if you have any follow-up questions please share them here or feel free to create a new post
♀️