Mike Schornak

Use "reference" files to validate data?

Is there a way, with the free DQ Analyzer, to compare/validate the values in fields on one file against a list of values in other separate "reference" files?
1 person has
this question
+1
Reply

  • Hello,
    there isn't support for Dictionary files in DQ Analyzer, but you may use Foreign Key analysis feature with Drill-through to find the values, which are and are not in the reference file.

    You can do that by Creating a profile using both files as an input and creating a plan. After opening the Profiling step, there is a Foreign keys tab on the root node, where you add columns for analysis between the sources.

    Also in a case you do not have many values in reference files, you may validate the data using expression "is in {}". For example:
    src_gender is in {'F', 'M', 'FEMALE', 'MALE'}
    This won't help you if you have a lot of enumerated values in the reference file, unless you want to rewrite them manually.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly

  • In the DQC, we are using a Lookup files for validation against reference files. The Lookup files can be used in many ways (in many different steps) and usage of them is described in a DQC Lookup Video Tutorial.
  • (some HTML allowed)
    How does this make you feel?
    Add Image
    I'm

    e.g. happy, confident, thankful, excited sad, anxious, confused, frustrated indifferent, undecided, unconcerned kidding, amused, unsure, silly