Skip to main content
Solved

Need DQ Rule

  • September 7, 2022
  • 4 replies
  • 191 views

Forum|alt.badge.img+1

How to 

Check whether the target column has only DATE values or not.

?

Best answer by anna.spakova

Hello @KarthikaSPillai , thanks for clarifying. I don’t think there is some native check for the data type. You see the data type after the Catalog Item is imported into Catalog and as I mentioned above, you can verify in the Profiling results, if the values correspond to what you expect.

From the DQ rules perspective, you can write rules that will check if the values are e.g. only numbers (example would be to use trashNonDigits function and compare to the original value) etc., but I can imagine this can get pretty complicated for DATE as you can have different date formats. You may need to use some regex expressions… Also important is to keep in mind that the rule can get assigned only to the attribute with the corresponding data type as the input attribute of the rule (so if you have an INTEGER column, you are not able to assign rule with STRING as an input attribute).

View original
Did this topic help you find an answer to your question?

4 replies

anna.spakova
Ataccamer
Forum|alt.badge.img+3
  • Ataccamer
  • 144 replies
  • September 8, 2022

Hi,

if you have the data with the column (Catalog Item) in a catalog, you can check the profiling results to see what the data in the column look like.

 

Does that answer your question?


Forum|alt.badge.img+1

Hello @anna.spakova I need to write DQ Evaluation rule to check whether the data in the column is in intended data type. Eg., Is the column defined as integer and all contents of the column as integer?


anna.spakova
Ataccamer
Forum|alt.badge.img+3
  • Ataccamer
  • 144 replies
  • Answer
  • September 9, 2022

Hello @KarthikaSPillai , thanks for clarifying. I don’t think there is some native check for the data type. You see the data type after the Catalog Item is imported into Catalog and as I mentioned above, you can verify in the Profiling results, if the values correspond to what you expect.

From the DQ rules perspective, you can write rules that will check if the values are e.g. only numbers (example would be to use trashNonDigits function and compare to the original value) etc., but I can imagine this can get pretty complicated for DATE as you can have different date formats. You may need to use some regex expressions… Also important is to keep in mind that the rule can get assigned only to the attribute with the corresponding data type as the input attribute of the rule (so if you have an INTEGER column, you are not able to assign rule with STRING as an input attribute).


Forum|alt.badge.img+3
  • Data Pioneer L2
  • 31 replies
  • September 13, 2022

You can assoicate a term to this column and have DQ rules associated to TERM, this will ensure the underlying rule is executed whenever you reprofile and validate dq evaluation results


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings