Skip to main content
Solved

ONE Expression - Trim


JTH
Data Pioneer
Forum|alt.badge.img+2
  • Data Pioneer
  • 18 replies

Hi,

There are certain records that contain “white spaces” at the beginning and/or at the end. Ex: “    100137069       “, “    100017129       “, etc. We don't allow the use of white spaces and want to build a DQ rule to check the invalid records. Hence the advanced expression:  length(String_ATTRIBUTE_1) <> length(trim(String_ATTRIBUTE_1)). When I test the DQ rule it works as intended. I copy the above results and paste them in the test environment and it shows me the record is invalid. When I remove the white spaces it will considered valid. However, when I run the Monitoring Project based on the same records and DQ rule it gives me 100% validity. How is this possible? And maybe more importantly, how can I create a rule that will show the invalid records shown earlier?
 

 

Best answer by AKislyakov

Hi @JTH,

By default, Ataccama trims leading and trailing spaces for all columns. If you need to preserve them, you need to set following parameter:

-Dengine.trimStringValues=false (doc: JVM Configuration :: Ataccama ONE)

View original
Did this topic help you find an answer to your question?

6 replies

Forum|alt.badge.img+2
  • Ataccamer
  • 165 replies
  • Answer
  • May 30, 2024

Hi @JTH,

By default, Ataccama trims leading and trailing spaces for all columns. If you need to preserve them, you need to set following parameter:

-Dengine.trimStringValues=false (doc: JVM Configuration :: Ataccama ONE)


jdordregter
Data Voyager
Forum|alt.badge.img+1
  • Data Voyager
  • 6 replies
  • May 30, 2024

😂 So in this case Ataccama increases the data quality before checking the data quality. 


Cansu
Community Manager
Forum|alt.badge.img+3
  • Community Manager
  • 693 replies
  • June 3, 2024

Hello @JTH I’m closing this thread, please feel free to follow up here with any of your questions or create a new post 🙋‍♀️


Marcel-Jan
Star Blazer L2
Forum|alt.badge.img+1
  • Star Blazer L2
  • 21 replies
  • June 6, 2024

I still have some questions. I'm the person on @JTH 's team who implements the solution. And I don't understand where this -Dengine.trimStringValues property should be implemented. I read that it should be set in the Runtime Server. We have an on premise implementation of Ataccama One. And I don't think we have that running?

Normally when data is read I see it go passed the dpe, dpm or mmm-backend components. But they don't seem to have an engine related property.


JTH
Data Pioneer
Forum|alt.badge.img+2
  • Author
  • Data Pioneer
  • 18 replies
  • June 13, 2024

@AKislyakov @Cansu can any of you please follow up on @Marcel-Jan's comment so we can proceed on this issue further. We want to see what we can do with this solution 😊


Forum|alt.badge.img+2
  • Ataccamer
  • 165 replies
  • June 13, 2024

Hi @Marcel-Jan, @JTH 

For Ataccama ONE you need to set this property on DPE using plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS
Like following:

plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS=-Dengine.trimStringValues=false

Please note, that this parameter only works if plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.exec is configured (DPE Configuration :: Ataccama ONE)


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings