Skip to main content
Solved

ONE Expression - Trim


JTH
Data Pioneer
Forum|alt.badge.img+2
  • Data Pioneer
  • 16 replies

Hi,

There are certain records that contain β€œwhite spaces” at the beginning and/or at the end. Ex: β€œ    100137069       β€œ, β€œ    100017129       β€œ, etc. We don't allow the use of white spaces and want to build a DQ rule to check the invalid records. Hence the advanced expression:  length(String_ATTRIBUTE_1) <> length(trim(String_ATTRIBUTE_1)). When I test the DQ rule it works as intended. I copy the above results and paste them in the test environment and it shows me the record is invalid. When I remove the white spaces it will considered valid. However, when I run the Monitoring Project based on the same records and DQ rule it gives me 100% validity. How is this possible? And maybe more importantly, how can I create a rule that will show the invalid records shown earlier?
 

 

Best answer by AKislyakov

Hi @JTH,

By default, Ataccama trims leading and trailing spaces for all columns. If you need to preserve them, you need to set following parameter:

-Dengine.trimStringValues=false (doc: JVM Configuration :: Ataccama ONE)

View original
Did this topic help you find an answer to your question?

6 replies

Forum|alt.badge.img+2
  • Ataccamer
  • 150 replies
  • Answer
  • May 30, 2024

Hi @JTH,

By default, Ataccama trims leading and trailing spaces for all columns. If you need to preserve them, you need to set following parameter:

-Dengine.trimStringValues=false (doc: JVM Configuration :: Ataccama ONE)


jdordregter
Data Voyager
Forum|alt.badge.img+1
  • Data Voyager
  • 3 replies
  • May 30, 2024

πŸ˜‚ So in this case Ataccama increases the data quality before checking the data quality. 


Cansu
Community Manager
Forum|alt.badge.img+3
  • Community Manager
  • 633 replies
  • June 3, 2024

Hello @JTH I’m closing this thread, please feel free to follow up here with any of your questions or create a new post πŸ™‹β€β™€οΈ


Marcel-Jan
Star Blazer L2
Forum|alt.badge.img+1
  • Star Blazer L2
  • 21 replies
  • June 6, 2024

I still have some questions. I'm the person on @JTH 's team who implements the solution. And I don't understand where this -Dengine.trimStringValues property should be implemented. I read that it should be set in the Runtime Server. We have an on premise implementation of Ataccama One. And I don't think we have that running?

Normally when data is read I see it go passed the dpe, dpm or mmm-backend components. But they don't seem to have an engine related property.


JTH
Data Pioneer
Forum|alt.badge.img+2
  • Author
  • Data Pioneer
  • 16 replies
  • June 13, 2024

@AKislyakov @Cansu can any of you please follow up on @Marcel-Jan's comment so we can proceed on this issue further. We want to see what we can do with this solution πŸ˜Š


Forum|alt.badge.img+2
  • Ataccamer
  • 150 replies
  • June 13, 2024

Hi @Marcel-Jan, @JTH 

For Ataccama ONE you need to set this property on DPE using plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS
Like following:

plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS=-Dengine.trimStringValues=false

Please note, that this parameter only works if plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.exec is configured (DPE Configuration :: Ataccama ONE)


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings