ONE Expression - Trim

Hi,

There are certain records that contain “white spaces” at the beginning and/or at the end. Ex: “ 100137069 “, “ 100017129 “, etc. We don't allow the use of white spaces and want to build a DQ rule to check the invalid records. Hence the advanced expression: length(String_ATTRIBUTE_1) <> length(trim(String_ATTRIBUTE_1)). When I test the DQ rule it works as intended. I copy the above results and paste them in the test environment and it shows me the record is invalid. When I remove the white spaces it will considered valid. However, when I run the Monitoring Project based on the same records and DQ rule it gives me 100% validity. How is this possible? And maybe more importantly, how can I create a rule that will show the invalid records shown earlier?

Page 1 / 1

Hi @JTH,

By default, Ataccama trims leading and trailing spaces for all columns. If you need to preserve them, you need to set following parameter:

-Dengine.trimStringValues=false (doc: JVM Configuration :: Ataccama ONE)

So in this case Ataccama increases the data quality before checking the data quality.

Hello @JTH I’m closing this thread, please feel free to follow up here with any of your questions or create a new post ‍♀️

I still have some questions. I'm the person on @JTH 's team who implements the solution. And I don't understand where this -Dengine.trimStringValues property should be implemented. I read that it should be set in the Runtime Server. We have an on premise implementation of Ataccama One. And I don't think we have that running?

Normally when data is read I see it go passed the dpe, dpm or mmm-backend components. But they don't seem to have an engine related property.

@AKislyakov @Cansu can any of you please follow up on @Marcel-Jan's comment so we can proceed on this issue further. We want to see what we can do with this solution

Hi @Marcel-Jan, @JTH

For Ataccama ONE you need to set this property on DPE using plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS
Like following:

plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.env.JAVA_OPTS=-Dengine.trimStringValues=false

Please note, that this parameter only works if plugin.executor-launch-model.ataccama.one.launch-type-properties.LOCAL.exec is configured (DPE Configuration :: Ataccama ONE)

Reply

Sign up

Login to the Ataccama Community

Scanning file for viruses.

This file cannot be downloaded