Memory Issue in running DQC process


(sivaraman gangadharan) #1

Hello ,
I am running a dqc plan with batch script and plan fails with insufficient memory error . I tried increasing from 4 to 10 GB memory allocated through command in a script and it still fails . PROD server has only 16 G memory and would like to know if there is any better ways to manage the memory allocation . I have also attached error log .hs_err_pid10044.log (27.8 KB)

Batch script
@echo off

set JAVA_OPTS=-Xmx10g -Djava.io.tmpdir=E:/Temp
set DQC_HOME=C:/Program Files/DQC/runtime
set projectpath=…/Files
set logpath=…Files/logs/
set logfile=name.log
set statefile=name.xml
echo %DATE% %TIME% > %logpath%%logfile%

call “%DQC_HOME%\bin\run_java.bat” com.ataccama.adt.runtime.EwfProcessor -runtimeConfig=%projectpath%\etc\dqc.runtimeConfig -state %logpath%%statefile% -log=true %projectpath%\workflows\wf.ewf >> %logpath%%logfile%


(Katrin Popova) #2

Hello Siva,

In this case, I would advice you to test the plan via the IDE, to see how much allocated memory the plan would require. You can set it up in Run Configuration, in Runtimes tab in the VM arguments field. As explained in this documentation.

Regards,
Katrin


#3

Hi Sivaraman,

I’m not a java developer so I might be very wrong.

Looks like it’s not the plan using all allocated heap space, but rather your system couldn’t allocate enough memory for requested heap size.
So I would try to decrease the Xmx to some reasonable amount that you believe is available on your system (like 1g) and try to run the plan/workflow.


(sivaraman gangadharan) #4

Thanks for the suggestion . I am able to run the plan successfully by splitting the process in to 2 different plans . I believe running parallel tasks in same plan had caused the issue with memory . I would like to know in detail about enabling parallelism for a specific plans in a workflow . right now its configured as default 1 as value .


(Evgeni Enchev) #5

Hello Siva,

Could you please give more information on the tasks inside the failing plan? You mention a batch script, are there any CPU or memory-intensive operations performed?

Thank you,
Evgeni