Skip to main content

I want to load around 832,032 records from Excel file to DB table using Plan. Plan is not executing.

It giving below error message :-

1)Not enough memory to run the plan file.

 

  1. java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.io.IOUtils.byteArray(IOUtils.java:338)
        at org.apache.commons.io.output.AbstractByteArrayOutputStream.needNewBuffer(AbstractByteArrayOutputStream.java:104)
        at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.<init>(UnsynchronizedByteArrayOutputStream.java:51)
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:181)
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:153)
        at org.apache.poi.openxml4j.util.ZipArchiveFakeEntry.<init>(ZipArchiveFakeEntry.java:67)
        at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.<init>(ZipInputStreamZipEntrySource.java:98)
        at org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:132)
        at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:312)
        at com.ataccama.dqc.io.msoffice.OpenXmlFile.open(OpenXmlFile.java:71)
        at com.ataccama.dqc.tasks.msoffice.excel.read.ExcelFileReaderInstance.runEventDriven(ExcelFileReaderInstance.java:196)
        at com.ataccama.dqc.tasks.msoffice.excel.read.ExcelFileReaderInstance.runStream(ExcelFileReaderInstance.java:119)
        at com.ataccama.dqc.tasks.msoffice.excel.read.ExcelFileReaderInstance.run(ExcelFileReaderInstance.java:97)
        at com.ataccama.dqc.processor.internal.runner.ComplexStepNode.runNode(ComplexStepNode.java:69)
        at com.ataccama.dqc.processor.internal.runner.RunnableNode.run(RunnableNode.java:28)
        at com.ataccama.dqc.commons.threads.AsyncExecutor$RunningTask.run(AsyncExecutor.java:135)
        at java.base/java.lang.Thread.run(Thread.java:834)

Anyone faced this situation? How to resolve this issue? Thanks in advance.

Hi Ojaswini,

There is the Knowledge Base article on how to deal with the bigger plans. In your case I think the section Allocating Additional Memory for a Particular Plan Execution suits best.

Thank you very much.

Yes, that helped to run the plan. Basically, I had to change memory limit at all specified location in the given link. Thank you.

HI,

Those links are redirecting to the new support root page.  Are the knowledge base articles still available?

 

thanks,

 

Greg

 

Hi,

 

You are right. They're not available now. Or maybe I don't the new links for those. Given links areonly redirecting to root page.

Thanks,

Ojaswini

Hi everyone @Ojaswini @greglvaughan , 

Because of Support portal re-design the KB articles are indeed currently unavailable. Here is how to allocate more memory to a single plan:

 

  1. Click the drop down arrow beside run and select Run Configuration…
  2. From within the Runtimes tab, and in VM arguments, you can input the same variables for the VM that are added in the environment variable JAVA_OPTS.
  3. Set the Xmx variable, which configures to the maximum heap size reserved by the Java Virtual Machine. Follow the notation for passing this parameter, e.g., -Xmx2g stands for reserving 2 GB of memory as the maximum allocated for the plan. Notice that you can see a complete list of all the plans being opened and executed by DQC in the left pane, with their particular VM argument settings.

     

 

HI,

 

Thanks so much! Is there a timeline for , or a way to access the old knowledge base articles.

 

Greg

Hi @greglvaughan, you can find the versions from 13.9 and lower in PDF formats here. Our team is working to get them on the new portal at the moment, you can learn more about it here.

Thanks @Cansu   I will take a look!  

All the best!

gV

Reply


Community Terms of Use & ConditionsCookie settings