Solved

Not enough memory to run the plan file.

  • 6 December 2022
  • 9 replies
  • 158 views
O
Userlevel 2
Badge +5

I want to load around 832,032 records from Excel file to DB table using Plan. Plan is not executing.

It giving below error message :-

1)Not enough memory to run the plan file.

 

  1. java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.io.IOUtils.byteArray(IOUtils.java:338)
        at org.apache.commons.io.output.AbstractByteArrayOutputStream.needNewBuffer(AbstractByteArrayOutputStream.java:104)
        at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.<init>(UnsynchronizedByteArrayOutputStream.java:51)
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:181)
        at org.apache.poi.util.IOUtils.toByteArray(IOUtils.java:153)
        at org.apache.poi.openxml4j.util.ZipArchiveFakeEntry.<init>(ZipArchiveFakeEntry.java:67)
        at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.<init>(ZipInputStreamZipEntrySource.java:98)
        at org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:132)
        at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:312)
        at com.ataccama.dqc.io.msoffice.OpenXmlFile.open(OpenXmlFile.java:71)
        at com.ataccama.dqc.tasks.msoffice.excel.read.ExcelFileReaderInstance.runEventDriven(ExcelFileReaderInstance.java:196)
        at com.ataccama.dqc.tasks.msoffice.excel.read.ExcelFileReaderInstance.runStream(ExcelFileReaderInstance.java:119)
        at com.ataccama.dqc.tasks.msoffice.excel.read.ExcelFileReaderInstance.run(ExcelFileReaderInstance.java:97)
        at com.ataccama.dqc.processor.internal.runner.ComplexStepNode.runNode(ComplexStepNode.java:69)
        at com.ataccama.dqc.processor.internal.runner.RunnableNode.run(RunnableNode.java:28)
        at com.ataccama.dqc.commons.threads.AsyncExecutor$RunningTask.run(AsyncExecutor.java:135)
        at java.base/java.lang.Thread.run(Thread.java:834)

Anyone faced this situation? How to resolve this issue? Thanks in advance.

icon

Best answer by AKislyakov 12 March 2024, 04:15

View original

9 replies

A
Rank
Userlevel 4
Badge +4

Hi Ojaswini,

There is the Knowledge Base article on how to deal with the bigger plans. In your case I think the section Allocating Additional Memory for a Particular Plan Execution suits best.

O
Userlevel 2
Badge +5

Thank you very much.

O
Userlevel 2
Badge +5

Yes, that helped to run the plan. Basically, I had to change memory limit at all specified location in the given link. Thank you.

greglvaughan
Badge

HI,

Those links are redirecting to the new support root page.  Are the knowledge base articles still available?

 

thanks,

 

Greg

 

O
Userlevel 2
Badge +5

Hi,

 

You are right. They're not available now. Or maybe I don't the new links for those. Given links areonly redirecting to root page.

Thanks,

Ojaswini

A
Rank
Userlevel 4
Badge +4

Hi everyone @Ojaswini @greglvaughan , 

Because of Support portal re-design the KB articles are indeed currently unavailable. Here is how to allocate more memory to a single plan:

 

  1. Click the drop down arrow beside run and select Run Configuration…
  2. From within the Runtimes tab, and in VM arguments, you can input the same variables for the VM that are added in the environment variable JAVA_OPTS.
  3. Set the Xmx variable, which configures to the maximum heap size reserved by the Java Virtual Machine. Follow the notation for passing this parameter, e.g., -Xmx2g stands for reserving 2 GB of memory as the maximum allocated for the plan. Notice that you can see a complete list of all the plans being opened and executed by DQC in the left pane, with their particular VM argument settings.

     

 

greglvaughan
Badge

HI,

 

Thanks so much! Is there a timeline for , or a way to access the old knowledge base articles.

 

Greg

Cansu
Rank
Userlevel 6
Badge +7

Hi @greglvaughan, you can find the versions from 13.9 and lower in PDF formats here. Our team is working to get them on the new portal at the moment, you can learn more about it here.

Check out Our Community Recommended Journey to get started on the community 🙋‍♀️
greglvaughan
Badge

Thanks @Cansu   I will take a look!  

All the best!

gV

Reply


Community Terms of Use & ConditionsCookie settings