Not enough memory to run the plan file

(ananda krishna) #1

I have input file with 662,000 records and 21 columns, each column length not exceeding 50 chars.
I am getting following error when I use group aggregator step, with 10 columns with countDistinct() function.

java.lang.OutOfMemoryError: GC overhead limit exceeded

Is there any workaround to fix the issue?

(Danny Ryan) #2

I have experienced this error a fair number of times.
As far as I understand GC stands for Garbage Collection and the error message means that thr Java Virtual Machine to which your plans run inside of, spend more than a specified amount of time doing memory garbage collection (retrieving memory back from old processes) than actually running the plan.
The GC kicks in when the Java Virtual Machine runs low on available memory.

I fixed this a number of ways.

  1. Increase the memory available to the JVM.
  2. Alternative split the file into more files of smaller record volume.

Hope this helps