JSON Parser: Creating Shadow Columns generates java.lang.ArrayIndexOutOfBoundsException Error

(Lennard Ramirez) #1

Creating Shadow Columns in JSON Parser generates java.lang.ArrayIndexOutOfBoundsException Error when running the plan. Note that this works fine when using JSON Reader. Anyone else having this issue?

Here’s a screen shot of the JSON Parser configuration:

Here are the message details on the error:
Failed while processing JSON input: 1

Detail Information:

java.lang.ArrayIndexOutOfBoundsException: 1
	at com.ataccama.dqc.expressions.model.elements.ColumnValue$ColumnValueInstance.evaluate(ColumnValue.java:142)
	at com.ataccama.dqc.model.expressions.DqcExpression$DqcExpressionInstance.evaluate(DqcExpression.java:49)
	at com.ataccama.dqc.tasks.common.column.ColumnAssignRuntime.evaluate(ColumnAssignRuntime.java:143)
	at com.ataccama.dqc.tasks.common.column.ColumnAssignRuntime.execute(ColumnAssignRuntime.java:110)
	at com.ataccama.dqc.tasks.io.json.reader.JsonStreamConfig$JsonStream.evalShadowRecords(JsonStreamConfig.java:477)
	at com.ataccama.dqc.tasks.io.json.reader.JsonStreamConfig$JsonStream.process(JsonStreamConfig.java:423)
	at com.ataccama.dqc.tasks.io.json.reader.JsonStreamConfig$JsonStream.process(JsonStreamConfig.java:458)
	at com.ataccama.dqc.tasks.io.json.reader.AbstractJsonReaderRuntime.parseFile(AbstractJsonReaderRuntime.java:98)
	at com.ataccama.dqc.tasks.io.json.reader.AbstractJsonReaderRuntime.process(AbstractJsonReaderRuntime.java:41)
	at com.ataccama.dqc.tasks.io.json.parser.JsonParser$JsonParserInstance.run(JsonParser.java:216)
	at com.ataccama.dqc.processor.internal.runner.ComplexStepNode.runNode(ComplexStepNode.java:69)
	at com.ataccama.dqc.processor.internal.runner.RunnableNode.run(RunnableNode.java:28)
	at com.ataccama.dqc.commons.threads.AsyncExecutor$RunningTask.run(AsyncExecutor.java:131)
	at java.lang.Thread.run(Thread.java:745)

(Aleksandr Lisichkin) #2

Hi Ramirez!
This bug was discovered by me a month ago. Responsive developers from the ataccama confirmed this bug and now the bug is in fixing stage. Shadow columns throws excaptions in 11.1 version. So it is unable to put data from response or put data from input source in the shadow columns.

I see that you have to fill the shadow column from response data. You can add the Column Assigner and fill your column within any column data from JSON Response output.

In my case I need to put the packetID column from input data to response output data of JSON component (In a special table I create packages for further sending for handling the requests of each packet, so I need to have the packageID in response message), because REST don’t return the same data from input source like packageID.
Personally, I was able to solve this problem in the following way:

  1. I have shadow column difinition within empty expression and filled data type. In my case it was packageID.
  2. In General Tab I was fill the Parent ID Column (in.packageID) and fill the Record Discriptor using the shadow column name defined as packageID.
  3. After JSON Component add the Column Assigner component to JSON Response output and apply the expression for this column

trim(substr(packageID,0,iif(indexOF(packageID,’:’) IS NULL,length(packageID),indexOF(packageID,’:’))))

I hope that my solution will help anyone who have encountered this problem

(Lennard Ramirez) #3

Thanks for the reply Aleksandr. That worked. Not sure exactly why it’s working but it works. I say that because the source data element in the JSON that ties the streams together is called “message_control_id”. All the definitions make references to the new shadow column called “hdrMsgCtrlId” and there’s no reference at all to “message_control_id” and yet the column assigner is populating correctly. See screen shot below:

Thanks a bunch for the tip. I hope the fix for this comes out soon since this is a band-aid.

(Aleksandr Lisichkin) #4

I am very glad that so soon helped to solve your problem. And I’m sorry that I first call you Ramirez. As I sure yuor name is Lennard :slight_smile: