Help get this topic noticed by sharing it on Twitter Twitter, Facebook Facebook, or email.
Steve
sad I’m frustrated
Steve February 01, 2011 12:53

Runtime error in jdbc reader with large table

I am analyzing a db of approx. 4 million records for the first time. I am using MySQL 5.1 connected via JDBC. I have created a profile for a SMALL table with no problems.
For this LARGE table I connect the db with no problems. I create the plan also with no problems. When I run the plan it begins normally and runs for 1 hr 25 min, then stops.
01.02.2011 04:52:01 [INFO] Using following licences:
01.02.2011 04:52:01 [INFO] C:\Ataccama DQ Analyzer 7\runtime\lib\..\license_keys\license.plf
01.02.2011 04:52:01 [INFO] Using DQ Analyzer configuration file C:\Ataccama DQ Analyzer 7\workspace\1SYNC_IM\test1.plan
01.02.2011 04:52:01 [INFO] Using runtime configuration file C:\Users\svazzano\AppData\Local\Temp\runtimeCfg8862999405978560054.tmp
01.02.2011 04:52:04 [INFO] DQ Analyzer engine initialized.
01.02.2011 04:52:04 [INFO] Creating runtime...
01.02.2011 04:52:08 [INFO] Starting runtime...
01.02.2011 04:52:09 [INFO] Running runtime...
01.02.2011 06:17:43 [FATAL] Runtime error in jdbc reader.[STEP test1[Jdbc Reader]]
01.02.2011 06:17:43 [INFO] Stopping runtime...
01.02.2011 06:17:44 [INFO] Finished!

01.02.2011 06:17:44 [WARNING] TASK COMPLETED WITH ERRORS. Elapsed time: 01:25:39

The fetch parameter in the connection is 1000. (not sure this is important).
Thank you for any help you can provide.
1 person has
this question +1
Reply

  • Martin Landa EMPLOYEE
    Martin Landa (Official Rep) February 01, 2011 12:59
    Hello Steve,
    could you, please, send me a description of the error from Run Results tab? You should find it when the job failed next to the Console view below Plan editor.
    Comment
    good answer! good answer! (undo)
  • (some HTML allowed)
    How does this make you feel?     Add Image
    I'm

    e.g. kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned sad, anxious, confused, frustrated

    Cancel
  • Steve
    sad I’m frustrated
    Steve February 01, 2011 15:21
    I increased some of the resource parameters in MySQL and re-ran it. It ran in 53 minutes (vs. 1.5 hrs before resetting resources). The Run Results tab has only the following information:
    Type: Error
    Description: Runtime error in jdbc reader
    Location: test1
    Date: 2011-02-01 09:37:03
    Comment
    good answer! good answer! (undo)
  • (some HTML allowed)
    How does this make you feel?     Add Image
    I'm

    e.g. kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned sad, anxious, confused, frustrated

    Cancel
  • Martin Landa EMPLOYEE
    Martin Landa (Official Rep) February 01, 2011 15:28
    Hello,
    by Run Results I mean the tab at the bottom of DQ Analyzer. It should contain an additional description of an error then the Console tab. On a following screenshot I have it opened at the bottom of DQ Analyzer. After the run is finished with an error, you get a dialog whether to open the tab or not or you can display it by choosing Run Results in a Windows->Show View->Other dialog.

    Comment
    good answer! good answer! (undo)
  • (some HTML allowed)
    How does this make you feel?     Add Image
    I'm

    e.g. kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned sad, anxious, confused, frustrated

    Cancel
  • Steve
    indifferent
    Steve February 01, 2011 16:04
    I see Martin. I clicked on the error and saw the info below (I installed DQ Analyzer with the Apache Darby db.):
    java.sql.SQLException: Incorrect key file for table 'C:\Windows\TEMP\#sql7f4_27_2.MYI'; try to repair it
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3593)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3525)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1986)
    at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1347)
    at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:845)
    at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1362)
    at com.mysql.jdbc.StatementImpl.createResultSetUsingServerFetch(StatementImpl.java:556)
    at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1406)
    at com.ataccama.dqc.model.internal.modelimpl.PoolingDataSource$StatementDecorator.executeQuery(PoolingDataSource.java:773)
    at com.ataccama.dqc.tasks.io.jdbc.read.JdbcReaderInstance.run(JdbcReaderInstance.java:123)
    at com.ataccama.dqc.processor.internal.runner.ComplexStepNode.runNode(ComplexStepNode.java:57)
    at com.ataccama.dqc.processor.internal.runner.Node.run(Node.java:123)
    at java.lang.Thread.run(Unknown Source)
    Comment
    good answer! good answer! (undo)
  • (some HTML allowed)
    How does this make you feel?     Add Image
    I'm

    e.g. kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned sad, anxious, confused, frustrated

    Cancel
  • Steve
    indifferent I’m unconcerned
    Steve February 03, 2011 17:24
    Martin, do you have any other advice on this? The database has 200 columns and 4.3 million records. As I stated above, the profile is created successfully when the same database has a few thousand records. It fails with 4.2 million records. Is it a resource issue?

    For now I have reduced the number of columns from 200 to 20 and the profile is created successfully.

    Many thanks for any help you can provide.
    Comment
    good answer! good answer! (undo)
  • (some HTML allowed)
    How does this make you feel?     Add Image
    I'm

    e.g. kidding, amused, unsure, silly happy, confident, thankful, excited indifferent, undecided, unconcerned sad, anxious, confused, frustrated

    Cancel