Loading Profile...
Twitter,
Facebook, or email.
FYI... I've been having some difficulty getting the drill-through to work with some long fixed-length flat file records and have tried using both the built-in and external (PostgreSQL) databases. Tried the recommended run-time solution but still kept getting out of memory errors.
Problem... I've fallen back to not drilling down with these large files but I have noticed a minor problem with the plan files during this process. Once I de-select the drill-down option (and all of the input fields are shown as cleared) and then save the plan I get a yellow exclamation point "message". Further research shows that DQA does not remove the "drillThroughLimit="xxx"" statement from the profilingInput line in the .plan file. Once I remove that statement manually from the .plan file using a text editor everything is OK.
I’m
providing feedback
"Clean-up" of .plan files related to drill-down option
Official
Response
-
Hello again,
you definitely have big files and gave it a nice try with the -Dsorter.inMemory=1000 option.
What would I try is to use more memory for the run if you didn't try it already and to lower the number of records in memory at once in case you have wide records:
-Dprofiling.inMemoryTotal=100000 -Xmx1024M
I am going to generate a data similar to yours, so we have an exact advice for such a case.
Regards
Martin
EMPLOYEE
-
Hello Kirby,
thanks for your feedback, we'll try to erase the value when the checkbox is unchecked.
But you've come up with a nice solution, even the warning is harmless. What can you also do is to switch the Profiling step dialog to "Normal layout" mode by clicking the icon in the upper right corner. The Profiling step will be shown as an another step. There you can find the Drill-through limit under the node named by your input and delete its contents.
Regarding the memory errors, could you please describe your file? How many records, how many columns and how many characters in a field are there?Comment -
-
Hi Martin,
I'm working with several fixed-length column files that are large and I'm using the default profiling options for each column. Even when not using the drill-down the first two files are the ones giving me the trouble:
#1: 501 columns, 3237 total length, over 1.48 million records. This one hits the out of memory error at 192,775 records in pass 1/2.
#2: 630 columns, 4064 total length, over 1.4 million records. This one hits the out of memory error at 301,500 records in pass 1/2, using the default profiling options for all columns. I hit the error whether or not I use the "-Dsorter.inMemory=1000" argument.
#3: 492 columns, 3064 total length, 144K records. No problems with this file.
#4: 90 columns, 820 total length, 340K records. No problems with this file.
Thanks,
... Kirby -
-
Hello again,
you definitely have big files and gave it a nice try with the -Dsorter.inMemory=1000 option.
What would I try is to use more memory for the run if you didn't try it already and to lower the number of records in memory at once in case you have wide records:
-Dprofiling.inMemoryTotal=100000 -Xmx1024M
I am going to generate a data similar to yours, so we have an exact advice for such a case.
Regards
Martin -
-
OK. Will try that. Before I try, more info on attempting #1 again:
Changed plan to create two profiles with 1 condition based on a column have a "Y" or "N" value determining which profile to create. I also changed the default profiling options to "false" for Domain and Group Size for all columns.
With these settings and no run-time argument, I received the out of memory error at 153,623 records in pass 1/2.
Will try same plan now with the run-time argument you just provided.
... Kirby -
-
Oops... sorry. The previous post refers to #2, not #1. I'm trying #2 again with the new run-time argument.
-
-
Success with #2! Using that run-time argument it ran without the memory error. Total of 1,438,831 records read, two profiles created, one with 530,257 records and the other with 908,574 records.
Ran much faster also.
Will try same with #1 now but will try with default profiling options for all columns... -
-
Success with #1also using the run-time argument. Total of 1,536,801 records read, two profiles created: 734,320 and 802,481 records respectively.
-
-
Hello,
good to hear it worked for you.
We'll try to do something about the parameters, I understand they are more or less hidden for most of DQ Analyzer users, but they are usually not needed. -
-
OK. Thanks Martin. DQA is a great product and I sincerely appreciate the timely assistance.
-
