Re: Speed in Reading

So, did you first get the total number of records and then split the rrn
10 ways?  Or did you just alot a certain number of records per instance
and when you had covered the last rrn you were done?

Also,  why not drive the CPU hard?  If there is any part of the system
that can handle a consistently high load it should be the cpu.  Rev that
baby up.  Idle cycles are lost cycles... can't ever get them back so you
might as well use them.  It would be interesting to see how a nightly
batched MRP run would work if this approach was taken.  Wonder if we could
take it from 2 hours to down to 20 minutes....


Thanks
Bryce Martin
Programmer/Analyst I
570-546-4777



"Lennon_s_j@xxxxxxxxxxx" <lennon_s_j@xxxxxxxxxxx>
Sent by: rpg400-l-bounces@xxxxxxxxxxxx
06/04/2010 06:29 PM
Please respond to
RPG programming on the IBM i / System i <rpg400-l@xxxxxxxxxxxx>


To
rpg400-l@xxxxxxxxxxxx
cc

Subject
Re: Speed in Reading






Yes, I've done that too with good results, but in my case I was reading
a significantly large transaction file and updating many summary files.
 Splitting up the transaction file largely by RRN and processing
between 7 and 10 in parallel significantly reduced the elapsed time, but
did drive the CPU hard.

Sam

On 6/4/2010 5:54 PM, Peter Connell wrote:

Kurt,
While there are undoubtedly horses for courses I have been tasked with

data mining tasks over the last year where there are 10 million or more
records driving the process each of which itself may generate scores of
other reads. I've found that submitting up to simultaneous 10 jobs, each
of which accepts parameters as to which portion of the input file drives
it, has yielded exceptional performance. This does drive CPU right up but
permits huge volumes to be processed overnight.

Peter

--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing
list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.



--- This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us and destroy this message immediately. ---
--
This is the RPG programming on the IBM i / System i (RPG400-L) mailing list
To post a message email: RPG400-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/rpg400-l
or email: RPG400-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at http://archive.midrange.com/rpg400-l.