RE: Speed in Reading

Rick,

The app was essentially complete when subsequently the performance of reading the large input file from start to finish became a more serious issue.
The requirement is to have multiple jobs all of which are simultaneously processing records on the input file.
I figure a data queue solution would require the creation of a queue, a change to the app to wait on the queue and write a program submitted to read every input record anyway just to send it to the data queue.
I opted just to change the app to accept a start and end position on the input file. Then I can just manually submit as many jobs as I like referencing different chunks of the file. If needed, submitting these threads could be automated. It also made rerunning any portion of the file easier if the job crashed.

Peter



-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Rick.Chevalier@xxxxxxxxxxxxxxx
Sent: Tuesday, 8 June 2010 12:28 a.m.
To: rpg400-l@xxxxxxxxxxxx
Subject: RE: Speed in Reading

Peter,

Not to hijack the thread, but I'm curious if you considered a data queue for this and if so why did you decide against it?

Rick

-----Original Message-----
From: rpg400-l-bounces@xxxxxxxxxxxx [mailto:rpg400-l-bounces@xxxxxxxxxxxx] On Behalf Of Peter Connell
Sent: Friday, June 04, 2010 4:54 PM
To: 'RPG programming on the IBM i / System i'
Subject: RE: Speed in Reading

Kurt,
While there are undoubtedly horses for courses I have been tasked with data mining tasks over the last year where there are 10 million or more records driving the process each of which itself may generate scores of other reads. I've found that submitting up to simultaneous 10 jobs, each of which accepts parameters as to which portion of the input file drives it, has yielded exceptional performance. This does drive CPU right up but permits huge volumes to be processed overnight.

Peter

Privileged and Confidential. This e-mail, and any attachments there to, is intended only for use by the addressee(s) named herein and may contain privileged or confidential information. If you have received this e-mail in error, please notify me immediately by a return e-mail and delete this e-mail. You are hereby notified that any dissemination, distribution or copying of this e-mail and/or any attachments thereto, is strictly prohibited.