re: processing a file -Reply -Reply

Date:   7/15/97  5:33 PM
RE:     re: processing a file -Reply -Reply

>Robert Kerr wrote:

>>...to gain speed in large file processing you need to
>> change the paradyne... Set up a feeder program to load
>> records into a data queue...now write your process
>> programs to pick records off the queue and do their
>> thing...

>Hey, that's pretty slick!  I'm curious though - there is a
>maximum size for data queue objects (I don't know what it is
>off the top of my head) and if you fill up your queue, the
>system pukes (I know 'cause we have a data queue driven
>app that has exactly that problem sometimes).  Doesn't
>your technique run the risk of bombing if your feeder
>program gets too far ahead of of the processing
>programs?  Of course, you can "tune" your performance
>to avoid this by running many instances of the process
>program compared to few (or 1 instance) of the feeder
>program, but I'd be interested to hear your experiences (if
>any) with this problem.
>
>Scott Cornell, Mercy Information Systems

Scott
You could condition the call to QSNDDTAQ program with an error handling
routine.  This routine might call a CL program(or the API) to receive the
program messages,  You could tell if you just filled the data queue.  In
that case,  you could just DLYJOB for a while to let the processing 
programs catch up.   You still have the record in your driver program
buffer,  just resend it to the data queue after awhile.  

This or something like it would handle the filling up of the data queue.

I happen to remember John Sears talking about a similar technique about 10
years ago at COMMON on the SYS/38.  Without the data queues.  A program
with the file opened with just Input would read the records,  This ensured
that they were brought into memory, and then then another program with
the file opened for Update would read the records also.  The second program
would never have a Data Base Page Fault, because the records were always
in main memory!.   The two programs communicated with each other to tell
the first program where the second program was in the file, etc. etc.

Roberts example is pretty slick though. It might have some problems if
you needed a serial update of the file.  One of the programs reading from
the data queue might get held up and one of the other ones might update
the file out of order. 

John Carr
EdgeTech
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
* This is the Midrange System Mailing List!  To submit a new message,   *
* send your mail to "MIDRANGE-L@midrange.com".  To unsubscribe from     *
* this list send email to MAJORDOMO@midrange.com and specify            *
* 'unsubscribe MIDRANGE-L' in the body of your message.  Questions      *
* should be directed to the list owner / operator: david@midrange.com   *
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *