Re: "Parallel Processing" Tool -- was: [WDSCI-L] Namespaces -- was: Debug slows program by a factor of 30?

On 12/7/2017 10:39 AM, dlclark@xxxxxxxxxxxxxxxx
wrote:

"WDSCI-L" <wdsci-l-bounces@xxxxxxxxxxxx> wrote on 12/07/2017 10:00:51 AM:

http://www.easy400.net/syncjob/html/page1.htm

That's an interesting tool. Presuming that the parallel jobs are
submitted through the tool, is there a way to contact the author to ask
about an additional feature?

Probably the best way to contact Giovanni is via the Yahoo Group CGIDEV2.

So, what do you guys think? Would you rather a tool control the
number of active parallel jobs in this type of situation? ...or, would
you rather have a separate job queue that controls it? Any other ideas?

Generally speaking, I try to keep my work management at the system
level, and not at the program level. But that's not always dynamic /
granular enough.

Some years ago, when I needed more dynamic control than a flat number of
job queues / maxjobs, I had a system that submitted 'work' into a data
queue. The 'processor' would read the dtaq, and if enough time had
passed between the 'submit' and 'read', it would SBMJOB another copy of
itself if the system wasn't already too busy. The secret sauce is that
each of the submitted jobs has the ability to throttle the process.

There's a possibility of ping-ponging between start/stop; the way around
that is to change the 'busy' threshold when crossing it. If the system
goes from idle to busy and spawns a new reader, move the idle threshold
down, and vice versa.

You could recycle this idea to use a job queue easily enough. Submit
all 600 jobs to a single job queue that begins at maxjobs(1) or 5 or
whatever fits. Each of the jobs (when it begins to execute) makes a
call to the throttle program/function. The throttle checks the system's
busy state - if idle, increase maxjobs(); if busy, decrease it.