Re: Design for data queue-driven subsystem

["Buck" <buck.calabro@xxxxxxxxxxxx>]

I cannot post specifics because it's part of our product but I can give you
some general ideas.

> The trickiest part was identifying the number
> of active jobs receiving queue entries.

One way to do this is to time stamp the message when you put it on the
queue.  When you receive it, see how long it sat in the queue: if too long
for your application, fire up another listener.

When to tear a listener down is another story.  You pretty much need a
separate watchdog process to keep track of all the listeners running, and
their workload.  There are plenty of ways to do that, from polling each
listener ('How long since the last work?') to having the listeners send
status back to the dispatcher ('I've been quiet for 5 minutes now.')  The
main issue here is that you do NOT want to tear a listener down, only to
have the workload start another up in a few milliseconds.  So you need to
account for hysteresis in the system (if a listener has been torn down in
the past 3 minutes, don't start another one.  If a listener has been started
in the past 3 minutes, don't tear one down.)

You can store the variables in a control file, data areas or even a user
space.  Also, operations typically want to be able to see what's going on in
the internals of the dispatcher, so think about a file/space to store
current status and perhaps the past hour (or 20 transitions of start/stop)
somewhere so they can easily look at what is happening.  It'll help a lot
during testing too.

I hope some of these ideas are useful...
  --buck