shared storage across many jobs

Folks, I have a design question to throw out.  

 

We have a program that runs in many (thousands) of jobs and presently we use
a user space to store certain job information.  So each job has its own slot
in the user space.  Slot is sizeable (about 3000 bytes) so we can fit about
5500 slots in the 16MB user space.  User space is handy since we access it
by pointer (excellent performance), is a persistent object, doesn't get
paged out when all these jobs are active at the same time and is easily
accessible by other jobs that consume information stored in it.

 

However, number of jobs that use it is growing beyond the max we can fit in
it so we're looking for an alternative solution that would give us same or
similar benefits without many more hassles.

Some ideas folks here are throwing around is shared memory, teraspace heap
and like.  We don't want to use files due to performance overhead.  

It's absolutely critical that performance stays the same (i.e. access by
pointer to memory resident storage shared by thousands of jobs on the
system).

 

There are probably many ways to do this, but I figured someone may have done
it already and is willing to share what are some of the better alternatives.

Thanks.

 

Elvis