Socket Server getting 'hung' & how to diagnose

I've been using Scott Klement's socket server program to handle many
sockets at once using select() in non-blocking mode. The program has been
stable for over two years and stays up on one established connection for
up to 30 thirty consecutive days (the only time it goes down is during our
monthly savsys). I have it set to allow a maxclients value of 20, but for
this client it never establishes more than one connection.

Recently a second instance of the socket server program (note there is
only one *pgm) went into production to listen to another client (on a
different port) over a different VPN tunnel.

For some reason this second instance may do one of two things and I'm not
sure why. Over the course of 24 hours I've seen up to six connections
established from the one client. My first question is why would
connections other than the first one be initiated, when the client doesn't
show any disconnect at their end?

The second problem is that in viewing netstat *cnn for the client, after a
random period of valid activity, typically in hours, I've monitored and
seen where the Idle Time for the client is being reset to zero after every
90 seconds. Something is happening causing the idle time to reset itself
every 90 seconds. When I look at the call stack of the socket server it is
at the statement in my program that performs the "eval rc = select(max+1:
%addr(readset):%addr(writeset): %addr(excpset): to)", and the job status
is SELW. If I look in the joblog I can see when I last received valid data
(which was received more than 90 seconds ago). So it appears that
something is trying to get through but nothing's happening. At this point
the only option is to force the program to end and restart it. The client
then picks up and starts sending data as it should and everything is good.

We use the same VPN appliance to manage both tunnels and both
configurations are 'identical' at our end. We did determine that the VPN
tunnel remains active during either of the two situations.

When this first started happening I changed the server program to set
maxclients from 2 to 20 so that I can 'reliably' ensure an available
connection that allows me to run scheduled jobs daily to issue a shutdown
(to 'reset the socket'), and then restart the socket server. This seems to
handle the first problem okay since we continue to receive data 'all day
long' sometimes over multiple established connections (from the same
client), but the second problem prevents new connections from being
established (from the client), which in effect stops the server from
receiving anything until it is shutdown and restarted at 6:00 am.
Unfortunately this happened this morning before 8:00 and we can't wait 22
hours for the program to be restarted (unattended).

I am stumped. How would I go about diagnosing the server socket just to be
sure the error isn't happening with my socket server? (I did add code to
notify me in the event that the data received isn't what is expected,
thinking maybe a virus or worm was trying to wiggle it's way through the
port, but so far I've not received any notification.) I'm 99% sure the
problem lies with the client since they've told me that they have other
facilities that experience similar connectivity issues. The unfortunate
thing is this client outsources their network operations to Perot Systems,
and Perot support leaves a lot to be desired.


Regards, Jerry

Gerald Kern - MIS Project Leader
Lotus Notes/Domino Administrator
IBM Certified RPG IV Developer
The Toledo Clinic, Inc.
4235 Secor Road
Toledo, OH 43623-4299
Phone 419-479-5535
gkern@xxxxxxxxxxxxxxxx


This e-mail message, including any attachments, is for the sole use of the
intended recipient(s) and may contain confidential and privileged
information. Any unauthorized use, disclosure or distribution is
prohibited. If you are not the intended recipient, please inform the
sender by reply e-mail and destroy this and all copies of this message.