Need help diagnosing Java application problem

Hi All -

I need an expert help me diagnose a Java application problem. My expertise
is more in the Windows and Unix environments but this application also
needs to run on the iSeries and I have exhausted the extent of my limited
iSeries troubleshooting knowledge.

I have a user running my Java application on a V5R4 platform.
The application runs through a batch job and is invoked via the following
CL commands:

CHGCURDIR '/appPath'

ADDENVVAR ENVVAR(JAVA_HOME) VALUE(*NULL) +
LEVEL(*JOB) REPLACE(*YES)


JAVA CLASS('classname') +
PARM('-s' "service") +
PROP((java.ext.dirs '/QIBM/ProdData+
/OS400/jt400/lib:/appPath/lib:+
(org.apache.commons.logging.Log +
'LoggerClassname') +
(java.version '1.5')) +
OPTIMIZE(40) CHKPATH(*IGNORE)

/* If this command has been invoked and the service is not */
/* currently running, ignore that error and exit gracefully */

MonMsg MsgID(JVA0122)


This application runs fine for several days and then just quits unexpectedly. The job logs show that
my application and the BCI QJVACMDSRV job have encountered a normal end of job scenario
(i.e., job ending codes shown in the associated job logs are 0) however I have added all
kinds of diagnostic logging to my application to store stack traces, write messages to the
QSYSOPR log on end and write other locator information in the application debug logs
when the application ends normally, however none of this is ever logged.

We have never had any other iSeries user report this problem and I cannot reproduce it
in my own V5R4 test environment. The closest I can come is to manually stop the
QJVACMDSRV process but in that scenario, a job ending code of 50 is recorded.
My suspicion is that the Java Virtual Machine is somehow shutting down, thinking it is a
normal exit and is bringing my application down with it so none of my normal system end
or diagnostic logging can be recorded at the time of the shutdown given the proverbial rug
is being pulled out from under it. This is my suspicion but I cannot prove it.

As background information, my application starts as a service and connects to the
RMI port (the default is 1099). On a normal shutdown another CL command
runs and signals the shutdown with a '-s', "service,stop" command via this RMI port.
I have already suggested that the user configure the application to attach to another
non-standard RMI port in case some other application thinks it is supposed to be using
the standard RMI port as well - but that did not cure the problem. I have also asked
that they verify that the shutdown job isn't somehow "stuck" and that there are no
other "rogue" processes running.

I have asked for screen dumps of the QSYSOPR log and have received them twice now.
In both cases, the message "Service Agent is analyzing your system product activity log entries."
is logged around the time of the unexpected shutdowns. I'm not sure if this is related to the
problem or not. The customer has stated that it only runs on Tuesdays however the last two
unexpected shutdowns were on a Thursday and a Sunday. Could this Service Agent be shutting us down?

Has anyone ever experienced any application problems something similar to this?
I welcome any suggestions that anybody has about what other troubleshooting steps I can take next.