This is the second time. On the same customer box. Tomcat gets stuck shutting down.

We've never seen this happen anywhere else. But this is the second time we've seen it happen on this particular customer box.

We have a Tomcat shutdown program. First, it submits a batch job, QShelling the Tomcat server's SHUTDOWN.SH script, and NOT in a way that would allow it to get stuck behind another batch job in a one-at-a-time batch queue. Then it monitors that job, to make sure it launches, and runs to completion. Then it monitors the CATALINA job, to make sure that actually DOES end, and if it fails to do so within a reasonable amount of time, it does an ENDJOB on the CATALINA job, with OPTION(*IMMED).

So it's a belt-and-suspenders approach.

This particular customer has a Tomcat restart program that uses our ENDTOMCAT CL program, and THEN goes through and does an ENDJOB with OPTION(*IMMED) on any CATALINA jobs still in the system, before it starts Tomcat with our STRTOMCAT CL program.

So for this customer, it's a "belt, suspenders, and nail the pants on" approach.

And yet, for the second time, I'm seeing a supposedly terminated Tomcat server, both the CATALINA job and the JVM job, still in the system, hours after multiple attempts to terminate it. The call stack on the JVM job shows:

QP0ZSPWT QSYS _CXX_PEP__Fv
QP0ZPCPN QSYS 132 Qp0zPJNewProcess
QP0ZPCPN QSYS 278 Qp0zNewProcess
QZSHCHLD QSHELL _CXX_PEP__Fv
QZSHCHLD QSHELL 6 main
QZSHSRV1 QSHELL 148 QzshChildMain
QZSHSRV1 QSHELL 172 evalcmdpart2__FP4nodeiP7bac
QZSHSRV1 QSHELL 1 shellexec__FPcPPcT2
QP0ZEXEC QSYS 47 Qp0zExecve
QP0ZEXEC QSYS 6 run__14Qp0zExecutableFv
QZSHSH QSHELL _CXX_PEP__Fv
QZSHSH QSHELL 16 main
QZSHSRV1 QSHELL 108 QzshMain
QZSHSRV1 QSHELL 47 cmdloop__Fi
QZSHSRV1 QSHELL 65 evaltree__FP4nodei
QZSHSRV1 QSHELL 117 evaltree__FP4nodei
QZSHSRV1 QSHELL 248 evalcommand__FP4nodeiP7back
QZSHSRV1 QSHELL 124 evalcmdpart2__FP4nodeiP7bac
QZSHSRV1 QSHELL 15 execcmd__FiPPc
QZSHSRV1 QSHELL 1 shellexec__FPcPPcT2
QP0ZEXEC QSYS 47 Qp0zExecve
QP0ZEXEC QSYS 6 run__14Qp0zExecutableFv
QJVATOOLS QJAVA _CXX_PEP__Fv
QJVATOOLS QJAVA 5 main
QJVAQSHCMD QJAVA 716 QjvaQshCmd
QJVAJVM QJAVA 215 RunJava
QJVAJVM QJAVA 17 RunJava_Cleanup__FP11RunJav
QXJ9VM QJVM7064 10 DestroyJavaVM__FP7JavaVM_
QXJ9VM QJVM7064 5 iCallPase__14Qxj9PaseAccess
QP2USER QSYS 1 Qp2CallPase
QP2USER2 QSYS 9 __Qp2CallPase2
QP2USER2 QSYS 4 runpase_common__FiPvT2
unix <callbackrtn>
libi5osenv.so 716 00000064 J9_i5OS_DestroyJavaVM
libjvm.so 00000048 DestroyJavaVM
libj9vm26.so 000002B4 DestroyJavaVM
libj9prt26.so 00000388 j9sig_protect
libj9vm26.so 00000084 protectedDestroyJavaVM
libj9thr26.so 00000034 j9thread_monitor_wait
libj9thr26.so 00000934 monitor_wait_original
libpthreads.a(shr_xpg > 000001A0 pthread_cond_wait
libpthreads.a(shr_xpg > 000000CC _cond_wait
libpthreads.a(shr_xpg > 000004E0 _cond_wait_local
libpthreads.a(shr_xpg > 00000348 _event_wait
libpthreads.a(shr_xpg > 00000584 _event_sleep
unix 00000008 <syscall64>:thread_tsleep

And it yields immediately to an ENDJOB OPTION(*IMMED) issued via Option 4 on WRKACTJOB.

I can't make head or tail of it. Any ideas?

--
JHHL