Re: This is the second time. On the same customer box. Tomcat gets stuck shutting down.

You have a thread not exiting because (me theenk) the monitor thread is
waiting on the original thread to exit. Possibly a bug in J9 on PASE?

On Tue, Jul 10, 2018 at 9:50 AM, James H. H. Lampert <
jamesl@xxxxxxxxxxxxxxxxx> wrote:

We've never seen this happen anywhere else. But this is the second time
we've seen it happen on this particular customer box.

We have a Tomcat shutdown program. First, it submits a batch job,
QShelling the Tomcat server's SHUTDOWN.SH script, and NOT in a way that
would allow it to get stuck behind another batch job in a one-at-a-time
batch queue. Then it monitors that job, to make sure it launches, and runs
to completion. Then it monitors the CATALINA job, to make sure that
actually DOES end, and if it fails to do so within a reasonable amount of
time, it does an ENDJOB on the CATALINA job, with OPTION(*IMMED).

So it's a belt-and-suspenders approach.

This particular customer has a Tomcat restart program that uses our
ENDTOMCAT CL program, and THEN goes through and does an ENDJOB with
OPTION(*IMMED) on any CATALINA jobs still in the system, before it starts
Tomcat with our STRTOMCAT CL program.

So for this customer, it's a "belt, suspenders, and nail the pants on"
approach.

And yet, for the second time, I'm seeing a supposedly terminated Tomcat
server, both the CATALINA job and the JVM job, still in the system, hours
after multiple attempts to terminate it. The call stack on the JVM job
shows:

QP0ZSPWT QSYS _CXX_PEP__Fv
QP0ZPCPN QSYS 132 Qp0zPJNewProcess
QP0ZPCPN QSYS 278 Qp0zNewProcess
QZSHCHLD QSHELL _CXX_PEP__Fv
QZSHCHLD QSHELL 6 main
QZSHSRV1 QSHELL 148 QzshChildMain
QZSHSRV1 QSHELL 172 evalcmdpart2__FP4nodeiP7bac
QZSHSRV1 QSHELL 1 shellexec__FPcPPcT2
QP0ZEXEC QSYS 47 Qp0zExecve
QP0ZEXEC QSYS 6 run__14Qp0zExecutableFv
QZSHSH QSHELL _CXX_PEP__Fv
QZSHSH QSHELL 16 main
QZSHSRV1 QSHELL 108 QzshMain
QZSHSRV1 QSHELL 47 cmdloop__Fi
QZSHSRV1 QSHELL 65 evaltree__FP4nodei
QZSHSRV1 QSHELL 117 evaltree__FP4nodei
QZSHSRV1 QSHELL 248 evalcommand__FP4nodeiP7back
QZSHSRV1 QSHELL 124 evalcmdpart2__FP4nodeiP7bac
QZSHSRV1 QSHELL 15 execcmd__FiPPc
QZSHSRV1 QSHELL 1 shellexec__FPcPPcT2
QP0ZEXEC QSYS 47 Qp0zExecve
QP0ZEXEC QSYS 6 run__14Qp0zExecutableFv
QJVATOOLS QJAVA _CXX_PEP__Fv
QJVATOOLS QJAVA 5 main
QJVAQSHCMD QJAVA 716 QjvaQshCmd
QJVAJVM QJAVA 215 RunJava
QJVAJVM QJAVA 17 RunJava_Cleanup__FP11RunJav
QXJ9VM QJVM7064 10 DestroyJavaVM__FP7JavaVM_
QXJ9VM QJVM7064 5 iCallPase__14Qxj9PaseAccess
QP2USER QSYS 1 Qp2CallPase
QP2USER2 QSYS 9 __Qp2CallPase2
QP2USER2 QSYS 4 runpase_common__FiPvT2
unix <callbackrtn>
libi5osenv.so 716 00000064 J9_i5OS_DestroyJavaVM
libjvm.so 00000048 DestroyJavaVM
libj9vm26.so 000002B4 DestroyJavaVM
libj9prt26.so 00000388 j9sig_protect
libj9vm26.so 00000084 protectedDestroyJavaVM
libj9thr26.so 00000034 j9thread_monitor_wait
libj9thr26.so 00000934 monitor_wait_original
libpthreads.a(shr_xpg > 000001A0 pthread_cond_wait
libpthreads.a(shr_xpg > 000000CC _cond_wait
libpthreads.a(shr_xpg > 000004E0 _cond_wait_local
libpthreads.a(shr_xpg > 00000348 _event_wait
libpthreads.a(shr_xpg > 00000584 _event_sleep
unix 00000008 <syscall64>:thread_tsleep

And it yields immediately to an ENDJOB OPTION(*IMMED) issued via Option 4
on WRKACTJOB.

I can't make head or tail of it. Any ideas?

--
JHHL
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list
To post a message email: MIDRANGE-L@xxxxxxxxxxxx
To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxx
Before posting, please take a moment to review the archives
at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our affiliate
link: http://amzn.to/2dEadiD

--
Absolute Performance, Inc.
12303 Airport Way, Suite 100
Broomfield, CO 80021

NON-DISCLOSURE NOTICE: This communication including any and all
attachments is for the intended recipient(s) only and may contain
confidential and privileged information. If you are not the intended
recipient of this communication, any disclosure, copying further
distribution or use of this communication is prohibited. If you received
this communication in error, please contact the sender and delete/destroy
all copies of this communication immediately.