Re: Unexpected JVM crash, assistance requested

Hi there

After another JVM crash yesterday we've found some more information in std
err:

Assertion failed at /afs/
rchland.ibm.com/usr8/j9/src/rebuilt/jdk80/64bit/sr5fp20.v7r3/jre/lib/ppc64/compressedrefs/compiler/../omr/c
ompiler/codegen/OMRTreeEvaluator.cpp:348: false
VMState: 0x0005ff04
badILOp BadILOp cannot be evaluated
compiling
com/internationalpresence/impscloudgateway/util/AS400LogHandler.flush()V at
level: hot
assumeDontCallMeDirectlyImpl(const char*,int,const char*,const
char*,char*,bool)+0x4d0
TR::fatal_assertion(const char*,int,const char*,const char*,...)+0x40
OMR::TreeEvaluator::badILOpEvaluator(TR::Node*,TR::CodeGenerator*)+0x4c
OMR::CodeGenerator::evaluate(TR::Node*)+0x510
J9::CodeGenerator::doInstructionSelection()+0x4ed8
OMR::CodeGenPhase::performInstructionSelectionPhase(TR::CodeGenerator*,TR::CodeGenPhase*)+0x5c0
OMR::CodeGenPhase::performAll()+0x2b4
OMR::CodeGenerator::generateCode()+0x484
OMR::Compilation::compile()+0x2188
TR::CompilationInfoPerThreadBase::compile(J9VMThread*,TR::Compilation*,TR_ResolvedMethod*,TR_J9VMBase&,TR_OptimizationPlan*,const
TR
::SegmentAllocator&)+0x161c
TR::CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*,void*)+0x1cd4
omrsig_protect+0x5a8
TR::CompilationInfoPerThreadBase::compile(J9VMThread*,TR_MethodToBeCompiled*,J9::J9SegmentProvider&)+0x774
TR::CompilationInfoPerThread::processEntry(TR_MethodToBeCompiled&,J9::J9SegmentProvider&)+0x854
TR::CompilationInfoPerThread::processEntries()+0x7f8
protectedCompilationThreadProc(J9PortLibrary*,TR::CompilationInfoPerThread*)+0x4e0
omrsig_protect+0x5a8
compilationThreadProc(void*)+0x5b8
thread_wrapper+0x254
_pthread_body+0x138
+0x0
The assert subroutine failed: 0, file /afs/
rchland.ibm.com/usr8/j9/src/rebuilt/jdk80/64bit/sr5fp20.v7r3/jre/lib/ppc64/compressedrefs
/compiler/../omr/compiler/infra/Assert.cpp, line 72
JVMDUMP039I Processing dump event "abort", detail "" at 2018/11/15 07:41:09
- please wait.
JVMDUMP032I JVM requested System dump using
'//core.20181115.074109.435449.0001.dmp' in response to an event
JVMDUMP010I System dump written to //core.20181115.074109.435449.0001.dmp
JVMDUMP032I JVM requested Java dump using
'//javacore.20181115.074109.435449.0002.txt' in response to an event
JVMDUMP010I Java dump written to //javacore.20181115.074109.435449.0002.txt
JVMDUMP032I JVM requested Snap dump using
'//Snap.20181115.074109.435449.0003.trc' in response to an event
JVMDUMP010I Snap dump written to //Snap.20181115.074109.435449.0003.trc
JVMDUMP007I JVM Requesting JIT dump using
'//jitdump.20181115.074109.435449.0004.dmp'
#JITDUMP: vmThread=000000003000EF00 Crashed while printing out current
IL.JVMDUMP010I JIT dump written to //jitdump.20181115.074109
.435449.0004.dmp
JVMDUMP013I Processed dump event "abort", detail "".

Does this help pinpoint the cause?

Thanks
Matt


On Thu, 15 Nov 2018 at 13:32, Matt Pryor <pryor@xxxxxxxxxxxxxxxxxxxxxxxxxx>
wrote:

Hi again Steve

From the customer:

QJVACMDSRV authorities are *PUBLIC *USE. I verified the TLXDRIVER
authorities match what you noted below.

Best wishes
Matt

On Thu, 15 Nov 2018 at 09:45, Matt Pryor <pryor@xxxxxxxxxxxxxxxxxxxxxxxxxx>
wrote:

Hi Steve

Thanks for your reply.

Java is started via a CLP, this is the section that calls it:

JAVA +
CLASS('com.internationalpresence.impscloudg+
ateway.CloudQ') PARM(&DRIVER &INTERNET +
&PRODLIB &CLDFLR &TYPE &ENV &HOSTIP +
&CLDSSL &LOGTYP) CLASSPATH(*ENVVAR) +
CHKPATH(*IGNORE) PROP((java.version &JVM) +
(java.awt.headless true) +
(os400.awt.native true) +
(com.ibm.cacheLocalHost true) +
(java.net.preferIPv4Stack &IP4) +
(java.net.preferIPv6Addresses &IP6)) +
OPTION(&OPT) OUTPUT(*PRINT)

The program runs for several hours or days before the JVM crashes. We
have a 2 gb dump file and also a java core text file which shows all the
threat stack traces etc.

All Java call stacks available show threads to either be in a wait state,
or a sleep state, or a socket read or accept state with the exception of
the following:

1)
4XESTACKTRACE at
com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
4XESTACKTRACE at
com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:73)

2)
4XESTACKTRACE at
com/ibm/tools/attach/target/IPC.waitSemaphore(Native Method)
4XESTACKTRACE at
com/ibm/tools/attach/target/CommonDirectory.waitSemaphore(CommonDirectory.java:251)

3)
4XESTACKTRACE at
com/ibm/lang/management/internal/MemoryNotificationThread.processNotificationLoop(Native
Method)
4XESTACKTRACE at
com/ibm/lang/management/internal/MemoryNotificationThread.run(MemoryNotificationThread.java:185)

In case it's any use to you I've uploaded the java core text file to


http://www.international-presence.com/software/matt_transfer/javacore.20181104.102029.215708.0002.txt

TLXDRIVER is the user profile that the CL program runs as, which should
belong to *SYSOPR user group and have special authority for *ALLOBJ,
*JOBCTL, *SECADM, *SPLCTL

I will ask the customer to check on authority for QJVACMDSRV *pgm.

Many thanks
Matt

1. Re: Unexpected JVM crash, assistance requested
(Steve Hinrichs via JAVA400-L)


----------------------------------------------------------------------

message: 1
date: Tue, 13 Nov 2018 19:09:50 +0000
from: Steve Hinrichs via JAVA400-L <java400-l@xxxxxxxxxxxx>
subject: Re: Unexpected JVM crash, assistance requested

How is the java process started, from RPG, RUNJVA, QSH?
Is their any call stack or is it failing right away?
What is the class/method being called?
What authority does the user have that is running the java process,
maybe try QSECOFR.

I would check authority and existence of QJAVA/QJVACMDSRV *pgm.
What is TLXDRIVER? A user profile? Searching for TLXDRIVER I find 2
items related to user profiles.

Steve

--
Matt Pryor
Software Developer

The International Presence Group of Companies
EMAIL: pryor@xxxxxxxxxxxxxxx
URL: www.International-presence.com