RE: WAS non-responsive

Todd,

We upgraded to WAS 6.0.2.15 about 4 weeks ago.  Right now, no known
major issues like yours.  Our production stuff is Portal and I have some
test servers that are straight WAS so my workload is very different then
yours.

Out of curiosity, what levels are you at for OS and CUM?  I am at V5R3
and 06255.



Michael Crump
 
Manager, Computing Services
Saint-Gobain Containers, Inc.
1509 S. Macedonia Ave.
Muncie, IN  47302
 
765.741.7696
765.741.7012 f 
 


Dare to Slack
When birds fly in the right formation, they need only exert half the
effort.  Even in nature, teamwork results in collective laziness.
-----Original Message-----
From: java400-l-bounces@xxxxxxxxxxxx
[mailto:java400-l-bounces@xxxxxxxxxxxx] On Behalf Of Todd Bryant
Sent: Tuesday, December 05, 2006 2:28 PM
To: java400-l@xxxxxxxxxxxx
Subject: WAS non-responsive

We are having a problem since we have upgraded to WAS 6.0.2.15 on the
iSeries. Unfortunately, we have little information at this time, but I
thought I would throw out what is happening and see if anyone has any
input. 

 

On three separate occasions WAS has ground to a halt and stopped
responding to page requests. It is not really hung, per se. It is more
like it is stuck in a loop or paging out memory or something. When it
happens, CPU usage, when viewed using WRKACTJOB, shows minimal usage,
not only for WAS but also for the whole system. Other jobs seem to be
running fine e.g, interactive and batch jobs.

 

We had one instance where there were two remote data connections that
someone had that were using only a little cpu, but we thought those jobs
may be doing intensive I/O, so we killed those jobs and WAS became
responsive immediately. However, in the other cases another job causing
the problem could not be identified absolutely. 

 

The last time this happened I tried running the DMPJVM command, but it
simply hung and never finished. I have also tried to get into the admin
to check the Tivoli Performance Viewer to look at heap size, but the
admin app would not come up.

 

WAS is currently set up to use the BASE memory pool. We have 5 or 6 gig
of ram in the machine and BASE normally has 2.5 to 3.5 gig allocated to
it, depending on interactive. Using the Tivoli Performance Viewer in the
WAS admin, WAS usually has a memory allocation of 500-700 meg, and the
used memory is 250-300meg. 

 

We have gotten some error messages in the logs occasionally that make me
think that this may be a memory issue:

[12/4/06 23:47:43:931 CST] 00000011 SystemOut     O C400WARNI:
485997/QEJBSVR/SERVER1 GC heap uses 107% of the non-reserved pool.

 JVM GC Heap Size(kB)  Effective PoolSize(kB):408273 381196

 

 

I am not sure how to read this. It appears it may be saying that the
pool size has shrunk to 381meg and it is using 408meg. One hypothesis I
have at this point is that other jobs in the BASE memory pool are using
a large amount of ram and that ram available to WAS is being cut back
because of it and WAS is either running the GC constantly, or worse, it
is paging out to disk. 

 

One time when we tried to shut down the server when it became
non-responsive we had this in the logs:

[11/17/06 8:58:01:651 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 32" (00000fcd) has been active for 736230 milliseconds
and may be hung.  There is/are 2 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:07:848 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 34" (00000fcf) has been active for 735964 milliseconds
and may be hung.  There is/are 3 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:08:712 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 17" (00000fbe) has been active for 737383 milliseconds
and may be hung.  There is/are 4 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:09:596 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 20" (00000fc1) has been active for 736983 milliseconds
and may be hung.  There is/are 5 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:10:161 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 33" (00000fce) has been active for 736005 milliseconds
and may be hung.  There is/are 6 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:10:809 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 31" (00000fcc) has been active for 736379 milliseconds
and may be hung.  There is/are 7 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:11:222 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 4" (00000182) has been active for 739389 milliseconds
and may be hung.  There is/are 8 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:11:480 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 8" (00000186) has been active for 730294 milliseconds
and may be hung.  There is/are 9 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:11:837 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 19" (00000fc0) has been active for 730323 milliseconds
and may be hung.  There is/are 10 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:12:131 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 44" (00000fda) has been active for 701857 milliseconds
and may be hung.  There is/are 11 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:13:075 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 18" (00000fbf) has been active for 737204 milliseconds
and may be hung.  There is/are 12 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:13:326 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 43" (00000fd9) has been active for 704627 milliseconds
and may be hung.  There is/are 13 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:13:548 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 13" (00000fba) has been active for 738022 milliseconds
and may be hung.  There is/are 14 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:13:886 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 30" (00000fcb) has been active for 736480 milliseconds
and may be hung.  There is/are 15 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:14:119 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 29" (00000fca) has been active for 736551 milliseconds
and may be hung.  There is/are 16 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:14:338 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 24" (00000fc5) has been active for 736812 milliseconds
and may be hung.  There is/are 17 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:14:548 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 28" (00000fc9) has been active for 736611 milliseconds
and may be hung.  There is/are 18 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:14:821 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 12" (00000fb9) has been active for 738008 milliseconds
and may be hung.  There is/are 19 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:15:009 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 23" (00000fc4) has been active for 730255 milliseconds
and may be hung.  There is/are 20 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:15:340 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 42" (00000fd8) has been active for 713749 milliseconds
and may be hung.  There is/are 21 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:15:515 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 14" (00000fbb) has been active for 737527 milliseconds
and may be hung.  There is/are 22 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:15:705 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 35" (00000fd0) has been active for 735872 milliseconds
and may be hung.  There is/are 23 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:15:986 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 26" (00000fc7) has been active for 736671 milliseconds
and may be hung.  There is/are 24 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:16:150 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 47" (00000fdd) has been active for 687165 milliseconds
and may be hung.  There is/are 25 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:16:310 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 37" (00000fd2) has been active for 735783 milliseconds
and may be hung.  There is/are 26 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:16:497 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 10" (00000fb7) has been active for 738634 milliseconds
and may be hung.  There is/are 27 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:16:765 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 25" (00000fc6) has been active for 736671 milliseconds
and may be hung.  There is/are 28 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:16:969 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 36" (00000fd1) has been active for 735868 milliseconds
and may be hung.  There is/are 29 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:17:138 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 39" (00000fd4) has been active for 735728 milliseconds
and may be hung.  There is/are 30 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:17:309 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 27" (00000fc8) has been active for 736611 milliseconds
and may be hung.  There is/are 31 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:17:506 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 16" (00000fbd) has been active for 737454 milliseconds
and may be hung.  There is/are 32 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:17:720 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 46" (00000fdc) has been active for 689796 milliseconds
and may be hung.  There is/are 33 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:18:110 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 11" (00000fb8) has been active for 738302 milliseconds
and may be hung.  There is/are 34 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:18:294 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 22" (00000fc3) has been active for 736911 milliseconds
and may be hung.  There is/are 35 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:18:604 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 6" (00000184) has been active for 730294 milliseconds
and may be hung.  There is/are 36 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:18:830 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 21" (00000fc2) has been active for 736890 milliseconds
and may be hung.  There is/are 37 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:19:136 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 41" (00000fd6) has been active for 730189 milliseconds
and may be hung.  There is/are 38 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:19:326 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 48" (00000fde) has been active for 675698 milliseconds
and may be hung.  There is/are 39 thread(s) in total in the server that
may be hung.

[11/17/06 8:58:19:597 CST] 000001a8 ThreadMonitor W   WSVR0605W: Thread
"WebContainer : 40" (00000fd5) has been active for 735688 milliseconds
and may be hung.  There is/are 40 thread(s) in total in the server that
may be hung.

 

 

Which made me wonder if it was a thread issue, but I could also see this
happening if the GC was dominating or the jvm was paging to disk. 

 

What I am wondering is if anyone else has had problems like this or if
anyone can give me any ideas on what to check or change. Thanks for any
help you can give us.

 

 

Todd Bryant

Programmer/Analyst

University of Nebraska Foundation

phone#: 402.458.1131

 




NOTE: This electronic message and attachment(s), if any, contains
information which is intended solely for the designated recipient(s).
Unauthorized disclosure, copying, distribution, or other use of the
contents of this message or attachment(s), in whole or in part, is
prohibited without the express authorization of the sender of this
message.