Re: JVM job for Tomcat taking lots and lots of CPU

From the :

How many VPs does your partition have?
Hardware: Power7 vs Power8/Power9?
How many GC threads?

I wish I knew how to determine the answers to these. What I *do* know is that the box in question self-IDs as model "22A" (which I've never heard of, which in itself suggests Power9), with 22(!) LPARs.

Here is a "Display Java Threads" from the Tomcat server's JVM job:

> Total Aux
> Thread Name Status CPU I/O
> 00000011 main SELW 1.100 130
> 00000014 JIT Compilatio > THDW 61.932 89
> 00000015 JIT Compilatio > THDW 96.846 77
> 00000016 JIT Compilatio > THDW 11.640 10
> 00000017 JIT Compilatio > THDW 1.350 2
> 00000018 JIT-SamplerThr > THDW 23.866 0
> 00000019 IProfiler THDW 5.481 3
> 0000001A Signal Dispatc > THDW 0.983 1
> 0000001C Concurrent Mar > THDW 0.001 0
> 0000001D GC Slave THDW 187.726 157
> 0000001E GC Slave THDW 187.534 420
> 0000001F GC Slave THDW 187.862 217
> 00000020 GC Slave THDW 187.165 362
> 00000021 GC Slave THDW 187.572 340
> 00000022 GC Slave THDW 187.686 333
> 00000023 GC Slave THDW 187.870 161
> 00000024 i5/OS informat > JVAW 0.192 3
> 00000026 Attach API wai > SEMW 0.002 44
> 00000027 MemoryPoolMXBe > THDW 0.001 0
> 00000028 GC Daemon THDW 0.001 0
> 0000002B Finalizer thread THDW 5.939 1475
> 0000005B HSQLDB Timer @ > THDW 0.676 1
> 0000005E ContainerBackg > THDW 0.704 508
> 0000005F http-bio-443-A > SELW 1.634 42
> 00000060 http-bio-443-A > THDW 0.251 0
> 00000061 ajp-bio-8009-A > SELW 0.001 0
> 00000062 ajp-bio-8009-A > THDW 0.243 2
> 0000009B HSQLDB Timer @ > THDW 2.741 185854
> 000000DC Timer-6 THDW 0.001 1
> 000001FD MappedByteBuff > THDW 0.169 10361
> 00000AE2 Thread-2330 THDW 1.111 21159
> 00000EA8 Timer-59 THDW 0.002 2
> 00000EA9 Timer-60 THDW 0.245 0
> 00001C03 Timer-98 THDW 0.002 2
> 00001C04 Timer-99 THDW 0.251 0
> 0000465C Timer-201 THDW 0.001 0
> 0000465D Timer-202 THDW 0.252 0
> 000061BC Timer-244 THDW 0.001 0
> 000061BD Timer-245 THDW 0.178 0
> 00006C30 Timer-254 THDW 0.001 1
> 00006C31 Timer-255 THDW 0.215 0
> 00008C71 Timer-294 THDW 0.001 0
> 00008C72 Timer-295 THDW 0.173 0
> 0000994B Timer-302 THDW 0.001 2
> 0000994C Timer-303 THDW 0.150 0
> 0000C2BD Thread-47354 THDW 0.001 0
> 0000C2C4 Okio Watchdog THDW 0.001 0
> 0000C4E7 Timer-338 THDW 0.001 1
> 0000C4E8 Timer-339 THDW 0.143 0
> 0000D061 Timer-354 THDW 0.001 3
> 0000D062 Timer-355 THDW 0.141 0
> 0000D09E Timer-356 THDW 0.002 0
> 0000D09F Timer-357 THDW 0.150 0
> 00010124 Timer-426 THDW 0.001 0
> 0001012D Timer-427 THDW 0.120 0
> 00011E2B Timer-484 THDW 0.001 0
> 00011E2C Timer-485 THDW 0.103 0
> 000126C4 Timer-507 THDW 0.001 3
> 000126C6 Timer-508 THDW 0.105 0
> 00013C51 Timer-553 THDW 0.001 0
> 00013C52 Timer-554 THDW 0.094 0
> 00013E8D Timer-556 THDW 0.001 0
> 00013E93 Timer-557 THDW 0.093 0
> 000155C5 Timer-597 THDW 0.001 0
> 000155C7 Timer-598 THDW 0.105 0
> 00015911 Timer-605 THDW 0.001 2
> 00015913 Timer-606 THDW 0.084 0
> 00015A4A Heart beat che > THDW 0.013 1
> 00015AAF Heart beat che > THDW 0.012 0
> 00016531 Timer-627 THDW 0.001 1
> 00016532 Timer-628 THDW 0.087 0
> 00017039 Timer-645 THDW 0.001 0
> 0001703A Timer-646 THDW 0.105 0
> 00017363 Timer-651 THDW 0.001 0
> 00017364 Timer-652 THDW 0.092 0
> 000189A0 Timer-702 THDW 0.001 1
> 000189A1 Timer-703 THDW 0.083 0
> 00018FBB Timer-714 THDW 0.001 1
> 00018FBC Timer-715 THDW 0.088 0
> 00019D3E Timer-734 THDW 0.001 0
> 00019D3F Timer-735 THDW 0.065 0
> 0001A399 Timer-744 THDW 0.001 0
> 0001A39A Timer-745 THDW 0.062 0
> 0001B090 Timer-757 THDW 0.001 0
> 0001B091 Timer-758 THDW 0.056 0
> 0001B624 Heart beat che > THDW 0.005 0
> 0001C0D7 Timer-777 THDW 0.001 0
> 0001C0D8 Timer-778 THDW 0.041 0
> 0001C247 http-bio-443-e > THDW 5.652 1314
> 0001C33C Heart beat che > THDW 0.003 0
> 0001C75A Timer-791 THDW 0.001 0
> 0001C75B Timer-792 THDW 0.034 0
> 0001CB2F Timer-797 THDW 0.001 0
> 0001CB30 Timer-798 THDW 0.029 0
> 0001CE0A Timer-801 THDW 0.001 0
> 0001D07E Timer-807 THDW 0.001 0
> 0001D07F Timer-808 THDW 0.020 0
> 0001D08B http-bio-443-e > SELW 3.735 446
> 0001D0F1 http-bio-443-e > SELW 6.565 1176
> 0001D2B9 http-bio-443-e > SELW 0.071 0
> 0001D2D4 Timer-809 THDW 0.001 0
> 0001D2D5 Timer-810 THDW 0.023 0
> 0001D2EB Heart beat che > THDW 0.002 0
> 0001D395 Heart beat che > THDW 0.003 0
> 0001D6D2 http-bio-443-e > SELW 3.689 638
> 0001DD6C Timer-815 THDW 0.001 0
> 0001DD6D Timer-816 THDW 0.013 0
> 0001DEFC Heart beat che > THDW 0.002 0
> 0001DFCA Timer-819 THDW 0.001 0
> 0001DFCB Timer-820 THDW 0.011 0
> 0001E014 Timer-821 THDW 0.001 0
> 0001E015 Timer-822 THDW 0.010 0
> 0001E041 Timer-823 THDW 0.001 0
> 0001E042 Timer-824 THDW 0.012 0
> 0001E09E Timer-825 THDW 0.001 0
> 0001E09F Timer-826 THDW 0.012 0
> 0001E0B4 http-bio-443-e > SELW 1.082 120
> 0001E0F9 Timer-827 THDW 0.001 0
> 0001E0FA Timer-828 THDW 0.010 0
> 0001E224 Timer-829 THDW 0.001 0
> 0001E225 Timer-830 THDW 0.008 0
> 0001E2F7 http-bio-443-e > SELW 1.473 660
> 0001E2FE http-bio-443-e > SELW 0.174 41
> 0001E3D7 http-bio-443-e > SELW 1.069 190
> 0001E417 http-bio-443-e > SELW 1.109 96
> 0001E4AC http-bio-443-e > SELW 1.619 308
> 0001E4C1 http-bio-443-e > THDW 2.186 393
> 0001E4CD Timer-833 THDW 0.001 0
> 0001E51F Heart beat che > THDW 0.001 0
> 0001E578 http-bio-443-e > SELW 0.955 169
> 0001E57D http-bio-443-e > SELW 2.412 336
> 0001E57E http-bio-443-e > THDW 1.442 168
> 0001E581 http-bio-443-e > SELW 0.021 0
> 0001E589 http-bio-443-e > THDW 0.793 126
> 0001E58B http-bio-443-e > SELW 0.567 121
> 0001E58C http-bio-443-e > SELW 0.156 0
> 0001E58E http-bio-443-e > SELW 0.497 124
> 0001E57A http-bio-443-e > THDW 1.446 364
> 0001E590 http-bio-443-e > SELW 1.831 251
> 0001E585 http-bio-443-e > SELW 1.716 202
> 0001E586 http-bio-443-e > SELW 0.335 26
> 0001E58F http-bio-443-e > SELW 0.010 0
> 0001E57B http-bio-443-e > SELW 0.368 77
> 0001E579 http-bio-443-e > SELW 0.595 39
> 0001E646 http-bio-443-e > SELW 0.906 172
> 0001E64A http-bio-443-e > SELW 0.011 0
> 0001E64C http-bio-443-e > SELW 0.840 200
> 0001E64D http-bio-443-e > SELW 0.010 0
> 0001E651 http-bio-443-e > SELW 1.903 328
> 0001E654 Heart beat che > THDW 0.001 0
> 0001E65A Heart beat che > THDW 0.001 0
> 0001E68B Heart beat che > THDW 0.001 0
> 0001E68D Heart beat che > THDW 0.001 0
> 0001E6E0 Heart beat che > THDW 0.001 0
> 0001E6FC http-bio-443-e > SELW 0.988 100
> 0001E6FE http-bio-443-e > SELW 0.774 103
> 0001E70E http-bio-443-e > SELW 1.502 317
> 0001E710 Heart beat che > THDW 0.001 0
> 0001E721 Heart beat che > THDW 0.001 0
> 0001E723 Heart beat che > THDW 0.001 0
> 0001E72E http-bio-443-e > SELW 1.000 123
> 0001E733 http-bio-443-e > SELW 1.056 230
> 0001E738 http-bio-443-e > SELW 0.004 0
> 0001E752 Heart beat che > THDW 0.001 0
> 0001E754 Heart beat che > THDW 0.001 0
> 0001E762 Heart beat che > THDW 0.001 0
> 0001E781 Heart beat che > THDW 0.001 0
> 0001E7CE Timer-834 THDW 0.001 0
> 0001E7CF Timer-835 THDW 0.004 0
> 0001E7DC Heart beat che > THDW 0.001 0
> 0001E7DE Heart beat che > THDW 0.001 0
> 0001E7E5 http-bio-443-e > THDW 0.251 16
> 0001E7F1 Timer-836 THDW 0.001 0
> 0001E7F2 Timer-837 THDW 0.004 0
> 0001E805 http-bio-443-e > THDW 0.243 0
> 0001E824 http-bio-443-e > THDW 0.081 0
> 0001E826 http-bio-443-e > SELW 0.080 0
> 0001E828 http-bio-443-e > SELW 0.472 32
> 0001E82A Heart beat che > THDW 0.001 0
> 0001E841 Heart beat che > THDW 0.001 0
> 0001E851 Heart beat che > THDW 0.001 0
> 0001E853 Heart beat che > THDW 0.001 0
> 0001E866 Heart beat che > THDW 0.001 0
> 0001E8A8 Heart beat che > THDW 0.001 0
> 0001E8B4 Heart beat che > THDW 0.001 0
> 0001E8DB Heart beat che > THDW 0.001 0
> 0001E8EF Heart beat che > THDW 0.001 0
> 0001E8FC Heart beat che > THDW 0.001 0
> 0001E906 Heart beat che > THDW 0.001 0
> 0001E987 http-bio-443-e > THDW 0.758 1258
> 0001E98C Heart beat che > THDW 0.001 0
> 0001E99A Heart beat che > THDW 0.001 0
> 0001E9C4 Heart beat che > THDW 0.001 0
> 0001E9D6 Heart beat che > THDW 0.001 0
> 0001E9DF Heart beat che > THDW 0.001 0
> 0001E9F8 Heart beat che > THDW 0.001 0
> 0001E9FE Heart beat che > THDW 0.001 0
> 0001EA07 http-bio-443-e > SELW 0.001 0
> 0001EA0E Heart beat che > THDW 0.001 0
> 0001EA18 http-bio-443-e > THDW 0.301 85
> 0001EA19 http-bio-443-e > SELW 0.002 0
> 0001EA1A http-bio-443-e > SELW 0.004 0
> 0001EA1B http-bio-443-e > SELW 0.205 530
> 0001EA1C http-bio-443-e > THDW 0.021 0
> 0001EA1D http-bio-443-e > SELW 0.004 0
> 0001EA29 Ehcache [_defa > THDW 0.001 0
> 0001EA2C Ehcache [_defa > THDW 0.001 0
> 0001EA3E Heart beat che > THDW 0.001 0
> 0001EA4D Heart beat che > THDW 0.001 0
> 0001EA59 Thread-119782 TIMA 0.047 0

A lot of threads. Looks like a GC Daemon and 7 GC Slaves, among all that.

--
JHHL