RE: Reposting: TCP connections getting randomly cut off. Even loopback connections.

James,

Last week, on 2/7, we had an issue where iSeries Client Access sessions and Rocket Jwalk/Legasuite sessions dropping (All persistent connections) , with CPF5140 posted to joblog.
The PC non-persistent connections were not impacted.

We did find that each PC that dropped had its "lease renewed" minutes prior to CPF5140 time stamp.
We also found that each PC NIC connection was told to "stop/shutdown."
We don't know why.

No final resolution as of yet.

Started 2/07 at 12:45.
Ended 2/07 at 14:25.
Multiple iSeries LPARs - Pencor05, Pencor06, Pencor07, Penco08 My connections to 4 different LPARS, 3 different NICS, all at the same time.
Not all users at once.
About 20 every minute.
Each user only once.
No entries in any system logs. PAL, LIC, QHST
Only the CPF5140 in each user joblog.

iSeries performance graph showed high number of TCP connection resets during the 12:45 thru 14:25 problem window.

from 3 LPARS, 3 different NICs
Appears that only 5250 Client Access and GUI was impacted (PC traffic). from 3 LPARS, 3 different iSeries NICs All other applications (hundreds) connecting to both internal and external remotes were fine.

Paul



-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces@xxxxxxxxxxxx] On Behalf Of James H. H. Lampert
Sent: Wednesday, February 15, 2017 1:19 PM
To: Midrange Systems Technical Discussion
Subject: Reposting: TCP connections getting randomly cut off. Even loopback connections.

Has ANYBODY seen ANYTHING like this happen?

A week ago, I posted a question about a situation we've literally never seen before, in well over a decade of working with TCP connections. It appears to be happening with both the proprietary protocol of our CRM product, and with secured Telnet sessions, and it happens randomly. The only thing that the two appear to have in common is that they both maintain persistent connections.

Starting sometime (if I remember right) in January, we've had a customer with some weird things happening to TCP connections.

They will disconnect, seemingly at random, for no apparent reason, and often (particularly with terminal sessions via Telnet) with the job failing to notice the disconnection.

I have, on several occasions, had two terminal sessions going with the system in question, and one will lose its connection, while the other will not, and the job for the lost connection will just sit there in WRKACTJOB, apparently completely unaware that it was no longer communicating with the terminal emulator, and it will often take several
*minutes* to terminate it, even with OPTION(*IMMED).

And with the CRM product's propriatary protocol, it even happens (and indeed, was first noticed) in loopback connections: the web server for the product, running on the same physical box, connects to the proprietary server via an explicit IP address of 127.0.0.1.

--
JHHL