RE: HTTP intermittent daily failures, only between 4:02:30 thru 4:04:30

Thanks for all the responses.

No latency on the firewalls.
I is basically idle between 4 and 4:05, no perf issue here.
Multiple internal networks, all fiber connected.
The window is not exactly 4:02:30 thru 4:04:30, could vary +/- 10 seconds on either side.
Similar process with 50x more load run half hour later, between 4:30 and 5:30, no errors.

Paul

-----Original Message-----
From: MIDRANGE-L <midrange-l-bounces@xxxxxxxxxxxxxxxxxx> On Behalf Of Nathan Andelin
Sent: Monday, March 16, 2020 2:49 PM
To: Midrange Systems Technical Discussion <midrange-l@xxxxxxxxxxxxxxxxxx>
Subject: Re: HTTP intermittent daily failures, only between 4:02:30 thru 4:04:30

The precise 2-minute window when TCP resets are occurring stumps me. What could be happening on you network or your server at that time, that is different? An increase in workload? A scheduled process that affects the workload or creates a lock on something. In my experience, it has always been something in the application environment that causes these type of anomalies, as opposed to a network issue. I'd try increasing HTTP configuration parameters first:

ThreadsPerChild nn
KeepAliveTimeout nn
TimeOut nnn



On Mon, Mar 16, 2020 at 12:06 PM Steinmetz, Paul via MIDRANGE-L < midrange-l@xxxxxxxxxxxxxxxxxx> wrote:

We are having HTTP intermittent daily failures, only between 4:02:30
thru 4:04:30.
Majority of the transactions are successful within the same window.
The problem has existed since May 2018.

We now have a 2nd, new HTTP app, different IP and port, presenting the
same issue.

Not a load issue.
We run tens of thousands transactions later in the day, not a single error.
Both use non-persistent connections, meaning we open/close the ports
for every process.
V7R3, latest cume and groups.
NonSSL.
I side - RPGLE client application utilizing RXS_Transmit() Remote -
3rd party proprietary apps.
5 hops, 3 firewalls, several switches and routers.

From a WireShark pcap trace run from the I, IBM confirmed that RST
flag is being set, from the remote IP, causing the connection to reset/close.

We've been unable to determine the cause the RST.
What process and where would this be running for this brief two
minutes window (4:02:30 thru 4:04:30) that would cause RST,
reset/closing the connection.

Anyone in the group experience similar behavior and/or know how to
isolate the root cause/resolution?

Thank You
_____
Paul Steinmetz
IBM i Systems Administrator

Pencor Services, Inc.
462 Delaware Ave
Palmerton Pa 18071

610-826-9117 work
610-826-9188 fax
610-349-0913 cell
610-377-6012 home

psteinmetz@xxxxxxxxxx<mailto:psteinmetz@xxxxxxxxxx>
http://www.pencor.com/

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To
subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at
https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related
questions.

Help support midrange.com by shopping at amazon.com with our affiliate
link: https://amazon.midrange.com

--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L@xxxxxxxxxxxxxxxxxx To subscribe, unsubscribe, or change list options,
visit: https://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request@xxxxxxxxxxxxxxxxxx
Before posting, please take a moment to review the archives at https://archive.midrange.com/midrange-l.

Please contact support@xxxxxxxxxxxx for any subscription related questions.

Help support midrange.com by shopping at amazon.com with our affiliate link: https://amazon.midrange.com