Tl;dr: change persistent-con-timeout in the ISVA [server] stanza from 5 to a value larger than 6

We were routing traffic from various clients to an IBM Security Verify Access (ISVA, formerly ISAM) Reverse Proxy (WebSEAL) via an AWS Load Balancer (ALB). The ALB was intermittently rejecting the requests with a 502 response; according to the ALB logs, this was because the ISVA Reverse Proxy was closing the connection immediately, without providing a response to the ALB. There was nothing helpful in the Reverse proxy logs; when I ran a packet capture trace, I saw that there would be a packet from the ALB to the Reverse Proxy on an existing TCP/IP connection – but the Reverse Proxy responded with a TCP Reset (RST) packet. Why was this?

I did some experimenting with a bash script which used curl to send a request every few seconds, and found that if the interval between the requests was less than five seconds or more than six seconds then everything worked fine – but if the interval between the requests was between five and six seconds, then it would fail with the 502 error.

Based on this, I reckoned that the problem was an incompatibility in the timeout settings between ALB and ISVA: the persistent-con-timeout setting in the [server] stanza in the Reverse Proxy configuration file specifies, from an ISVA perspective, how long a session between the
ALB and ISVA can be idle; by default, it is five seconds. Based on the observed behaviour, the corresponding ALB figure appears to be six seconds (although we couldn’t find where this is specified).

Herein lies the problem: if there is a gap of, say, 5.5 seconds between requests, then as far as the ALB is concerned it can send the second request as part of an existing persistent connection; however, as far as the ISVA Reverse Proxy is concerned, that persistent connection has now timed out, so it rejects the connection with a TCP Reset (RST) packet. When we increased the persistent-con-timeout setting to more than 6, it resolved the problem.