Add setting for tcp_keepalive for oidc back-channel

[ywangd]

This PR adds a new setting to enable tcp keepalive probes for the
connections used by the oidc back-channel communication. It defaults to
true as tcp keepalive is generally useful for ES.

Relates: #87773

[elasticsearchmachine]

Hi @ywangd, I've created a changelog YAML for you.

[ywangd]

NOTE: This is currently a draft PR because:

1. I cannot find a way to test tcp keepalive
2. Depending on the OS's configuration, the keepalive probe may not be enough keep the connection up. Add setting for keep-alive duration for oidc back-channel #87773 is still the right candidate to address Custom keep-alive strategy for outgoing OIDC connections #75515. But maybe this change can be useful for some other situations.

[ywangd]

There is no easy way to test this setting. Other than invasive reflection, one possibility is to subclass DefaultConnectingIOReactor and override the prepareSocket method where we can assert the socket configuration. It feels heavy weighted for testing though. We may need the subclass in production if we decide to support for configuring keepalive_time, interval etc in future. So I think we can save the test till we actually need that in production.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[DaveCTurner]

It would be good to at least confirm manually that this does enable keepalives on the connection (e.g. using ss -toe). I expect it does so this LGTM.

If we were to do something like #84653 then we could check this timer directly in a test (at least on Linux).

[ywangd]

It would be good to at least confirm manually that this does enable keepalives on the connection (e.g. using ss -toe).

Thanks for the tip. I manually verified it and it worked:

Before

ESTAB 0 0 ***:60142 ***:443 uid:1000 ino:253847 sk:2002 <->

After

ESTAB 0 0 *:53448 *:443 timer:(keepalive,119min,0) uid:1000 ino:261040 sk:4001 <->

[tvernum]

LGTM, pending fixed typo in docs.

[ywangd]

@elasticmachine update branch