Add keepalive setting in client library.

[sushanb]

Java grpc does not have keepalive configuration by default. It will use the kernel tcp default configuration as the values

sysctl -a | grep tcp
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.ipv4.tcp_keepalive_time = 7200

Vals in seconds

Why Keepalive in GRPC Client

Keepalive is recommended to use when your peer dies before it is able to notify you or does not notify you. The reasons can be kernel panic, termination of the process handling that pee

Hence, if a server does not send GOAWAY to the client, the grpc client will send packets to the TCP peer but the TCP perr will not send any packets. This can happen when a server nodes goes down due to machine failure, release/rollouts.

As you noticed the tcp (unacknowledged , keepalive) times are in 5 mins and 2 hours, so the reasonable solution will be to add keepalive logic in client library.

Steps to reproduce

To reproduce the problem, run the following command on a gRPC client host:

iptables -I INPUT -s <server-ip> -p tcp --sport <server-port> -j DROP