Spanner emits 14-UNAVAILABLE GOAWAY

[pjenvey]

We began seeing 14-UNAVAILABLE GOAWAY received errors after the 0.5.x release to prod, e.g.:

https://sentry.prod.mozaws.net/operations/syncstorage-stage/issues/9186276/

These result in 500 errors (which in turn are 503s from nginx).

[pjenvey]

We have an issue open with google, inquiring on the best way to deal with this and if GOAWAYS are some indication of misbehavior on our end.

[pjenvey]

Some information on GOAWAYS from gRPC's perspective:

• Should we retry on the existing connection? Or disconnect and retry on a separate connection (the go client seems to do the latter)?

GOAWAY indicates that the server wants the client to send no new RPCs on this connection. If the client has more RPCs to send, it should open a new connection. It is OK to wait for existing RPCs to finish before closing the connection.

• Any idea why we haven't seen these before? Does it make sense that we're seeing them this frequently? (is this a bug? there's a recent grpc fix related to GOAWAYs, related?: Add missing reset for ping clocks to avoid mistakenly sending GOAWAY frames due to 'too_many_pings' grpc/grpc#23145

The GOAWAY is server behavior independent of the client, but client behavior such as keeping connections around for longer can trigger it. Many servers send GOAWAY after a fixed duration (say 1 hour) to force clients to reconnect. It is also common for servers to send GOAWAY after a period of inactivity (say 10 minutes) on the connection.

• What does GOAWAY mean besides "retry"? Does this signal something specific e.g. are we misusing our connections: using too many or holding too many idle connections?

See https://tools.ietf.org/html/rfc7540#section-6.8

[pjenvey]

To reduce the number of GOAWAYS we should try:

• Reducing our total number of connections. It's possible it would help (though it may not do much w/ deadpool's FIFO behavior). It may at least reduce a few occurrences of GOAWAYS due to inactivity, which might cause 503 errors QA has seen on an idle stage cluster (stage 0.6.1 POST batch=true&commit=true producing 500 errors  #846)
• Implement equivalent to r2d2/bb8's Builder::max_lifetime (unsupported by deadpool)
• Implement equivalent to r2d2/bb8's Builder::idle_timeout (unsupported by deadpool)

[pjenvey]

Reducing the total number of connections has helped the number of GOAWAYS, one extreme example:

300 max conns:
https://sentry.prod.mozaws.net/operations/syncstorage-prod/issues/9482220/

vs 30:
https://sentry.prod.mozaws.net/operations/syncstorage-prod/issues/9820650/

But there's still too many

[pjenvey]

Initial lifespan/max_idle settings: issue1698211