transport: add timeout for writing GOAWAY on http2Client.Close()

[aranjans]

fixes #7314

Problem

In gRPC 1.64, we released a feature "Send GOAWAY to server on Client Transport Shutdown", post which in case of network hang when we are not able to write GOAWAY frame from client, closing connections will take a very long time.

What does this PR fix

This PR fixes this issue by having a timeout for loopyWriter to write GOAWAY so that http2Client.Close() doesn't wait for too long.

RELEASE NOTES:

• transport/http2_client: fix waiting on http2Client.Close() in case of network hang for writing GOAWAY.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.85%. Comparing base (ced812e) to head (fad99ba).
Report is 15 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7371      +/-   ##
==========================================
+ Coverage   81.58%   81.85%   +0.27%     
==========================================
  Files         358      360       +2     
  Lines       27388    27539     +151     
==========================================
+ Hits        22345    22543     +198     
+ Misses       3828     3796      -32     
+ Partials     1215     1200      -15     

Files	Coverage Δ
internal/transport/http2_client.go	91.90% <100.00%> (-0.16%)	⬇️

... and 38 files with indirect coverage changes

[dfawley]

Remove this timer? Otherwise it will release on its own and pass on master@HEAD, won't it?

[dfawley]

Ping, this doesn't seem done.

[aranjans]

Umm... I think we would need to have timeout for hc.hangConn, otherwise this go-routine will be leaked via this call. So hc.Conn.Write will be hang after we made this write to the conn, which will eventually make the above go-routine to run forever and end up in leaked go-routine.(btw we already have deadline for net.Conn.Write defined here at the top of ct.Close, but then the current test will test only for this loopyWriter to timeout).
WDYT?

[purnesh42H]

Not entirely familiar with the goroutine you are referring to but one suggestion is that you probably don't have to do the actual write n, err = hc.Conn.Write(b) in case of goaway? You can just compare the passed bytes b if its goaway and then just hang? or is that not possible because different structs are being written?

[aranjans]

Ah, actually we don't even need to compare it, we can just make sure hangingConn hangs from the time it receives the goAway request.

@dfawley I have removed the timer logic for hc.Write, and updated the PR.

[purnesh42H]

probably good to release the <-hc.hangConn at the end of test?

[dfawley]

I'd really like to wrap this up before we cut the release branch. Will that be possible?

[dfawley]

Why did you remove the atomic?

This is worse, since you're just assuming that: 1. Nothing else written will be the size of the GOAWAY frame, and 2. the GOAWAY will be written to the connection, by itself, in a single operation.

[dfawley]

if x == true --> if x, please

[aranjans]

done