konnectivity-client: ensure connection read channel is closed #325

andrewsykim · 2022-02-14T21:15:34Z

This PR ensures that the readCh on the connection from a single use gRPC tunnel is always closed. While running some tests I noticed that goroutines can leak if it reads conn.readCh. This happens when the client never receives CLOSE_RSP, which seems possible if the transport already closed for some other reason.

Note: sorry I accidentally pushed this branch to upstream instead of my fork.

andrewsykim · 2022-02-14T21:17:14Z

tests/proxy_test.go

@@ -219,6 +220,58 @@ func TestProxyHandle_SlowContext_GRPC(t *testing.T) {
 	}
 }

+func TestProxyHandle_ContextCancelled_GRPC(t *testing.T) {
+	defer goleak.VerifyNone(t)


Without the bug fix in konnectivity-client/pkg/client/client.go:

$ go test ./... -run TestProxyHandle_ContextCancelled_GRPC E0214 16:09:13.663053 847761 client.go:103] "stream read failure" err="rpc error: code = Canceled desc = context canceled" E0214 16:09:13.663475 847761 server.go:390] "Stream read from frontend failure" err="rpc error: code = Canceled desc = context canceled" E0214 16:09:13.664071 847761 server.go:751] "send to client stream failure" err="rpc error: code = Unavailable desc = transport is closing" serverID="846800f0-2f8e-4f29-b25a-50153d1f00cd" agentID="57a4766a-a392-4c4f-8146-c5a25b0c510f" connectionID=1 E0214 16:09:13.664223 847761 client.go:371] "could not read stream" err="rpc error: code = Unavailable desc = error reading from server: EOF" E0214 16:09:13.664452 847761 server.go:515] "CLOSE_REQ to Backend failed" err="rpc error: code = Unavailable desc = transport is closing" serverID="846800f0-2f8e-4f29-b25a-50153d1f00cd" E0214 16:09:13.664551 847761 server.go:669] "stream read failure" err="rpc error: code = Canceled desc = context canceled" E0214 16:09:13.664551 847761 client.go:427] "close response failure" err="EOF" E0214 16:09:13.665131 847761 server.go:705] "CLOSE_RSP to frontend failed" err="rpc error: code = Unavailable desc = transport is closing" serverID="846800f0-2f8e-4f29-b25a-50153d1f00cd" agentID="57a4766a-a392-4c4f-8146-c5a25b0c510f" --- FAIL: TestProxyHandle_ContextCancelled_GRPC (2.45s) leaks.go:78: found unexpected goroutines: [Goroutine 19 in state chan receive, with k8s.io/klog/v2.(*loggingT).flushDaemon on top of the stack: goroutine 19 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x0) /home/andrewsy/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1169 +0x6a created by k8s.io/klog/v2.init.0 /home/andrewsy/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:417 +0xfb Goroutine 31 in state chan receive, with sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*conn).Read on top of the stack: goroutine 31 [chan receive]: sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*conn).Read(0xc000485310, {0xc00025e000, 0x1000, 0x101000000000000}) /home/andrewsy/go/src/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go:74 +0x5d net/http.(*persistConn).Read(0xc00015cea0, {0xc00025e000, 0x40717d, 0x60}) /usr/local/go/src/net/http/transport.go:1926 +0x4e bufio.(*Reader).fill(0xc0004de7e0) /usr/local/go/src/bufio/bufio.go:101 +0x103 bufio.(*Reader).Peek(0xc0004de7e0, 0x1) /usr/local/go/src/bufio/bufio.go:139 +0x5d net/http.(*persistConn).readLoop(0xc00015cea0) /usr/local/go/src/net/http/transport.go:2087 +0x1ac created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:1747 +0x1e05 ] FAIL FAIL sigs.k8s.io/apiserver-network-proxy/tests 2.532s FAIL

With the bug fix:

$ go test ./... -run TestProxyHandle_ContextCancelled_GRPC E0214 16:16:42.596087 848745 client.go:112] "stream read failure" err="rpc error: code = Canceled desc = context canceled" E0214 16:16:42.596265 848745 server.go:390] "Stream read from frontend failure" err="rpc error: code = Canceled desc = context canceled" E0214 16:16:42.596361 848745 server.go:751] "send to client stream failure" err="rpc error: code = Canceled desc = context canceled" serverID="235e6686-f70f-4a1d-a6c6-dc46a811ce01" agentID="a947c418-4784-4b14-8fc9-c4cf0f47834b" connectionID=1 E0214 16:16:42.596473 848745 client.go:371] "could not read stream" err="rpc error: code = Unavailable desc = error reading from server: EOF" E0214 16:16:42.596581 848745 server.go:515] "CLOSE_REQ to Backend failed" err="rpc error: code = Unavailable desc = transport is closing" serverID="235e6686-f70f-4a1d-a6c6-dc46a811ce01" E0214 16:16:42.596624 848745 server.go:669] "stream read failure" err="rpc error: code = Canceled desc = context canceled" E0214 16:16:42.596660 848745 client.go:427] "close response failure" err="EOF" E0214 16:16:42.596694 848745 server.go:705] "CLOSE_RSP to frontend failed" err="rpc error: code = Unavailable desc = transport is closing" serverID="235e6686-f70f-4a1d-a6c6-dc46a811ce01" agentID="a947c418-4784-4b14-8fc9-c4cf0f47834b" --- FAIL: TestProxyHandle_ContextCancelled_GRPC (2.45s) leaks.go:78: found unexpected goroutines: [Goroutine 6 in state chan receive, with k8s.io/klog/v2.(*loggingT).flushDaemon on top of the stack: goroutine 6 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0x0) /home/andrewsy/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1169 +0x6a created by k8s.io/klog/v2.init.0 /home/andrewsy/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:417 +0xfb ] FAIL FAIL sigs.k8s.io/apiserver-network-proxy/tests 2.528s FAIL

Note that the test always fails with 1 goroutine leak coming from klog which is expected.

What if we as soon as this test function starts, or if we can initialize klog early first, then we use IgnoreCurrent to make it ignore klog and keep the test?

It seems super useful this pkg, I'd love to keep it in all tests.

(btw, I wasn't aware of this module, seems quite cool. Thanks!!)

What if we as soon as this test function starts, or if we can initialize klog early first, then we use IgnoreCurrent to make it ignore klog and keep the test?

Good call, let me try this!

And just verifying the test works:

Without defer close(conn.readCh)

$ go test ./... -run TestProxyHandle_ContextCancelled_GRPC E0216 09:32:28.416716 1191438 client.go:103] "stream read failure" err="rpc error: code = Canceled desc = context canceled" E0216 09:32:28.417368 1191438 server.go:751] "send to client stream failure" err="rpc error: code = Unavailable desc = transport is closing" serverID="acbcdbed-f7a1-40b5-a02c-c65186c172a0" agentID="b441b2f6-c9c1-43a9-9f33-f30d1fb1306e" connectionID=1 E0216 09:32:28.417398 1191438 server.go:390] "Stream read from frontend failure" err="rpc error: code = Canceled desc = context canceled" E0216 09:32:28.417907 1191438 client.go:371] "could not read stream" err="rpc error: code = Unavailable desc = error reading from server: read tcp [::1]:39060->[::1]:40799: read: connection reset by peer" E0216 09:32:28.419031 1191438 client.go:427] "close response failure" err="EOF" E0216 09:32:28.419143 1191438 server.go:515] "CLOSE_REQ to Backend failed" err="rpc error: code = Unavailable desc = transport is closing" serverID="acbcdbed-f7a1-40b5-a02c-c65186c172a0" E0216 09:32:28.419955 1191438 server.go:669] "stream read failure" err="rpc error: code = Canceled desc = context canceled" E0216 09:32:28.420246 1191438 server.go:705] "CLOSE_RSP to frontend failed" err="rpc error: code = Unavailable desc = transport is closing" serverID="acbcdbed-f7a1-40b5-a02c-c65186c172a0" agentID="b441b2f6-c9c1-43a9-9f33-f30d1fb1306e" --- FAIL: TestProxyHandle_ContextCancelled_GRPC (2.46s) leaks.go:78: found unexpected goroutines: [Goroutine 100 in state chan receive, with sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*conn).Read on top of the stack: goroutine 100 [chan receive]: sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*conn).Read(0xc00003d450, {0xc0001a7000, 0x1000, 0x101000000000000}) /home/andrewsy/go/src/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go:74 +0x5d net/http.(*persistConn).Read(0xc0004207e0, {0xc0001a7000, 0x40717d, 0x60}) /usr/local/go/src/net/http/transport.go:1926 +0x4e bufio.(*Reader).fill(0xc000099e00) /usr/local/go/src/bufio/bufio.go:101 +0x103 bufio.(*Reader).Peek(0xc000099e00, 0x1) /usr/local/go/src/bufio/bufio.go:139 +0x5d net/http.(*persistConn).readLoop(0xc0004207e0) /usr/local/go/src/net/http/transport.go:2087 +0x1ac created by net/http.(*Transport).dialConn /usr/local/go/src/net/http/transport.go:1747 +0x1e05 ] FAIL FAIL sigs.k8s.io/apiserver-network-proxy/tests 2.549s FAIL

With defer close(conn.readCh):

$ go test ./... -run TestProxyHandle_ContextCancelled_GRPC -count 1 ok sigs.k8s.io/apiserver-network-proxy/tests 2.222s

Awesome, thanks! :)

andrewsykim · 2022-02-14T21:17:37Z

go.mod

@@ -9,7 +9,8 @@ require (
 	github.com/prometheus/client_golang v1.7.1
 	github.com/spf13/cobra v0.0.3
 	github.com/spf13/pflag v1.0.5
-	golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
+	go.uber.org/goleak v1.1.12


Going to remove the go.mod changes later, just adding them temporary to run some tests

I'm fine with keeping it as long as its just for use in our tests.

Signed-off-by: Andrew Sy Kim <[email protected]>

…textDeadlineExceeded_GRPC Signed-off-by: Andrew Sy Kim <[email protected]>

Signed-off-by: Andrew Sy Kim <[email protected]>

… ProxyHandle tests Signed-off-by: Andrew Sy Kim <[email protected]>

rata

Not a maintainer here, but this LGTM, just one minor comment :)

rata · 2022-02-16T14:39:29Z

tests/proxy_test.go

@@ -140,7 +145,9 @@ func TestProxyHandleDialError_GRPC(t *testing.T) {
 	}
 }

-func TestProxyHandle_DoneContext_GRPC(t *testing.T) {
+func TestProxyHandle_ContextDeadlineExceeded_GRPC(t *testing.T) {
+	defer goleak.VerifyNone(t, goleak.IgnoreCurrent())


nit: I'd add a comment saying we need to IgnoreCurrent due to klog leaks

Good point, will add that

k8s-ci-robot · 2022-02-16T14:40:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andrewsykim, rata

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andrewsykim]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rata · 2022-02-16T15:19:35Z

Hmm, the pull-apiserver-network-proxy-test test might be on spot. It panics due to closing an already closed channel, and here we add a close... maybe in some cases that channel is closed already but not in all the cases? 🤔

…log spawns goroutine for flushing logs) Signed-off-by: Andrew Sy Kim <[email protected]>

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim · 2022-02-16T15:38:12Z

I fixed the test -- in the happy path we do actually close the channel properly, so we should only close the readCh if we never saw the close happen

andrewsykim · 2022-02-16T15:41:21Z

I added the goroutine leak to a bunch of other testse in proxy_test.go and looks like we found another one :)

--- FAIL: TestBasicProxy_GRPC (1.50s)
    leaks.go:78: found unexpected goroutines:
        [Goroutine 1123 in state chan receive, with sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*conn).Read on top of the stack:
        goroutine 1123 [chan receive]:
        sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*conn).Read(0xc0054b33b0, {0xc005818000, 0x1000, 0x0})
        	/home/andrewsy/go/src/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go:75 +0xc5
        net/http.(*persistConn).Read(0xc0054ae7e0, {0xc005818000, 0x1000, 0x1000})
        	/usr/local/go/src/net/http/transport.go:1926 +0x110
        bufio.(*Reader).fill(0xc0054d6d20)
        	/usr/local/go/src/bufio/bufio.go:101 +0x294
        bufio.(*Reader).Peek(0xc0054d6d20, 0x1)
        	/usr/local/go/src/bufio/bufio.go:139 +0xcc
        net/http.(*persistConn).readLoop(0xc0054ae7e0)
        	/usr/local/go/src/net/http/transport.go:2087 +0x2db
        created by net/http.(*Transport).dialConn
        	/usr/local/go/src/net/http/transport.go:1747 +0x2ef8
        
         Goroutine 1124 in state select, with net/http.(*persistConn).writeLoop on top of the stack:
        goroutine 1124 [select]:
        net/http.(*persistConn).writeLoop(0xc0054ae7e0)
        	/usr/local/go/src/net/http/transport.go:2386 +0x1a9
        created by net/http.(*Transport).dialConn
        	/usr/local/go/src/net/http/transport.go:1748 +0x2f8b

andrewsykim · 2022-02-16T18:41:51Z

Turns out ^ is not an actual goroutine leak, we just need to ensure the client calls conn.Close by setting req.Close on the http request. This is only for the tests that actually establish a connection.

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim · 2022-02-17T17:51:33Z

@mihivagyok if you have time, do you think you can test this PR to see if you can reproduce hanging streams / memory leaks?

mihivagyok · 2022-02-17T18:05:19Z

@andrewsykim Yes, I could try it tomorrow. Let you know about the results! Thanks!

andrewsykim · 2022-02-18T02:23:09Z

@mihivagyok I realized that this patch requires a change in kube-apiserver so it might be bit difficult to test.

andrewsykim · 2022-02-18T15:51:59Z

konnectivity-client/pkg/client/client.go

@@ -95,7 +95,28 @@ func (t *grpcTunnel) serve(c clientConn) {
 	defer c.Close()

 	for {
-		pkt, err := t.stream.Recv()
+		done := make(chan struct{})


@mihivagyok I added a timeout on the client-side for stream.Recv(), similar to #324 (and based on grpc/grpc-go#1229). Going to test it against a real cluster later today

cheftako · 2022-02-18T19:56:59Z

konnectivity-client/pkg/client/client.go

+			pkt, err := t.stream.Recv()
+			pktChan <- pkt
+			errChan <- err
+			close(done)


This should probably be a defer close(done) at the beginning of the anonymous function.

cheftako · 2022-02-18T19:58:17Z

konnectivity-client/pkg/client/client.go

+		done := make(chan struct{})
+		pktChan := make(chan *client.Packet, 1)
+		errChan := make(chan error, 1)
+		go func() {


Do we really want to spin up a separate goroutine for each packet we want to receive?

It's not ideal but it's a side effect of stream.Recv not accept a context.

I'm on the fence about keeping this change though, I pushed it so I can test it but I haven't seen any evidence yet that it helps with the issue.

andrewsykim · 2022-02-18T21:54:44Z

/assign @cheftako

cheftako · 2022-02-18T22:56:12Z

tests/proxy_test.go

+	if err != nil {
+		t.Error(err)
+	}
+	// client needs to Close the connection immediately to pass goroutine leak test


Could you elaborate? This seems a bit concerning.

Not clear we should be relying on the client to do this sort of things to prevent a leak...

The default behavior by the http client is to re-use connections between requests to the same hosts. But relying on that default behavior leaves us with a goroutine at the end of the test run. Explicitly setting request.Close is as-if we disabled keep alives on the connection which also means the goroutine completes when the test completes.

In other words, setting req.Close = true ensure that conn.Close() gets called, which exercises the read channel closing.

cheftako · 2022-02-18T23:00:36Z

Generally seems like a good fix. Couple of minor nits it would be good to address.

andrewsykim · 2022-02-23T02:29:19Z

/hold

rata · 2022-03-01T15:38:43Z

As I mentioned here, I saw once the apiserver crash with this:

panic: close of closed channel

goroutine 34197 [running]:
k8s.io/kubernetes/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.(*grpcTunnel).serve(0xc01046ade0, {0x53fd8e0, 0xc0103b0a80})
        /home/rodrigo/src/kinvolk/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go:166 +0x9a5
created by k8s.io/kubernetes/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client.CreateSingleUseGrpcTunnel
        /home/rodrigo/src/kinvolk/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go:90 +0x196

This was with this PR and #337 applied. See more details on the comment I linked, but I could not reproduce this, it only happened once. It was not on a "weird dirty state", I had just compiled k8s with #337 added, and manually run a kubectl logs command (not sure if apiserver crashed before I run the run.sh script we use for repro in #276 or not).

Have anyone seen this when testing this PR? PR #337 seems unrelated (but well, was applied, it might make something more easy or just be noise, don't know yet).

Do you see any case where this PR might add a double close on a channel, and therefore cause the apiserver to panic?

I'm a bit concerned as the apiserver to panic'ed and as this change is on konn-client, a follow-up fix requires the customers to update the k8s version, which isn't as easy to update as the proxy-server usually (at least for us). On the other hand I can't reproduce this, so unsure what to propose... If you can't reproduce either and this looks correct to you, I'm fine merging :)

andrewsykim · 2022-03-01T16:16:50Z

I think you're right, I don't think we need this PR anymore because the goroutine leak was only reproducible in the test because the http client in the test was not calling conn.Close(). Will close it for now.

andrewsykim · 2022-03-01T16:16:53Z

/close

k8s-ci-robot · 2022-03-01T16:17:06Z

@andrewsykim: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 14, 2022

k8s-ci-robot requested review from caesarxuchao and dberkov February 14, 2022 21:15

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 14, 2022

andrewsykim commented Feb 14, 2022

View reviewed changes

andrewsykim added 4 commits February 14, 2022 16:51

konnectivity-client: always close read channel on Close()

0851f7b

Signed-off-by: Andrew Sy Kim <[email protected]>

tests: rename TestProxyHandle_DoneContext_GRPC to TestProxyHandle_Con…

8cd017a

…textDeadlineExceeded_GRPC Signed-off-by: Andrew Sy Kim <[email protected]>

tests: add TestProxyHandle_ContextCancelled_GRPC test

c48fcaa

Signed-off-by: Andrew Sy Kim <[email protected]>

temporarily add goleak.VerifyNone, this commit will be removed later

5badf2f

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the fix-connection-leak branch from acdb437 to 5badf2f Compare February 14, 2022 21:52

tests: use goleak.IgnoreCurrent() and call goleak.VerifyNone in other…

b53a792

… ProxyHandle tests Signed-off-by: Andrew Sy Kim <[email protected]>

rata approved these changes Feb 16, 2022

View reviewed changes

andrewsykim added 2 commits February 16, 2022 10:33

tests: add comments for why goleak.IgnoreCurrent is needed (because k…

ecf484b

…log spawns goroutine for flushing logs) Signed-off-by: Andrew Sy Kim <[email protected]>

konnectivity-client: only close c.readCh if CLOSE_RSP was not received

44890f7

Signed-off-by: Andrew Sy Kim <[email protected]>

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 16, 2022

andrewsykim changed the title ~~Clean up all connections from single use grpc tunnnel~~ Ensure read channel is closed for single use grpc tunnnel Feb 16, 2022

tests: set req.Close to ensure client calls conn.Close

e4cb139

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the fix-connection-leak branch from 542dd2e to e4cb139 Compare February 16, 2022 18:49

andrewsykim changed the title ~~Ensure read channel is closed for single use grpc tunnnel~~ konnectivity-client: fix goroutine leaks Feb 18, 2022

andrewsykim commented Feb 18, 2022

View reviewed changes

This was referenced Feb 18, 2022

Leaking/Too many open File descriptors in Konnectivity server with GRPC mode and UDS transport #276

Closed

Konnectivity server leaks memory and free sockets #255

Open

andrewsykim force-pushed the fix-connection-leak branch from 85a1a50 to 851963a Compare February 18, 2022 16:39

cheftako reviewed Feb 18, 2022

View reviewed changes

k8s-ci-robot assigned cheftako Feb 18, 2022

andrewsykim force-pushed the fix-connection-leak branch from 851963a to e4cb139 Compare February 18, 2022 21:57

cheftako reviewed Feb 18, 2022

View reviewed changes

andrewsykim changed the title ~~konnectivity-client: fix goroutine leaks~~ konnectivity-client: ensure connection read channel is closed Feb 19, 2022

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 23, 2022

andrewsykim mentioned this pull request Feb 24, 2022

Add tests for goroutine leaks #333

Closed

k8s-ci-robot closed this Mar 1, 2022

This was referenced Mar 3, 2022

Goroutine leak in konnectivity-client if transport is closed before CLOSE_RSP is received #340

Closed

konnectivity-client: close channels for connections that never received CLOSE_RSP #341

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

konnectivity-client: ensure connection read channel is closed #325

konnectivity-client: ensure connection read channel is closed #325

andrewsykim commented Feb 14, 2022 •

edited

Loading

andrewsykim Feb 14, 2022

rata Feb 16, 2022

andrewsykim Feb 16, 2022

andrewsykim Feb 16, 2022

andrewsykim Feb 16, 2022

rata Feb 16, 2022

andrewsykim Feb 14, 2022

cheftako Feb 18, 2022

rata left a comment

rata Feb 16, 2022

andrewsykim Feb 16, 2022

k8s-ci-robot commented Feb 16, 2022

rata commented Feb 16, 2022 •

edited

Loading

andrewsykim commented Feb 16, 2022

andrewsykim commented Feb 16, 2022

andrewsykim commented Feb 16, 2022 •

edited

Loading

andrewsykim commented Feb 17, 2022

mihivagyok commented Feb 17, 2022

andrewsykim commented Feb 18, 2022

andrewsykim Feb 18, 2022 •

edited

Loading

cheftako Feb 18, 2022

cheftako Feb 18, 2022

andrewsykim Feb 18, 2022

andrewsykim commented Feb 18, 2022

cheftako Feb 18, 2022

cheftako Feb 18, 2022

andrewsykim Feb 19, 2022 •

edited

Loading

cheftako commented Feb 18, 2022

andrewsykim commented Feb 23, 2022

rata commented Mar 1, 2022

andrewsykim commented Mar 1, 2022

andrewsykim commented Mar 1, 2022

k8s-ci-robot commented Mar 1, 2022

konnectivity-client: ensure connection read channel is closed #325

konnectivity-client: ensure connection read channel is closed #325

Conversation

andrewsykim commented Feb 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rata left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 16, 2022

rata commented Feb 16, 2022 • edited Loading

andrewsykim commented Feb 16, 2022

andrewsykim commented Feb 16, 2022

andrewsykim commented Feb 16, 2022 • edited Loading

andrewsykim commented Feb 17, 2022

mihivagyok commented Feb 17, 2022

andrewsykim commented Feb 18, 2022

andrewsykim Feb 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsykim commented Feb 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsykim Feb 19, 2022 • edited Loading

Choose a reason for hiding this comment

cheftako commented Feb 18, 2022

andrewsykim commented Feb 23, 2022

rata commented Mar 1, 2022

andrewsykim commented Mar 1, 2022

andrewsykim commented Mar 1, 2022

k8s-ci-robot commented Mar 1, 2022

andrewsykim commented Feb 14, 2022 •

edited

Loading

rata commented Feb 16, 2022 •

edited

Loading

andrewsykim commented Feb 16, 2022 •

edited

Loading

andrewsykim Feb 18, 2022 •

edited

Loading

andrewsykim Feb 19, 2022 •

edited

Loading