karmada-search: Fix lock race affects watch RestChan not close, causing client watch api to hang. #4212

xigang · 2023-11-08T07:23:43Z

What type of PR is this?

/kind bug

What this PR does / why we need it:
When the result channel of watchMux is blocked, it will cause goroutine leak and lock race. this problem affects the client watch api interface to hang when consuming watch.RestChan.

fetch goroutine stack:

curl http://localhost:6062/debug/pprof/goroutine?debug=1

Through the goroutine stack below, there are 1892 goroutines blocked in runtime_SemacquireMutex.

1892 @ 0x439936 0x44a80c 0x44a7e6 0x466ba5 0x1a1a5fd 0x1a1a5de 0x1a1a32a 0x46acc1
#       0x466ba4        sync.runtime_SemacquireMutex+0x24                                                                               /usr/lib/golang/src/runtime/sema.go:71
#       0x1a1a5fc       sync.(*RWMutex).RLock+0x7c                                                                                      /usr/lib/golang/src/sync/rwmutex.go:63
#       0x1a1a5dd       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).startWatchSource.func1+0x5d     /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:229
#       0x1a1a329       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).startWatchSource+0xe9           /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:237

967 @ 0x439936 0x405565 0x40511d 0x1a1a691 0x1a1a32a 0x46acc1
#       0x1a1a690       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).startWatchSource.func1+0x110    /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:235
#       0x1a1a329       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).startWatchSource+0xe9           /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:237

573 @ 0x439936 0x44a80c 0x44a7e6 0x466ba5 0x474fa5 0x476576 0x476555 0x1a1a132 0x18e1a0d 0x18ce325 0x16aadc2 0x18cd8fb 0x79d3cf 0x16acb6c 0x79d3cf 0x1a3cbcb 0x79d3cf 0x18da0a6 0x16aadc2 0x18d9a1b 0x18f6def 0x16ac4a2 0x16976f1 0x19214e6 0x19214b6 0x16be6a5 0x79d3cf 0x16c807d 0x79d3cf 0x16be098 0x79d3cf 0x186a2e4
#       0x466ba4        sync.runtime_SemacquireMutex+0x24                                                                               /usr/lib/golang/src/runtime/sema.go:71
#       0x474fa4        sync.(*Mutex).lockSlow+0x164                                                                                    /usr/lib/golang/src/sync/mutex.go:138
#       0x476575        sync.(*Mutex).Lock+0x35                                                                                         /usr/lib/golang/src/sync/mutex.go:81
#       0x476554        sync.(*RWMutex).Lock+0x14                                                                                       /usr/lib/golang/src/sync/rwmutex.go:111
#       0x1a1a131       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).Stop+0x51                       /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:193
#       0x18e1a0c       k8s.io/apiserver/pkg/endpoints/handlers.serveWatch+0xf8c                                                        /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:139
#       0x18ce324       k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1.2+0x64                                               /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:270
#       0x16aadc1       k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning+0x721                                                  /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:427
#       0x18cd8fa       k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1+0xf1a                                                /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:269
#       0x79d3ce        net/http.HandlerFunc.ServeHTTP+0x2e                                                                             /usr/lib/golang/src/net/http/server.go:2046
#       0x16acb6b       k8s.io/apiserver/pkg/endpoints/metrics.InstrumentHandlerFunc.func1+0x2cb                                        /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:509
#       0x79d3ce        net/http.HandlerFunc.ServeHTTP+0x2e                                                                             /usr/lib/golang/src/net/http/server.go:2046
#       0x1a3cbca       go/src/github.com/karmada/pkg/controllers/proxy.(*Controller).Connect.func1+0x6aa        /go/src/go/src/github.com/karmada/pkg/controllers/proxy/proxy_controller.go:256
#       0x79d3ce        net/http.HandlerFunc.ServeHTTP+0x2e                                                                             /usr/lib/golang/src/net/http/server.go:2046
#       0x18da0a5       k8s.io/apiserver/pkg/endpoints/handlers.ConnectResource.func1.1+0x245                                           /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:224
#       0x16aadc1       k8s.io/apiserver/pkg/endpoints/metrics.RecordLongRunning+0x721                                                  /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:427
#       0x18d9a1a       k8s.io/apiserver/pkg/endpoints/handlers.ConnectResource.func1+0xcba                                             /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:218
#       0x18f6dee       k8s.io/apiserver/pkg/endpoints.restfulConnectResource.func1+0x6e                                                /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1228
#       0x16ac4a1       k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1+0x2e1                                          /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:484
#       0x16976f0       github.com/emicklei/go-restful.(*Container).dispatch+0x9d0                                                      /go/src/go/src/github.com/karmada/vendor/github.com/emicklei/go-restful/container.go:288
#       0x19214e5       github.com/emicklei/go-restful.(*Container).Dispatch+0x645                                                      /go/src/go/src/github.com/karmada/vendor/github.com/emicklei/go-restful/container.go:199
#       0x19214b5       k8s.io/apiserver/pkg/server.director.ServeHTTP+0x615                                                            /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/server/handler.go:146
#       0x16be6a4       k8s.io/apiserver/pkg/endpoints/filterlatency.trackCompleted.func1+0x1a4                                         /go/src/go/src/github.com/karmada/vendor/k8s.io/apiserver/pkg/endpoints/filterlatency/filterlatency.go:103

Refer to cacheWatcher send event to input channel to implement adding a timeout.

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go#L216

	// OK, block sending, but only until timer fires.
	select {
	case c.input <- event:
		return true
	case <-timer.C:
		closeFunc()
		return false
	}

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
@ikaven1024 @RainbowMango @XiShanYongYe-Chang

Does this PR introduce a user-facing change?:

`karmada-search`: Fix lock race affects watch RestChan not close, causing client hanging when performing watch.

codecov-commenter · 2023-11-08T07:41:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (1b2c6ed) 52.78% compared to head (9c9c52c) 52.77%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4212      +/-   ##
==========================================
- Coverage   52.78%   52.77%   -0.01%     
==========================================
  Files         239      239              
  Lines       23584    23585       +1     
==========================================
  Hits        12448    12448              
  Misses      10460    10460              
- Partials      676      677       +1

Flag	Coverage Δ
unittests	`52.77% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pkg/search/proxy/store/util.go	`92.88% <100.00%> (+0.82%)`	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

xigang · 2023-11-08T09:35:52Z

/cc @ikaven1024 @RainbowMango @XiShanYongYe-Chang

XiShanYongYe-Chang · 2023-11-10T07:43:28Z

Thanks @xigang
/assign

RainbowMango · 2023-11-10T08:22:50Z

@xigang Do you know how to reproduce it?

When the result channel of watchMux is blocked, it will cause goroutine leak and lock race.

Since the result channel is no cached, so I can understand that the goroutine will be blocked when trying send event to it:

karmada/pkg/search/proxy/store/util.go

Line 259 in d6e2881

w.result <- copyEvent

Can you remind me why it will cause goroutine leak and lock race?

xigang · 2023-11-10T08:23:01Z

Currently, our federated cluster has 1.8W+ nodes (the size of the nodes will increase). The client uses the NodeInformer ListWatch Nodes resource for a period of time (more than 10 hours will hang. I guess the lock race may have intensified at this time), and the client's watch interface will hang.

xigang · 2023-11-10T08:50:59Z

@xigang Do you know how to reproduce it?

When the result channel of watchMux is blocked, it will cause goroutine leak and lock race.

Since the result channel is no cached, so I can understand that the goroutine will be blocked when trying send event to it:

karmada/pkg/search/proxy/store/util.go

Line 259 in d6e2881

w.result <- copyEvent

Can you remind me why it will cause goroutine leak and lock race?

[client side]
First, we added debug log for the watchHandler of the reflector of client-go and found that the client was stuck w.ResultChan().

	for {
		select {
		case <-stopCh:
			klog.Infof("reflector stopped")
			return errorStopRequested
		case err := <-errc:
			klog.Infof("watch handler received error: %v", err.Error())
			return err
		case event, ok := <-w.ResultChan():
			if !ok {
				klog.Infof("watch server result chan is closed. for loop")
				break loop
			}
			klog.Infof("consume event success")

reproduce log:

[server side]
Then we added debug log for serveWatch and ServeHTTP to WatchServer on the server side, and found that when the watch http2 connection closed, it got stuck while executing defer watcher.Stop().

ServeHTTP:

	for {
		select {
		case <-done:
			klog.Infof("client %v watch server shutdown.", userAgent)
			return
		case <-timeoutCh:
			klog.Infof("client %v watch server timeout.", userAgent)
			return
		case event, ok := <-ch:
			if !ok {
				// End of results.
				klog.Infof("client %v watch server closed.", userAgent)
				return
			}

serveWatch:

// serveWatch will serve a watch response.
// TODO: the functionality in this method and in WatchServer.Serve is not cleanly decoupled.
func serveWatch(watcher watch.Interface, scope *RequestScope, mediaTypeOptions negotiation.MediaTypeOptions, req *http.Request, w http.ResponseWriter, timeout time.Duration) {
	klog.Infof("watcher %v connecting", req.UserAgent())
	defer func(){
		klog.Infof("watcher %v stop done in serveWatch", req.UserAgent()) //No execution here
	}()
	defer watcher.Stop() //code block here
	defer func(){
		klog.Infof("watcher %v http2 closed.", req.UserAgent()) //The code is executed here
	}()

reproduce log:

Finally, through the goroutine profile, it was found that a large number of goroutines were stuck and locked runtime_SemacquireMutex.

1892 @ 0x439936 0x44a80c 0x44a7e6 0x466ba5 0x1a1a5fd 0x1a1a5de 0x1a1a32a 0x46acc1
#       0x466ba4        sync.runtime_SemacquireMutex+0x24                                                                               /usr/lib/golang/src/runtime/sema.go:71
#       0x1a1a5fc       sync.(*RWMutex).RLock+0x7c                                                                                      /usr/lib/golang/src/sync/rwmutex.go:63
#       0x1a1a5dd       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).startWatchSource.func1+0x5d     /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:229
#       0x1a1a329       go/src/github.com/karmada/pkg/search/proxy/store.(*watchMux).startWatchSource+0xe9           /go/src/go/src/github.com/karmada/pkg/search/proxy/store/util.go:237

ikaven1024 · 2023-11-10T11:46:43Z

Why dose it block at fourth renow, not before?

xigang · 2023-11-10T11:57:09Z

Why dose it block at fourth renow, not before?

@ikaven1024 In the log, the hang appears not for the fourth watch, but for the Nth time. Because the server will regularly close the watch api connection, the client will renew the watch. This prevents the client from always watching a single kube-apiserver.

ikaven1024 · 2023-11-10T12:37:08Z

The leak come here:

watchMux receives an event fron Cacher, locked and enter to the default case, waiting for sending to result chan.

karmada/pkg/search/proxy/store/util.go

Lines 252 to 262 in 67351f4

    
           	func() { 
        
           		w.lock.RLock() 
        
           		defer w.lock.RUnlock() 
        
           		select { 
        
           		case <-w.done: 
        
           			return 
        
           		default: 
        
           			w.result <- copyEvent 
        
           		} 
        
           	}() 
        
           }

While client not receive this event and close the watch, calling

karmada/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go

Lines 66 to 67 in 3f5c907

    
           func serveWatch(watcher watch.Interface, scope *RequestScope, mediaTypeOptions negotiation.MediaTypeOptions, req *http.Request, w http.ResponseWriter, timeout time.Duration) { 
        
           	defer watcher.Stop()

In Close, alse want lock. So dead lock occur.

karmada/pkg/search/proxy/store/util.go

Lines 209 to 225 in 67351f4

    
           func (w *watchMux) Stop() { 
        
           	select { 
        
           	case <-w.done: 
        
           		return 
        
           	default: 
        
           	} 
        
           	w.lock.Lock() 
        
           	defer w.lock.Unlock() 
        
           	select { 
        
           	case <-w.done: 
        
           	default: 
        
           		close(w.done) 
        
           		close(w.result) 
        
           	} 
        
           }

This can be tested with:

func TestName(t *testing.T) {
	// watcher from cacher
	cacheW := watch.NewFakeWithChanSize(1, false)

	// client start watch, and not receive the event
	clientW := newWatchMux()
	clientW.AddSource(cacheW, func(event watch.Event) {})
	clientW.Start()

	// receive an object from cacher.
	// while client does not consume it.
	cacheW.Add(&corev1.Pod{})

	time.Sleep(time.Second)
	// client close.
	clientW.Stop()
	// DEADLOCK!!!
}

ikaven1024 · 2023-11-10T12:40:52Z

Does below code work,without adding a timeout. @xigang

		func() {
			// w.lock.RLock()
			// defer w.lock.RUnlock()
			select {
			case <-w.done:
				return
			case w.result <- copyEvent:
			}
		}()

xigang · 2023-11-11T02:13:09Z

@ikaven1024 Thanks for the analysis. Is there any problem in remove RLock when multiple goroutines process w.done and w.result?

If RLock is retained. Is it better to use timeout to reduce lock race?

ikaven1024 · 2023-11-11T03:05:01Z

Is there any problem in remove RLock when multiple goroutines process w.done and w.result?

It's safe reading, writing and closing a chan in multiple goroutines.

If RLock is retained. Is it better to use timeout to reduce lock race?

It will cause client break the watch, and send a new watch request, Increasing the burden on the server.

xigang · 2023-11-11T03:26:19Z

Is there any problem in remove RLock when multiple goroutines process w.done and w.result?

It's safe reading, writing and closing a chan in multiple goroutines.

If RLock is retained. Is it better to use timeout to reduce lock race?

It will cause client break the watch, and send a new watch request, Increasing the burden on the server.

@ikaven1024 I think it's ok, I resubmitted the code. To avoid the exception, I added additional recover() for startWatchSource.

ikaven1024 · 2023-11-11T04:51:26Z

pkg/search/proxy/store/util.go

 			select {
 			case <-w.done:
 				return
-			default:
-				w.result <- copyEvent
+			case w.result <- copyEvent:


Simplify it:

--- a/pkg/search/proxy/store/util.go +++ b/pkg/search/proxy/store/util.go @@ -246,19 +246,8 @@ func (w *watchMux) startWatchSource(source watch.Interface, decorator func(watch select { case <-w.done: return - default: + case w.result <- copyEvent: } - - func() { - w.lock.RLock() - defer w.lock.RUnlock() - select { - case <-w.done: - return - default: - w.result <- copyEvent - } - }() } }

@ikaven1024 code updated. Ask for help, the unit test did not pass, and I couldn't find what went wrong.😄

Which test fails, would you show the error message?

ikaven1024 · 2023-11-11T04:52:02Z

pkg/search/proxy/store/util.go

@@ -225,6 +225,7 @@ func (w *watchMux) Stop() {
 }

 func (w *watchMux) startWatchSource(source watch.Interface, decorator func(watch.Event)) {
+	defer func() { _ = recover() }()


add log when panic?

Yes, This avoids writing panic after w.result is closed.

This recover() suppresses panic forever, can you explain why we need to do that?

This avoids writing panic after w.result is closed.

Can we eliminate this risk?

My concern about the recover() is it just suppresses issues, might bring debug effort.

Yes, This avoids writing panic after w.result is closed.

I mean we should print log when panic.

@RainbowMango The reason for adding recover() is that I am worried that multiple gourinue write the w.result channel at the same time. When one goroutine close w.result, other goroutines continue to write w.result, triggering panic.

Write to a closed channel won't panic in select case, like https://github.com/karmada-io/karmada/pull/4212/files#diff-688048f0ec4cf2d213f3bd0437bde2531b124c71d172319b20f25cd140fc238cR255.

RainbowMango · 2023-11-13T03:10:02Z

@XiShanYongYe-Chang Yes, @ikaven1024 open a new PR #4221 to fix this bug.

We usually don't open another PR to replace the current in-review PR. I don't know which PR we should move forward with.

@xigang What do you think?

xigang · 2023-11-13T03:18:12Z

@XiShanYongYe-Chang Yes, @ikaven1024 open a new PR #4221 to fix this bug.

We usually don't open another PR to replace the current in-review PR. I don't know which PR we should move forward with.

@xigang What do you think?

@RainbowMango Agreed, in order to track this issue, let's continue to fix this issue in this PR.

RainbowMango · 2023-11-13T03:40:40Z

Yes, we can focus on fixing this issue and keep the minimum change( so that we can easily backport this to release branches), and let #4221 focus on additional improvement.

Does that make sense to you? @xigang @ikaven1024

xigang · 2023-11-13T04:11:20Z

Yes, we can focus on fixing this issue and keep the minimum change( so that we can easily backport this to release branches), and let #4221 focus on additional improvement.

Does that make sense to you? @xigang @ikaven1024

@ikaven1024 Migrate #4221 code to this PR? Or use timeout to temporarily fix this issue?

ikaven1024 · 2023-11-13T04:16:16Z

@ikaven1024 Migrate #4221 code to this PR? Or use timeout to temporarily fix this issue?

You can migrate #4221 here, and i will close it.

xigang · 2023-11-13T04:18:50Z

@ikaven1024 Migrate #4221 code to this PR? Or use timeout to temporarily fix this issue?

You can migrate #4221 here, and i will close it.

Done.

RainbowMango · 2023-11-13T06:20:47Z

pkg/search/proxy/store/util.go

 	for _, source := range w.sources {
-		go w.startWatchSource(source.watcher, source.decorator)
+		source := source


Reports declarations of variables that overlap with the declarations in the outer scope.

A warning from Goland IDE.

If you want to pin the variable, we can give it another name, or:

diff --git a/pkg/search/proxy/store/util.go b/pkg/search/proxy/store/util.go index 00f93c467..dbb765959 100644 --- a/pkg/search/proxy/store/util.go +++ b/pkg/search/proxy/store/util.go @@ -196,8 +196,8 @@ func (w *watchMux) AddSource(watcher watch.Interface, decorator func(watch.Event // Start run the watcher func (w *watchMux) Start() { wg := sync.WaitGroup{} - for _, source := range w.sources { - source := source + for i := range w.sources { + source := w.sources[i] wg.Add(1) go func() { defer wg.Done()

@RainbowMango Fixed. please review again.

…ng client watch api to hang Signed-off-by: xigang <[email protected]>

RainbowMango

/lgtm

/assign @ikaven1024
For the final decision.

ikaven1024 · 2023-11-13T07:44:28Z

/lgtm
/approve

karmada-bot · 2023-11-13T07:44:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ikaven1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/search/OWNERS~~ [ikaven1024]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

XiShanYongYe-Chang · 2023-11-14T07:52:34Z

Hi @xigang do you think we need to cherry-pick this patch to the previous branch?

xigang · 2023-11-14T13:59:01Z

cherry-pick

@XiShanYongYe-Chang I think we should cherry-pick the previous branch and let me handle it.

XiShanYongYe-Chang · 2023-11-15T01:24:35Z

@XiShanYongYe-Chang I think we should cherry-pick the previous branch and let me handle it.

Thanks~

cherry pick of #4212: Fix lock race affects watch RestChan not close, causing client watch api to hang.

karmada-bot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 8, 2023

karmada-bot requested review from huntsman-li and ikaven1024 November 8, 2023 07:23

karmada-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 8, 2023

xigang force-pushed the fix_watch_api_hang branch from 376ff1b to 40922ea Compare November 8, 2023 08:15

karmada-bot requested review from RainbowMango and XiShanYongYe-Chang November 8, 2023 09:35

karmada-bot assigned XiShanYongYe-Chang Nov 10, 2023

xigang force-pushed the fix_watch_api_hang branch from 40922ea to 4eadcdf Compare November 11, 2023 03:20

karmada-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 11, 2023

xigang force-pushed the fix_watch_api_hang branch from 4eadcdf to 76ffd32 Compare November 11, 2023 03:23

karmada-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 11, 2023

xigang force-pushed the fix_watch_api_hang branch from 76ffd32 to cb289a2 Compare November 11, 2023 03:39

ikaven1024 reviewed Nov 11, 2023

View reviewed changes

xigang force-pushed the fix_watch_api_hang branch from cb289a2 to 249d571 Compare November 11, 2023 04:53

xigang force-pushed the fix_watch_api_hang branch from f5fb699 to 640f6cc Compare November 13, 2023 04:16

RainbowMango reviewed Nov 13, 2023

View reviewed changes

karmada-search: Fix lock race affects watch RestChan not close, causi…

9c9c52c

…ng client watch api to hang Signed-off-by: xigang <[email protected]>

xigang force-pushed the fix_watch_api_hang branch from 640f6cc to 9c9c52c Compare November 13, 2023 06:39

karmada-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 13, 2023

RainbowMango reviewed Nov 13, 2023

View reviewed changes

karmada-bot assigned ikaven1024 and RainbowMango Nov 13, 2023

karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Nov 13, 2023

karmada-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 13, 2023

karmada-bot merged commit 67e45cd into karmada-io:master Nov 13, 2023
12 checks passed

karmada-bot added a commit that referenced this pull request Nov 15, 2023

Merge pull request #4239 from xigang/release-1.7

6b5b3f7

cherry pick of #4212: Fix lock race affects watch RestChan not close, causing client watch api to hang.

karmada-bot added a commit that referenced this pull request Nov 15, 2023

Merge pull request #4241 from xigang/release-1.6

7a25a9a

cherry pick of #4212: Fix lock race affects watch RestChan not close, causing client watch api to hang.

karmada-bot added a commit that referenced this pull request Nov 15, 2023

Merge pull request #4242 from xigang/release-1.5

bc59377

cherry pick of #4212: Fix lock race affects watch RestChan not close, causing client watch api to hang.

karmada-search: Fix lock race affects watch RestChan not close, causing client watch api to hang. #4212

karmada-search: Fix lock race affects watch RestChan not close, causing client watch api to hang. #4212

Conversation

xigang commented Nov 8, 2023 • edited by RainbowMango Loading

codecov-commenter commented Nov 8, 2023 • edited Loading

Codecov Report

xigang commented Nov 8, 2023

XiShanYongYe-Chang commented Nov 10, 2023

RainbowMango commented Nov 10, 2023

xigang commented Nov 10, 2023 • edited Loading

xigang commented Nov 10, 2023 • edited Loading

ikaven1024 commented Nov 10, 2023

xigang commented Nov 10, 2023

ikaven1024 commented Nov 10, 2023

ikaven1024 commented Nov 10, 2023

xigang commented Nov 11, 2023 • edited Loading

ikaven1024 commented Nov 11, 2023

xigang commented Nov 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RainbowMango commented Nov 13, 2023

xigang commented Nov 13, 2023

RainbowMango commented Nov 13, 2023

xigang commented Nov 13, 2023 • edited Loading

ikaven1024 commented Nov 13, 2023

xigang commented Nov 13, 2023

Choose a reason for hiding this comment

xigang Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

RainbowMango left a comment

Choose a reason for hiding this comment

ikaven1024 commented Nov 13, 2023

karmada-bot commented Nov 13, 2023

XiShanYongYe-Chang commented Nov 14, 2023

xigang commented Nov 14, 2023

XiShanYongYe-Chang commented Nov 15, 2023

xigang commented Nov 8, 2023 •

edited by RainbowMango

Loading

codecov-commenter commented Nov 8, 2023 •

edited

Loading

xigang commented Nov 10, 2023 •

edited

Loading

xigang commented Nov 10, 2023 •

edited

Loading

xigang commented Nov 11, 2023 •

edited

Loading

xigang commented Nov 11, 2023 •

edited

Loading

xigang commented Nov 13, 2023 •

edited

Loading

xigang Nov 13, 2023 •

edited

Loading