io.fabric8.kubernetes.client.informers.impl.cache.Reflector.ReflectorWatcher#reconnecting() { return true;

[chenylh]

Describe the bug

io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager#cannotReconnect（）， always returns false，this will cause the client to keep trying to establish a connection asynchronously

Fabric8 Kubernetes Client version

6.4.1

Steps to reproduce

k8s ApiServer rate limit

Expected behavior

Cancel asynchronous task

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

other (please specify in additional context)

Environment

Linux

Fabric8 Kubernetes Client Logs

2023-07-11 18:00:48.727 DEBUG -- [pool-2-thread-3] .i.AbstractWatchManager : Scheduling reconnect task
2023-07-11 18:00:48.727 DEBUG -- [pool-2-thread-3] ckoffIntervalCalculator : Current reconnect backoff is 32000 milliseconds (T336)
2023-07-11 18:00:49.678 DEBUG -- [138.36:6443/...] i.f.k.c.i.i.c.Reflector : Event received MODIFIED Service resourceVersion v1536952902 for v1/services
2023-07-11 18:00:49.678 DEBUG -- [138.36:6443/...] i.f.k.c.i.i.c.Reflector : Event received MODIFIED Service resourceVersion v1536952902 for v1/services
2023-07-11 18:00:50.082 DEBUG -- [ool-2-thread-35] .i.AbstractWatchManager : Watching https://11.50.138.174:6443/api/v1/services?resourceVersion=152868113&allowWatchBookmarks=true&watch=true...
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Trying to configure client from Kubernetes config...
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Did not find Kubernetes config at: [/root/.kube/config]. Ignoring.
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Trying to configure client from service account...
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Found service account host and port: 10.233.0.1:443
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt}].
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Trying to configure client namespace from Kubernetes service account namespace path...
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] i.f.k.client.Config     : Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2023-07-11 18:00:50.083 DEBUG -- [ool-2-thread-35] atcherWebSocketListener : WebSocket error received
java.util.concurrent.CompletionException: java.io.InterruptedIOException: executor rejected
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
        at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1177)
        at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
        at io.fabric8.kubernetes.client.http.StandardHttpClient.buildWebSocket(StandardHttpClient.java:128)
        at io.fabric8.kubernetes.client.http.StandardWebSocketBuilder.buildAsync(StandardWebSocketBuilder.java:43)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.start(WatchConnectionManager.java:113)
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.startWatch(AbstractWatchManager.java:224)
        at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.reconnect(AbstractWatchManager.java:151)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:735)
        at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:732)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
        at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:732)
        at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.io.InterruptedIOException: executor rejected
        at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:157)

Additional context

No response

[shawkins]

This is against a pretty old version at this point. With later versions anything that you through the client to shutdown the okhttp dispatcher, should cause all asynch tasks to stop.

[chenylh]

This is against a pretty old version at this point. With later versions anything that you through the client to shutdown the okhttp dispatcher, should cause all asynch tasks to stop.

Which version do I need to update to?

[shawkins]

Sorry I was not considering some of these tasks are being scheduled through our static scheduler, which makes them initially unaware that client has been shutdown. At least on master the you've shown here is attempted one more time, the exception logged at a debug level, and the task is never scheduled again.

On the older version do you see repeated attempts at re-establishing the watch?

Ideally we would surpress this debug message or proactively purge the tasks.

There is a related problem - later releases added a client side timeout of informers. That task does appear like it will keep executing even after the client is closed.

[chenylh]

Sorry I was not considering some of these tasks are being scheduled through our static scheduler, which makes them initially unaware that client has been shutdown. At least on master the you've shown here is attempted one more time, the exception logged at a debug level, and the task is never scheduled again.

On the older version do you see repeated attempts at re-establishing the watch?

Ideally we would surpress this debug message or proactively purge the tasks.

There is a related problem - later releases added a client side timeout of informers. That task does appear like it will keep executing even after the client is closed.

yes，I see repeated too many attempts at re-establishing the watch，but k8s api-server is opened rate limit ，
【Current reconnect backoff is 32000 milliseconds (T336)】