add enable/disable invoker support to old scheduler #5205
Conversation
…d route for invoker
core/invoker/src/main/scala/org/apache/openwhisk/core/invoker/InvokerReactive.scala
Outdated
Show resolved
Hide resolved
| complete("not supported") | ||
| if (healthScheduler.nonEmpty) { | ||
| actorSystem.stop(healthScheduler.get) | ||
| healthScheduler = None |
There was a problem hiding this comment.
JFYI.
While it would be effective, invokers in my downstream are sending a certain message to controllers so that they can make invokers unhealthy immediately to avoid a long-running activation coming to a disabled invoker during the timeout.
But I think this is just a minor tweak.
There was a problem hiding this comment.
Is that on the old scheduler for your downstream or a part of the new scheduler?
If on the old scheduler, I could do that as a follow up since it would require a change to the message bus and logic in the controller to read different health pings off the health topic. But think it's more of an optimization since the controller should recognize it as offline within ten seconds of disabling.
There was a problem hiding this comment.
Since the new scheduler relies on ETCD for health, the logic resides in the old scheduler.
I looked into the downstream code and realize that we just did a workaround to minimize the change by using an instance ID that is less than 0.
I think this would not be an official way but just a workaround.
|
We don't use ansible so I wasn't planning on to here, but I'm sure it could be added easily with a config of how long you want to wait after disabling before stopping the invoker. |
|
I have the same question as Dominic. When you disable an invoker (invoker reactive) what happens with the active acks it’s still sending since it could still be draining activations off the topic. Also is the idea to then take down the invoker? Should it shut down? |
|
The active acks would continue to still be sent back to the controller until there's no activations left to process off of kafka. This is where it's not perfect since you would configure a sleep time between disabling the invoker and then shutting it down. You may also not want to shut down but stop processing on that invoker to debug some issue.
Dominic did bring up a good point that I could add a The new scheduler's invoker enable / disable mechanism has more available insight on whether an invoker is safe to stop because of the scheduler metrics you can get through the http routes it has |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5205 +/- ##
===========================================
+ Coverage 44.67% 72.98% +28.31%
===========================================
Files 238 238
Lines 13937 13957 +20
Branches 581 570 -11
===========================================
+ Hits 6226 10187 +3961
+ Misses 7711 3770 -3941 ☔ View full report in Codecov by Sentry. |
add enable/disable invoker support to old scheduler
Description
Adds enable / disabling of original invoker implementation. Disabling will turn off the scheduler of health pings so the controller will view the invoker as offline. Enabling will reschedule the health pings if it was disabled. Also added a route for both invoker types to check if an invoker is disabled so that the state of a running invoker does not get lost.
Related issue and scope
My changes affect the following components
Types of changes
Checklist: