Provide support for preventing interference of autoscaling a single scale target with arbitrary HPAs

[tomkerkhove]

Proposal

If somebody creates an HPA while it is already being autoscaled by an ScaledJob/ScaledObject, we should prevent the HPA from being created by using an admission controller that rejects the HPA.

Alternatively, KEDA can just log a warning so that end-users become aware of it but this does not reject HPA resources and might get lost.

Use-Case

Prevent end-users from having multiple actors scale workloads, either by accident or on purpose, to prevent flapping due to different criteria.

In this scenario, we want to prevent end-users creating arbitrary HPAs when a scale target is already scaled by a ScaledObject/ScaledJob.

Anything else?

No response

[bamboo12366]

I urgently need this one, if no one is working on this can I give a try?

[tomkerkhove]

That would be nice! But we need to close on the approach first, certainly if we introduce an admission controller we need to see where we will shove it in (core or separate).

Thoughts @kedacore/keda-maintainers?

[JorTurFer]

It's a really good idea IMO. Maybe not only for preventing HPA in case of already existing SO and also for preventing SO if there is already a HPA or another SO.

From technical pov, I have never implemented something like this, so I can't say anything, but I'm interested on how to do it

[tomkerkhove]

It's a really good idea IMO. Maybe not only for preventing HPA in case of already existing SO and also for preventing SO if there is already a HPA or another SO.

Already got you covered with #3083 ;)

[bamboo12366]

But I think this two can be covered by checking hpa and workload(i.e: deployment or statefulset) binding

Just check is there existing binding when new request for creating hpa no matter manually or scaleObject create:

• if manually, just report error in client side. User using kubectl or client-go will notice that
• if created by scaleObject, we can log this at scaleObject condition

Do I miss something or not correct?

[tomkerkhove]

If you create an HPA manually and deploy it, there is no way for KEDA to stop it except without using an admission controller since KEDA doesn't manage the HPAs but Kubernetes vanilla does.

[bamboo12366]

If you create an HPA manually and deploy it, there is no way for KEDA to stop it except without using an admission controller since KEDA doesn't manage the HPAs but Kubernetes vanilla does.

True, I mean by implement by using admission controller(which is part of keda components). The stop process is away from keda main logic.

[tomkerkhove]

Yes, but preferrably a SO/SJ is blocked from being created even instead of having a status

[zroubalik]

Admission controllers are always tricky. If we fallback to just reporting this in SO status and emitting events, then I am fine with it.

[zroubalik]

We would need a separate controller for watching HPAs in the cluster.

So I am inclined to make this optional.

[tomkerkhove]

This should definately be optional and configurable through config on operator.

Admission controllers are always tricky. If we fallback to just reporting this in SO status and emitting events, then I am fine with it.

How would you do that without an admission controller for this HPA scenario then? For SO we can do it but why put an SO in failed if you can prevent it from being created already? (either in operator or admission controller)

[zroubalik]

I mean to keep just the SO Status as the base feature and an option to fallback to for people which don't want to (or can't) use admission webhooks.

[bamboo12366]

For the base feature one, in the scaleObejct Controller main logic just to check no other hpa existing in the cluster binding with workload, if exist reporting in status and emit an event right?

[zroubalik]

Yeah, but please make it configurable, so it is not enabled by default.