Google pubsub custom metrics support

[fira42073]

Proposal

As a devops, I want to be able to add custom metrics to my scaledobject like "Oldest unacked message age", which are provided by GCP metrics (stackdriver). The only trigger I can use now is subscriptionSize which is "Unacked message count" in GCP.

Use-Case

I want to scale my deployment if there are a lot of delays in message processing.

Anything else?

No response

[tomkerkhove]

Good suggestion, are you willing to contribute this?

[fira42073]

Yep, but I'll need some support to understand the project structure

[zroubalik]

@Friedrich42 you can check the source code

keda/pkg/scalers/gcp_pub_sub_scaler.go

Line 156 in cabe4b5

func (s *pubsubScaler) GetSubscriptionSize(ctx context.Context) (int64, error) {

guides:
https://github.com/kedacore/keda/blob/main/CREATE-NEW-SCALER.md
https://github.com/kedacore/keda/blob/main/CONTRIBUTING.md
https://github.com/kedacore/keda/blob/main/BUILD.md

or feel free to ask on #keda or #keda-dev Slack channel on https://slack.k8s.io/

[fira42073]

I think that there should be new parameter in metadata object named oldestUnackedMessageAge.

I wonder if it should be optional or required? Making it required seems unreasonable, because scaler object works without it as well, but what should the validation rules be if it will be optional? Should subscriptionSize be also optional, so user can select which metrics he needs for scaling?

Is there another scaler with several metrics, so I can take a look at it?

[zroubalik]

https://keda.sh/docs/2.4/scalers/rabbitmq-queue/ has 2 modes.

In this case we can make it optional, but then validate that user specified at least one of the metrics. Do you think draft here how would the triggers spec look like and we can discuss on that?

[fira42073]

I think this should be ok enough. If one of the target metrics are more than specified, it should trigger scaling.

triggers:
- type: gcp-pubsub
  metadata:
    subscriptionSize: "5" # Optional - Default is 5
    oldestUnackedMessageAge: "1" # Optional - Default is 10 (seconds)
    subscriptionName: "mysubscription" # Required
    credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required

[zroubalik]

@tomkerkhove WDYT?

[zroubalik]

@Friedrich42 or we can allow only one of the metrics to be specified in the spec. And if users would like to scale on both of them, they can define multiple triggers in the ScaledObject.

[tomkerkhove]

I like that proposal, for the rest it looks fine to me

[fira42073]

@zroubalik So I propose we do the same thing as in rabbitmq and we have mode field. Should look like this =>

triggers:
- type: gcp-pubsub
  metadata:
    mode: "SubscriptionSize" # Optional - Default is SubscriptionSize - either SubscriptionSize or OldestUnackedMessageAge
    value: "5" # Optional (active if mode is SubscriptionSize) - Default is 5 for subscriptionsize | (active if mode is OldestUnackedMessageAge) - Default is 10 (seconds)
    subscriptionName: "mysubscription" # Required
    credentialsFromEnv: GOOGLE_APPLICATION_CREDENTIALS_JSON # Required

[tomkerkhove]

This might be fine, but we should at least be backwards compatible and deprecate the old field.

Would you mind updating the feature PR & docs to reflect that please?

[fira42073]

I'll update both PRs soon to add backwards compatibility