Skip to content
This repository has been archived by the owner on Jun 19, 2022. It is now read-only.

Add KEDA support for GCP Sources #551

Merged
merged 24 commits into from
Feb 21, 2020
Merged

Conversation

nachocano
Copy link
Member

@nachocano nachocano commented Feb 13, 2020

Fixes #423

Proposed Changes

  • Using annotations to configure autoscaling, as done in serving. The annotation names might change.
  • If you don't specify the keda autoscaling class (currently the only one supported), then the sources will be treated as non-scalable.
  • Separated reconciler for Pullsubscription into two. A keda-based one, which understands about KEDA ScaledObject and checks whether KEDA is installed... And a k8s one, which reconciles a plain deployment.
  • Adding a Resource duck type to be able to track changes to KEDA ScaledObjects. Should be upstreamed to pkg. Also leveraging ListableTracker from eventing, should probably upstream it to pkg as well.
  • Using Unstructured for ScaledObject. As KEDA still doesn't have client-go support for its types, and advises to use the dynamic client, I consider unnecessary to add the dependency for now. Also, it will bring a bunch of things like controller-runtime, etc.

TODOs:

  • Docs (in another PR)
  • E2E tests (in another PR)

Release Note

CloudPubSubSource, CloudSchedulerSource, CloudStorageSource, and CloudAuditLogsSource now support Keda-based scheduling. They can scale all the way down to zero. Note that you will need to have Keda installed. Refer to the installation docs for more details.

@nachocano
Copy link
Member Author

/assign @grantr
/assign @capri-xiyue

@nachocano
Copy link
Member Author

/assign @ericlem

@knative-prow-robot
Copy link
Contributor

@nachocano: GitHub didn't allow me to assign the following users: ericlem.

Note that only google members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @ericlem

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ericlem
Copy link
Contributor

ericlem commented Feb 19, 2020

/assign @ericlem

@nachocano
Copy link
Member Author

/assign @ericlem

@nachocano nachocano changed the title [WIP] Add KEDA support for GCP Sources Add KEDA support for GCP Sources Feb 20, 2020
@nachocano
Copy link
Member Author

/retest

@knative-metrics-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-google-knative-gcp-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/duck/v1alpha1/annotations.go Do not exist 93.2%
pkg/apis/duck/v1alpha1/register.go 0.0% 100.0% 100.0
pkg/apis/duck/v1alpha1/resource_types.go Do not exist 25.0%
pkg/apis/events/v1alpha1/cloudpubsubsource_validation.go 92.3% 88.9% -3.4
pkg/apis/events/v1alpha1/cloudschedulersource_validation.go 92.9% 93.1% 0.2
pkg/apis/events/v1alpha1/cloudstoragesource_defaults.go 75.0% 60.0% -15.0
pkg/apis/events/v1alpha1/cloudstoragesource_validation.go 93.3% 93.5% 0.2
pkg/apis/events/v1alpha1/register.go 0.0% 100.0% 100.0
pkg/apis/pubsub/v1alpha1/pullsubscription_types.go 100.0% 90.9% -9.1
pkg/apis/pubsub/v1alpha1/pullsubscription_validation.go 90.3% 87.5% -2.8
pkg/reconciler/pubsub/pullsubscription/keda/controller.go Do not exist 95.5%
pkg/reconciler/pubsub/pullsubscription/keda/pullsubscription.go Do not exist 63.4%
pkg/reconciler/pubsub/pullsubscription/keda/resources/names.go Do not exist 100.0%
pkg/reconciler/pubsub/pullsubscription/keda/resources/scaled_object.go Do not exist 100.0%
pkg/reconciler/pubsub/pullsubscription/static/controller.go Do not exist 95.2%
pkg/reconciler/pubsub/pullsubscription/static/pullsubscription.go Do not exist 72.7%
pkg/reconciler/pubsub/reconciler.go 71.0% 71.3% 0.3
pkg/reconciler/pubsub/resources/annotations.go Do not exist 100.0%
pkg/reconciler/pubsub/resources/pullsubscription.go 87.5% 85.7% -1.8

@knative-prow-robot
Copy link
Contributor

@nachocano: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-google-knative-gcp-go-coverage 010eb44 link /test pull-google-knative-gcp-go-coverage

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@capri-xiyue
Copy link
Contributor

/lgtm

@capri-xiyue
Copy link
Contributor

/approve

@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: capri-xiyue, nachocano

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [capri-xiyue,nachocano]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@nachocano nachocano mentioned this pull request Feb 21, 2020
@knative-prow-robot knative-prow-robot merged commit 26efd4c into google:master Feb 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add KEDA-scaling for Sources
7 participants