Allow probes to be defined for sidecar containers

[KaranJagtiani]

Issue - Link

This PR allows sidecar containers to have liveness & readiness probes to be defined.

• Before: We wouldn’t not allow liveness & readiness probes to be defined for the sidecar containers.
• Now: We would allow those to be defined, but probe port (either liveness or readiness) can’t be the same as the main container’s port.

The following is an example yaml file that creates a primary container on port 8080 and a sidecar container on port 8081. This type of configuration was not allowed before.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello
spec:
  template:
    metadata:
      creationTimestamp: null
      annotations:
        autoscaling.knative.dev/min-scale: "1"
    spec:
      containerConcurrency: 0
      containers:
        - image: "ghcr.io/knative/helloworld-go:latest"
          name: main-container
          ports:
            - containerPort: 8080
              name: http1
              protocol: TCP
          livenessProbe:
            initialDelaySeconds: 20
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 1
            successThreshold: 1
            httpGet:
              path: /
              port: 8080
          readinessProbe:
            initialDelaySeconds: 2
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 1
            successThreshold: 1
            httpGet:
              path: /
              port: 8080
          resources: {}
        - image: "karanjagtiani/go-server:latest"
          name: sidecar-container
          livenessProbe:
            initialDelaySeconds: 20
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 1
            successThreshold: 1
            httpGet:
              path: /
              port: 8081
          readinessProbe:
            initialDelaySeconds: 2
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 1
            successThreshold: 1
            httpGet:
              path: /
              port: 8081
          resources: {}
      enableServiceLinks: false
      timeoutSeconds: 300
  traffic:
    - latestRevision: true
      percent: 100

Release Note

With the recent update in Knative Revisions validation, liveness and readiness probes can now be defined for sidecar containers.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: KaranJagtiani / name: Karan Jagtiani (613707e, 6857f13, b20e2c3)

[knative-prow]

Welcome @KaranJagtiani! It looks like this is your first PR to knative/serving 🎉

[knative-prow]

Hi @KaranJagtiani. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nak3]

Thank you so much @KaranJagtiani
It is possible to add e2e test? I think you can refer to some of the tests like test/e2e/readinessport_test.go and test/e2e/multicontainer_test.go.

[codecov]

Codecov Report

Patch coverage: 54.83% and project coverage change: -0.07% ⚠️

Comparison is base (ad5455e) 86.16% compared to head (545a282) 86.09%.
Report is 17 commits behind head on main.

❗ Current head 545a282 differs from pull request most recent head b20e2c3. Consider uploading reports for the commit b20e2c3 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #14291      +/-   ##
==========================================
- Coverage   86.16%   86.09%   -0.07%     
==========================================
  Files         195      195              
  Lines       14702    14726      +24     
==========================================
+ Hits        12668    12679      +11     
- Misses       1729     1741      +12     
- Partials      305      306       +1     

Files Changed	Coverage Δ
pkg/apis/serving/k8s_validation.go	92.62% <54.83%> (-1.71%)	⬇️

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[KaranJagtiani]

Thank you for the feedback, @nak3!
I'll look into adding e2e test by referencing the tests you mentioned. I'll update the PR soon with the required tests.

[KaranJagtiani]

@nak3 thank you for promptly resolving the issue I was facing with e2e tests. I have added the e2e test based on your suggestion, also I have kept the beta feature skip check just like in the test/e2e/multicontainer_test.go. If there are any further modifications or additions needed, please let me know. :)

[KaranJagtiani]

/retest

[KaranJagtiani]

/retest

[nak3]

Thank you so much! LGTM other than some format issues.

[nak3]

/lgtm
/approve

I think I don't have a permission to merge though.

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: KaranJagtiani, nak3
Once this PR has been reviewed and has the lgtm label, please assign kvmware for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/apis/OWNERS
• test/OWNERS [nak3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ReToCode]

/assign @dprotaso

[dprotaso]

/hold

Posted some comments here: #8949 (comment)

tl;dr I don't think we can safely expose these as is because then the Pods the activator considers as ready is not what Kubernetes considers ready.

[KaranJagtiani]

@dprotaso I understand the problem, and I would agree that we can't expose these as is because the readiness of the pod wouldn't accurately represent the readiness of all its containers, including the sidecars.

In contemplating a solution, I have gathered that currently the queue-proxy container is set up to manage the probing for the user-container alone, as seen here. This leads me to think that a viable solution might be to extend the functionalities of the queue-proxy to include the probing of all N sidecar containers.

I am eager to know your thoughts on this approach.

[dprotaso]

This leads me to think that a viable solution might be to extend the functionalities of the queue-proxy to include the probing of all N sidecar containers.

Yeah I came to the same conclusion in the comment here #8949 (comment)

Even if we perform such probing if someone specifies an exec probe we'll might want to fallback onto PodReadiness (option 2)

[dprotaso]

Going to close this out as we don't want to merge as is. Further discussion can be done on the issue