Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage-version-migration job never completes on istio enabled cluster #8441

Closed
AceHack opened this issue Jun 23, 2020 · 7 comments · Fixed by #8442
Closed

storage-version-migration job never completes on istio enabled cluster #8441

AceHack opened this issue Jun 23, 2020 · 7 comments · Fixed by #8442
Labels
area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug. kind/process Changes in how we work

Comments

@AceHack
Copy link
Contributor

AceHack commented Jun 23, 2020

What version of Knative?

0.15.1

Expected Behavior

storage-version-migration job to finish on istio enabled cluster

Actual Behavior

storage-version-migration job never completes on istio enabled cluster

Steps to Reproduce the Problem

Install 0.15.1 on istio enabled cluster with auto-injection enabled

Related

This is the same issue from here
knative/eventing-operator#194

Also, it's related to:
istio/istio#6324

Both
https://github.com/knative/serving/blob/release-0.15/config/post-install/storage-version-migration.yaml
and
https://github.com/knative/serving/blob/release-0.15/config/post-install/default-domain.yaml

Need

    metadata:
      annotations:
        sidecar.istio.io/inject: "false"

I'm happy to make the PR to fix this
@AceHack AceHack added the kind/bug Categorizes issue or PR as related to a bug. label Jun 23, 2020
@knative-prow-robot knative-prow-robot added area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/process Changes in how we work labels Jun 23, 2020
@AceHack AceHack mentioned this issue Jun 23, 2020
Merged
@laballab
Copy link

I'm having this issue as well 😕

@ssandanshi
Copy link

I would appreciate it if this was resolved as well.

@dprotaso
Copy link
Member

Posted in slack but worth mentioning here:

I’m confused - why do our controllers work but not these jobs. For example the storage migration job just hits the K8s API directly.

serving/config/core/deployments/controller.yaml

Lines 27 to 32 in b221eab

metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
labels:
app: controller
serving.knative.dev/release: devel

@tcnghia
Copy link
Contributor

tcnghia commented Jun 23, 2020

@dprotaso Controllers allow Pod restarts, giving time for the sidecar to start up. I think batch jobs are different.

@AceHack
Copy link
Contributor Author

AceHack commented Jun 24, 2020

@dprotaso it's related to this istio issue where istio does not work correctly with k8s jobs
istio/istio#6324

@dprotaso
Copy link
Member

dprotaso commented Jun 24, 2020 •
edited
Loading

that work around mentioned in the issue isn't great :(

@dprotaso
Copy link
Member

thanks for the context

@AceHack AceHack mentioned this issue Jun 24, 2020
Merged
@houshengbo houshengbo mentioned this issue Jun 24, 2020
Closed
@vayyappaneni vayyappaneni mentioned this issue Jul 7, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking area/test-and-release It flags unit/e2e/conformance/perf test issues for product features kind/bug Categorizes issue or PR as related to a bug. kind/process Changes in how we work
Projects
None yet
Milestone
No milestone
6 participants
@dprotaso @tcnghia @AceHack @laballab @ssandanshi @knative-prow-robot