Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

invalid memory address or nil pointer dereference in getReferencedRolloutAnalyses() #1248

Closed
jessesuen opened this issue Jun 3, 2021 · 1 comment · Fixed by #1249
Closed
Assignees
Labels
bug Something isn't working cherry-pick/release-1.0
Milestone

Comments

@jessesuen
Copy link
Member

Summary

v1.0.1 can panic with:

time="2021-06-02T07:22:31Z" level=error msg="Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 387 [running]:
runtime/debug.Stack(0xc008507200, 0x1c18d40, 0x2d971f0)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1.1.1(0xc007c6aee0, 0xc008507b30)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:149 +0x5b
panic(0x1c18d40, 0x2d971f0)
    /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/argoproj/argo-rollouts/rollout.(*rolloutContext).getReferencedRolloutAnalyses(0xc008cac380, 0xc008df07f8, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:657 +0x36a
github.com/argoproj/argo-rollouts/rollout.(*rolloutContext).getRolloutReferencedResources(0xc008cac380, 0x2dea7c8, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:511 +0x9d
github.com/argoproj/argo-rollouts/rollout.(*rolloutContext).getRolloutValidationErrors(0xc008cac380, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:468 +0x6c
github.com/argoproj/argo-rollouts/rollout.(*rolloutContext).reconcile(0xc008cac380, 0xc0086c0000, 0xc008cac380)
    /go/src/github.com/argoproj/argo-rollouts/rollout/context.go:53 +0x2f
github.com/argoproj/argo-rollouts/rollout.(*Controller).syncHandler(0xc0007f2000, 0xc0068a32c0, 0x47, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:387 +0x51a
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1.1(0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:153 +0x7c
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem.func1(0x21621d0, 0xc000617e60, 0x1e8bdd7, 0x7, 0xc000d1be60, 0xc00098ef40, 0x1b8c1a0, 0xc0068f82b0, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:157 +0x323
github.com/argoproj/argo-rollouts/utils/controller.processNextWorkItem(0x21621d0, 0xc000617e60, 0x1e8bdd7, 0x7, 0xc000d1be60, 0xc00098ef40, 0x1)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:169 +0x9a
github.com/argoproj/argo-rollouts/utils/controller.RunWorker(...)
    /go/src/github.com/argoproj/argo-rollouts/utils/controller/controller.go:104
github.com/argoproj/argo-rollouts/rollout.(*Controller).Run.func1()
    /go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:319 +0xa5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc003060270)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003060270, 0x2107e40, 0xc006846db0, 0x1, 0xc0000b8c60)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003060270, 0x3b9aca00, 0x0, 0x1, 0xc0000b8c60)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc003060270, 0x3b9aca00, 0xc0000b8c60)
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by github.com/argoproj/argo-rollouts/rollout.(*Controller).Run
    /go/src/github.com/argoproj/argo-rollouts/rollout/controller.go:318 +0xac
" namespace=cloud-ops-vitefatjar2-usw2-e2e rollout=cloud-ops-rollout-vite-fat-jar2-e2e-usw2

Diagnostics

What version of Argo Rollouts are you running? v1.0.1

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME>

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

@jessesuen jessesuen added the bug Something isn't working label Jun 3, 2021
@jessesuen
Copy link
Member Author

Offending code is recently modified analysis validation:

	} else if c.rollout.Spec.Strategy.Canary != nil {
		canary := c.rollout.Spec.Strategy.Canary
		for i, step := range canary.Steps {
			if step.Analysis != nil {
				templates, err := c.getReferencedAnalysisTemplates(c.rollout, step.Analysis, validation.InlineAnalysis, i)
				templates.Args = step.Analysis.Args   // <<<<< nil pointer dereference
				if err != nil {
					return nil, err
				}
				analysisTemplates = append(analysisTemplates, *templates)
			}
		}

@khhirani can you take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working cherry-pick/release-1.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants