Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NPE when switching from LaunchConfig to LaunchTemplate via rollingUpdate #205

Closed
eytan-avisror opened this issue Nov 18, 2020 · 0 comments · Fixed by #207
Closed

NPE when switching from LaunchConfig to LaunchTemplate via rollingUpdate #205

eytan-avisror opened this issue Nov 18, 2020 · 0 comments · Fixed by #207
Labels
bug Something isn't working

Comments

@eytan-avisror
Copy link
Collaborator

Looks like we need more protection around NPE in the case of switching between launchConfig and launchTemplate:

E1117 19:34:34.002394   73731 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 238 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x27e0160, 0x3ac3a60)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x27e0160, 0x3ac3a60)
	/usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969 +0x1b9
github.com/keikoproj/instance-manager/controllers/provisioners/eks.(*EksInstanceGroupContext).NewRollingUpdateRequest(0xc00025c630, 0xd)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/provisioners/eks/upgrade.go:145 +0x28a
github.com/keikoproj/instance-manager/controllers/provisioners/eks.(*EksInstanceGroupContext).UpgradeNodes(0xc00025c630, 0x2aaf406, 0xb)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/provisioners/eks/upgrade.go:55 +0x645
github.com/keikoproj/instance-manager/controllers.HandleReconcileRequest(0x2dba600, 0xc00025c630, 0x0, 0xc000010050)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/interface.go:57 +0x383
github.com/keikoproj/instance-manager/controllers.(*InstanceGroupReconciler).Reconcile(0xc0007382a0, 0xc00080fa20, 0x10, 0xc0008330e0, 0x2c, 0x1, 0xc000b4cd38, 0x10849e8, 0xc00080e1e8)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/instancegroup_controller.go:167 +0x719
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00068e6c0, 0x287bde0, 0xc0000bc0c0, 0x23c4b00)
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00068e6c0, 0x0)
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00068e6c0)
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00072e240)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00072e240, 0x3b9aca00, 0x0, 0x100000002b69601, 0xc00079ca80)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc00072e240, 0x3b9aca00, 0xc00079ca80)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x252a48a]

goroutine 238 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x27e0160, 0x3ac3a60)
	/usr/local/Cellar/go/1.15.4/libexec/src/runtime/panic.go:969 +0x1b9
github.com/keikoproj/instance-manager/controllers/provisioners/eks.(*EksInstanceGroupContext).NewRollingUpdateRequest(0xc00025c630, 0xd)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/provisioners/eks/upgrade.go:145 +0x28a
github.com/keikoproj/instance-manager/controllers/provisioners/eks.(*EksInstanceGroupContext).UpgradeNodes(0xc00025c630, 0x2aaf406, 0xb)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/provisioners/eks/upgrade.go:55 +0x645
github.com/keikoproj/instance-manager/controllers.HandleReconcileRequest(0x2dba600, 0xc00025c630, 0x0, 0xc000010050)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/interface.go:57 +0x383
github.com/keikoproj/instance-manager/controllers.(*InstanceGroupReconciler).Reconcile(0xc0007382a0, 0xc00080fa20, 0x10, 0xc0008330e0, 0x2c, 0x1, 0xc000b4cd38, 0x10849e8, 0xc00080e1e8)
	/Users/eibissror/go/src/github.com/keikoproj/instance-manager/controllers/instancegroup_controller.go:167 +0x719
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00068e6c0, 0x287bde0, 0xc0000bc0c0, 0x23c4b00)
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00068e6c0, 0x0)
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00068e6c0)
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00072e240)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00072e240, 0x3b9aca00, 0x0, 0x100000002b69601, 0xc00079ca80)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc00072e240, 0x3b9aca00, 0xc00079ca80)
	/Users/eibissror/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/Users/eibissror/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:193 +0x32d

This happens here:

if awsprovider.IsUsingMixedInstances(scalingGroup) {
var (
config = aws.StringValue(instance.LaunchTemplate.LaunchTemplateName)

In this case, the scaling group is using MixedInstances (since it switched), however the instances are still using the launchConfig, so launchTemplate is nil.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant