Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV invalid memory address or nil pointer dereference #2344

Closed
ziouf opened this issue May 23, 2023 · 10 comments · Fixed by #2348
Closed

SIGSEGV invalid memory address or nil pointer dereference #2344

ziouf opened this issue May 23, 2023 · 10 comments · Fixed by #2348
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@ziouf
Copy link

ziouf commented May 23, 2023 •
edited
Loading

Describe the bug

SIGSEGV signal at pod startup

To Reproduce
Steps to reproduce the behavior:

  1. provide all relevant manifests

    • install latest helm chart external-secrets from https://charts.external-secrets.io including CRDs

  2. provide the Kubernetes and ESO version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:42Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl -n external-secrets get pod/external-secrets-67d6c87999-pqrkf -o json | jq '.spec.containers[0].image'
"ghcr.io/external-secrets/external-secrets:v0.8.2"

Expected behavior
Operator pod starts without any error

Additional context
Add any other context about the problem here.

$ kubectl -n external-secrets logs pod/external-secrets-67d6c87999-pqrkf
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x131c2f0]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/aggregated_discovery.go:114
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc0008ffef0, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/aggregated_discovery.go:95 +0x7d0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc00015f6f8, 0x15}, {0xc0008e60a0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/aggregated_discovery.go:49 +0x318
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x98?)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:328 +0x3de
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0x0?)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:203 +0x65
k8s.io/client-go/discovery.ServerGroupsAndResources({0x3ddea48, 0xc000ee9d10})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:413 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:376 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc00095eca8)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:651 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
	/home/runner/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:375 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x3ddea48?, 0xc000ee9d10?})
	/home/runner/go/pkg/mod/k8s.io/[email protected]/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/dynamicrestmapper.go:130
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc0009466c0?, {0x0, 0x0, 0xc14958bdce3be501?})
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc00019fea0?)
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc000edc6c0, {0xc000b3f648, 0x1, 0x0?})
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc00019fea0, 0x0, 0x0, {{0x3dd8318, 0xc00050c100}, 0x0}, 0x1, {0x0, 0x0}, ...})
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/manager.go:351 +0xf9
github.com/external-secrets/external-secrets/cmd.glob..func2(0x62ab120?, {0x373fc37?, 0x2?, 0x2?})
	/home/runner/work/external-secrets/external-secrets/cmd/root.go:132 +0x5fb
github.com/spf13/cobra.(*Command).execute(0x62ab120, {0xc000122070, 0x2, 0x2})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x62ab120)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
github.com/external-secrets/external-secrets/cmd.Execute()
	/home/runner/work/external-secrets/external-secrets/cmd/root.go:224 +0x25
main.main()
	/home/runner/work/external-secrets/external-secrets/main.go:21 +0x1
@ziouf ziouf added the kind/bug Categorizes issue or PR as related to a bug. label May 23, 2023
@gusfcarvalho
Copy link
Member

Hi @ziouf . Can you share which eso CRDs did you create when this issue appeared?

@ziouf
Copy link
Author

ziouf commented May 23, 2023 •
edited
Loading

Hi @ziouf . Can you share which eso CRDs did you create when this issue appeared?

None, the issue appeared at operator startup

This is a very new problem after upgraded k8s from 1.26.x to 1.27.1

It used to work well on 1.26.x with the same setup

@ziouf
Copy link
Author

ziouf commented May 23, 2023

I don't know if it share the same root cause, but I'm having the same issue with ECK operator (elastic/cloud-on-k8s#6848)

Other operators are working as expected

@ziouf
Copy link
Author

ziouf commented May 23, 2023

It looks like the panic appears when the configuration is loaded. I tried to read the code but I'm not experienced enough in go.

@gusfcarvalho
Copy link
Member

Does version 0.8.1 also crashes?

@ziouf
Copy link
Author

ziouf commented May 23, 2023

exact same behaviour with 0.8.1

@moolen
Copy link
Member

moolen commented May 24, 2023

Kubernetes 1.27 isn't supported yet, controller-runtime is blocking us to compile against client-go 1.27

@ziouf
Copy link
Author

ziouf commented May 24, 2023 •
edited
Loading

@moolen controller-runtime v0.15.x has been released some hours ago with support of k8s v1.27 (if I understood well)

https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.15.0

@moolen
Copy link
Member

moolen commented May 24, 2023

That's some great news! We'll upgrade and prep a release asap.

@ziouf
Copy link
Author

ziouf commented May 24, 2023 •
edited
Loading

@moolen maybe this comment could be a quick win : metallb/metallb#1893 (comment)
Upgrading client-go to v0.26.4+ seem to fix the issue kubernetes/kubernetes#116603

And there is a lot af breaking changes with [email protected]

@moolen moolen mentioned this issue May 24, 2023
Merged
@angelprada angelprada mentioned this issue Jul 5, 2023
Closed
@Moran-k Moran-k mentioned this issue Aug 24, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore: update dependencies external-secrets/external-secrets
3 participants
@moolen @ziouf @gusfcarvalho