Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Missing CRD - GrafanaAlertRuleGroups causes operator to fail to start #1463

Closed
MarkCupitt opened this issue Mar 19, 2024 · 9 comments
Closed

[Bug] Missing CRD - GrafanaAlertRuleGroups causes operator to fail to start #1463

MarkCupitt opened this issue Mar 19, 2024 · 9 comments
Labels
bug Something isn't working triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@MarkCupitt
Copy link

MarkCupitt commented Mar 19, 2024 •
edited
Loading

Describe the bug
Missing CRD GrafanaAlertRuleGroups causes operator to fail to start, caused multiple restarts

024-03-19T01:52:04Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "GrafanaAlertRuleGroup.grafana.integreatly.org", "error": "failed to get restmapping: no matches for kind \"GrafanaAlertRuleGroup\" in group \"grafana.integreatly.org\""}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
	sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:63
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	k8s.io/[email protected]/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	k8s.io/[email protected]/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	k8s.io/[email protected]/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
	sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56

Version
5.7.0

Expected behavior
Operator should start with no errors

Suspect component/Location where the bug might be occurring
kustomize-namespace_scoped.yaml

FIX

I added the CRD from

grafana-operator/config/grafana.integreatly.org_grafanaalertrulegroups.yaml

Line 4 in 33fa7f9

metadata:

and it seemed to start ok

2024-03-19T01:53:35Z	INFO	setup	operator running in namespace scoped mode	{"namespace": "grafana"}
2024-03-19T01:53:35Z	INFO	setup	starting manager
2024-03-19T01:53:35Z	INFO	controller-runtime.metrics	Starting metrics server
2024-03-19T01:53:35Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": "0.0.0.0:9090", "secure": false}
2024-03-19T01:53:35Z	INFO	starting server	{"kind": "health probe", "addr": "[::]:8081"}
I0319 01:53:35.313242       1 leaderelection.go:250] attempting to acquire leader lease grafana/f75f3bba.integreatly.org...
2024-03-19T01:53:45Z	INFO	GrafanaFolderReconciler	folder sync complete
2024-03-19T01:53:45Z	INFO	GrafanaDatasourceReconciler	datasources sync complete
2024-03-19T01:53:45Z	INFO	GrafanaDashboardReconciler	dashboard sync complete
I0319 01:53:50.377750       1 leaderelection.go:260] successfully acquired lease grafana/f75f3bba.integreatly.org
2024-03-19T01:53:50Z	DEBUG	events	grafana-operator-controller-manager-94b9dbbfb-9znvq_d4046823-fb22-4438-b1fe-41dbe96eb84b became leader	{"type": "Normal", "object": {"kind":"Lease","namespace":"grafana","name":"f75f3bba.integreatly.org","uid":"db8814f8-b900-47ff-bd66-06310921f2e4","apiVersion":"coordination.k8s.io/v1","resourceVersion":"129337701"}, "reason": "LeaderElection"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "source": "kind source: *v1beta1.Grafana"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "source": "kind source: *v1beta1.GrafanaDatasource"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "source": "kind source: *v1beta1.GrafanaDashboard"}
2024-03-19T01:53:50Z	INFO	Starting Controller	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "source": "kind source: *v1.Deployment"}
2024-03-19T01:53:50Z	INFO	Starting Controller	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "source": "kind source: *v1.ConfigMap"}
2024-03-19T01:53:50Z	INFO	Starting Controller	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup", "source": "kind source: *v1beta1.GrafanaAlertRuleGroup"}
2024-03-19T01:53:50Z	INFO	Starting Controller	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup"}
2024-03-19T01:53:50Z	INFO	Starting EventSource	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "source": "kind source: *v1beta1.GrafanaFolder"}
2024-03-19T01:53:50Z	INFO	Starting Controller	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder"}
2024-03-19T01:53:50Z	INFO	Starting workers	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "worker count": 1}
2024-03-19T01:53:50Z	INFO	Starting workers	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "worker count": 1}
2024-03-19T01:53:50Z	INFO	Starting workers	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup", "worker count": 1}
2024-03-19T01:53:50Z	INFO	Starting workers	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "worker count": 1}
2024-03-19T01:53:50Z	INFO	Starting workers	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "worker count": 1}
@MarkCupitt MarkCupitt added bug Something isn't working needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 19, 2024
@MarkCupitt MarkCupitt changed the title [Bug] MIssing GrafanaAlertRuleGroups causes operator to fail to start [Bug] Missing CRD - GrafanaAlertRuleGroups causes operator to fail to start Mar 19, 2024
@NissesSenap
Copy link
Collaborator

@MarkCupitt can you explain exactly how installed the operator so we can reproduce the error?

@MarkCupitt
Copy link
Author

MarkCupitt commented Mar 19, 2024 •
edited
Loading

@NissesSenap kustomize - namespace_scoped, except I download this yaml manually

https://github.com/grafana/grafana-operator/releases/download/v5.7.0/kustomize-namespace_scoped.yaml

The crd is missing from that file

@Leo1003
Copy link

Leo1003 commented Mar 19, 2024

Same here, we are using FluxCD to install Grafana Operator from oci://ghcr.io/grafana/grafana-operator:v5.7.0

grafana-operator-controller-manager keeps entering CrashLoopBackOff state since it cannot find GrafanaAlertRuleGroups CRD on the cluster

@pb82
Copy link
Collaborator

pb82 commented Mar 19, 2024

Alerting CRDs are missing here and make all doesn't include them. Looking into it now.

@theSuess theSuess added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 19, 2024
@theSuess
Copy link
Member

The fix for this is already in the works in #1464

@NissesSenap
Copy link
Collaborator

This should now be solved as a part of https://github.com/grafana/grafana-operator/releases/tag/v5.8.0
Thanks for reporting this issue @MarkCupitt

@peytonyip
Copy link

After updating to 5.8.1(or even 5.8.0 ), the issue still persists. The update command used was: helm upgrade -i grafana-operator oci://ghcr.io/grafana/helm-charts/grafana-operator --version v5.8.1
the log show:

2024-04-05T06:55:04Z	ERROR	controller-runtime.source.EventHandler	if kind is a CRD, it should be installed before calling Start	{"kind": "GrafanaAlertRuleGroup.grafana.integreatly.org", "error": "no matches for kind \"GrafanaAlertRuleGroup\" in version \"grafana.integreatly.org/v1beta1\""}
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1
	sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:63
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2
	k8s.io/[email protected]/pkg/util/wait/loop.go:87
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext
	k8s.io/[email protected]/pkg/util/wait/loop.go:88
k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel
	k8s.io/[email protected]/pkg/util/wait/poll.go:33
sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1
	sigs.k8s.io/[email protected]/pkg/internal/source/kind.go:56

@theSuess
Copy link
Member

theSuess commented Apr 5, 2024 •
edited
Loading

@peytonyip this issue specifically references kustomize as the installation method.

The issue you are facing is an upstream helm bug/missing feature (helm/helm#11969, helm/helm#6581, helm/helm#12362).

A workaround for this is to apply the CRDs manually. We publish the CRDs directly so you can use this command to upgrade them:

kubectl apply -f https://github.com/grafana/grafana-operator/releases/download/v5.8.1/crds.yaml

(you might need to also add --server-side and --force-conflicts for this to work)

@peytonyip
Copy link

@theSuess Thanks！this resolved my issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@theSuess @pb82 @MarkCupitt @NissesSenap @Leo1003 @peytonyip