[prometheus-kube-stack] Problem with my own secrets - Error: create: failed to create: Secret "sh.helm.release.v1.prom.v1" is invalid: data: Too long: must have at most 1048576 bytes

[d7volker]

Describe the bug a clear and concise description of what the bug is.

Hi all,

I have a problem I copied my secrects into the kube-prometheus-stack/template directory. I contains some certificates I need for grafana, around 78kb in size. During install helm reported the following error:

Error: create: failed to create: Secret "sh.helm.release.v1.prom.v1" is invalid: data: Too long: must have at most 1048576 bytes
helm.go:84: [debug] Secret "sh.helm.release.v1.prom.v1" is invalid: data: Too long: must have at most 1048576 bytes
create: failed to create
helm.sh/helm/v3/pkg/storage/driver.(*Secrets).Create
        helm.sh/helm/v3/pkg/storage/driver/secrets.go:164
helm.sh/helm/v3/pkg/storage.(*Storage).Create
        helm.sh/helm/v3/pkg/storage/storage.go:69
helm.sh/helm/v3/pkg/action.(*Install).RunWithContext
        helm.sh/helm/v3/pkg/action/install.go:340
main.runInstall
        helm.sh/helm/v3/cmd/helm/install.go:264
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:120
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581

I can say the file itself it not too big. as I can enroll it with kubctl.

Do you have any idea on the issue?

What's your helm version?

version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.6"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

version: 32.2.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm3 upgrade --install prom kube-prometheus-stack -f config/kube-prometheus-stack.yaml --namespace mon --create-namespace --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=2Ti --set prometheus.prometheusSpec.retentio=90d --set prometheus.prometheusSpec.retentionSize=2TB --set alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.resources.requests.storage=10Gi --set alertmanager.alertmanagerSpec.retention=200h --set alertmanager.config.receivers[0].name=pagerDuty --set alertmanager.config.receivers[0].pagerduty_configs[0].service_key=xxx --debug

Anything else we need to know?

No response

[zeritti]

I can suggest that you make sure no files that are not part of the chart are present in the chart directory, e.g. previous packaged charts when developing a chart. Helm will process all files that it finds in there. Using .helmignore to exclude the unwanted files is a way forward.

[d7volker]

Thank you for comment. Meanwhile I have found the issue, my file which contains certain certificates is to big which blows up the configmap data for the whole chart. There is nothing to exclude for me. I have been following a different approach. I exclude Grafana and roll it out in additional chart.

[wanghengheng]

Thank you for comment. Meanwhile I have found the issue, my file which contains certain certificates is to big which blows up the configmap data for the whole chart. There is nothing to exclude for me. I have been following a different approach. I exclude Grafana and roll it out in additional chart.

Are there any other solutions to this problem?

I have the same problem, but I can't split the chart

[inukisoft]

.helmignore is very important. I had two files, values-.yaml and values-.yaml . helm put this files into the secret, but this yamls is very bigs. Solution: create folder "configs" and add this in .helmignore. It works for me.

[brnck]

I have the same problem with 45.XX.XX version. 44.XX.XX is deployed without any issues, but with 45.XX.XX I keep getting:

Error: UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.prometheus.v3000" is invalid: data: Too long: must have at most 1048576 bytes

I am using this command to upgrade/install the chart:

helm upgrade --install prometheus kube-prometheus-stack --wait --timeout 1800s --repo https://prometheus-community.github.io/helm-charts --version 44.4.1 --namespace monitoring --cleanup-on-fail -f prometheus.yaml -f ./prometheus_alerts/common.yaml -f ./prometheus_alerts/kubernetes-mixin/kubernetes-apps.yaml -f ./prometheus_alerts/kubernetes-mixin/kubernetes-storage.yaml -f ./prometheus_alerts/kubernetes-mixin/general.rules.yaml -f ./prometheus_alerts/kubernetes-mixin/prometheus.yaml -f ./prometheus_alerts/product.yaml -f ./prometheus_alerts/test.yaml -f ./prometheus_record_rules/common.yaml -f test.yaml -f ./prometheus_alerts/cilium.yaml

[Tusharsd123]

@brnck were you able to resolve this ? getting same issue with 45.XX.XX helm chart.

[brnck]

Nope. We just stayed at 44.XX.XX and decided to observe the situation in this issue to plan our next steps...

[lowang-bh]

Thank you for comment. Meanwhile I have found the issue, my file which contains certain certificates is to big which blows up the configmap data for the whole chart. There is nothing to exclude for me. I have been following a different approach. I exclude Grafana and roll it out in additional chart.

same problem.Do you use a subchart to solve it? how to split it two subcharts?

[luisra51]

I add the README.md and CONTRIBUTING.md in .helmignore

# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
# helm/charts
OWNERS
hack/
ci/
kube-prometheus-*.tgz
CONTRIBUTING.md
README.md
unittests/

[samjax]

I had this issue. The fix is to make sure the helm charts directory is less than 1MB in size. I moved some contents out of the charts directory to make it 980K. After that it worked fine.