-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Etcdv3 backend. Send message larger than max. #25745
Comments
Hi @aneroid13, I hope it's not too late but we are currently working to remove the 512 KB limit in the Consul backend: #25856 |
The etcdv3 client has a default request send limit of 2.0 MiB. This change exposes the configuration option to increase that limit enabling larger state using the etcdv3 backend. This also requires that the corresponding --max-request-bytes flag be increased on the server side. The default there is 1.5 MiB. Fixes hashicorp#25745
We are running into this issue as well. I've opened a PR with a potential fix here: When attempting to test my changes I ran into a number of issues with the etcdv3 acceptance tests. I was able to work through a couple of them but I don't know that I can easily resolve the final one involving the locking feature of the backend. |
The etcdv3 client has a default request send limit of 2.0 MiB. This change exposes the configuration option to increase that limit enabling larger state using the etcdv3 backend. This also requires that the corresponding --max-request-bytes flag be increased on the server side. The default there is 1.5 MiB. Fixes hashicorp#25745
The etcdv3 client has a default request send limit of 2.0 MiB. This change exposes the configuration option to increase that limit enabling larger state using the etcdv3 backend. This also requires that the corresponding --max-request-bytes flag be increased on the server side. The default there is 1.5 MiB. Fixes hashicorp#25745
The etcdv3 client has a default request send limit of 2.0 MiB. This change exposes the configuration option to increase that limit enabling larger state using the etcdv3 backend. This also requires that the corresponding --max-request-bytes flag be increased on the server side. The default there is 1.5 MiB. Fixes #25745
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. |
Terraform Version
Terraform Configuration Files
Debug Output
TF_LOG=1 terraform state push errored.tfstate
2020/08/04 13:37:19 [WARN] Invalid log level: "1". Defaulting to level: TRACE. Valid levels are: [TRACE DEBUG INFO WARN ERROR]
2020/08/04 13:37:19 [INFO] Terraform version: 0.12.20
2020/08/04 13:37:19 [INFO] Go runtime version: go1.12.13
2020/08/04 13:37:19 [INFO] CLI args: []string{"/usr/local/bin/terraform", "state", "push", "errored.tfstate"}
2020/08/04 13:37:19 [DEBUG] Attempting to open CLI config file: /home/gitlab-runner/.terraformrc
2020/08/04 13:37:19 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/08/04 13:37:19 [INFO] CLI command args: []string{"state", "push", "errored.tfstate"}
2020/08/04 13:37:19 [TRACE] Meta.Backend: BackendOpts.Config not set, so using settings loaded from init.tf:14,3-19
2020/08/04 13:37:19 [TRACE] Meta.Backend: built configuration for "etcdv3" backend with hash value 1761853381
2020/08/04 13:37:19 [TRACE] Preserving existing state lineage "eddefc0a-07b6-297f-16a2-190281c12128"
2020/08/04 13:37:19 [TRACE] Preserving existing state lineage "eddefc0a-07b6-297f-16a2-190281c12128"
2020/08/04 13:37:19 [TRACE] Meta.Backend: working directory was previously initialized for "etcdv3" backend
2020/08/04 13:37:19 [TRACE] Meta.Backend: using already-initialized, unchanged "etcdv3" backend configuration
2020/08/04 13:37:20 [TRACE] Meta.Backend: instantiated backend of type etcd.Backend
2020/08/04 13:37:20 [DEBUG] checking for provider in "."
2020/08/04 13:37:20 [WARN] found legacy provider "terraform-provider-foreman.log"
2020/08/04 13:37:20 [DEBUG] checking for provider in "/usr/local/bin"
2020/08/04 13:37:20 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2020/08/04 13:37:20 [DEBUG] found valid plugin: "foreman.log", "0.0.0", "/home/gitlab-runner/tmp/tf-allzonesconfig/terraform-provider-foreman.log"
2020/08/04 13:37:20 [DEBUG] checking for provisioner in "."
2020/08/04 13:37:20 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/08/04 13:37:20 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2020/08/04 13:37:20 [TRACE] Meta.Backend: backend *etcd.Backend does not support operations, so wrapping it in a local backend
Failed to persist state: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2800411 vs. 2097152)
Crash Output
Failed to persist state: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2800411 vs. 2097152)
Expected Behavior
Update current etcd state.
Actual Behavior
Throw error:
Failed to persist state: rpc error: code = ResourceExhausted desc = trying to send message larger than max (2800411 vs. 2097152)
Steps to Reproduce
terraform init -get-plugins=false -plugin-dir /data/TF/plugins
terraform workspace select Testzone
terraform plan -input=false -out=plan.tfplan
terraform apply -input=false -auto-approve plan.tfplan
Additional Context
After issue with consul KV max field 512Kb. #24394
I try to switch my backend to etcdv3. And deploy 200 VMs in my test workspace.
VMs deployed successfully, but configuration save failed.
Terraform generate errored.tfstate file and throw error.
My etcd version is:
etcd Version: 3.4.10
Git SHA: 18dfb9cca
Go Version: go1.12.17
Go OS/Arch: linux/amd64
and etcd configuration has option --max-request-bytes 15728640
that increase client request size up to 15Mb
But command "terraform state push errored.tfstate" don't allow to do that either, because of grpc limits:
trying to send message larger than max (2800411 vs. 2097152)
errored.tfstate size is 2,7Mb.
It looks like terraform's grpc client not allow to send more than 2097152 bytes of data.
But may be there another reason.
References
The text was updated successfully, but these errors were encountered: