Vault Fails to Start with Large Number of Tokens #3772
Milestone
Comments
|
We can't generally paginate because this is not something supported by all storage backends. It would have to be something particular to the etcd backend. Ping @xiang90 ! |
|
options for this on the etcd client side have been added to etcd in etcd-io/etcd#9047 - vault's etcd backend code needs to use this to increase max recv size, possibly based on config/environment/etc? |
|
yea. we should bump the etcd client, and set the response size to unlimited by default. we can potentially add an option to change the size. would you like to help on that? |
|
I don't know how to bump the etcd client version that's used (how does Vault's vendoring work?), but I think the following should work if the client is updated: diff --git a/physical/etcd/etcd3.go b/physical/etcd/etcd3.go
index 04944e59..03af89dc 100644
--- a/physical/etcd/etcd3.go
+++ b/physical/etcd/etcd3.go
@@ -108,6 +108,15 @@ func newEtcd3Backend(conf map[string]string, logger log.Logger) (physical.Backen
cfg.Password = password
}
+ if maxReceive, ok := conf["max_receive_size"]; ok {
+ // grpc converts this to uint32 internally, so parse as that to avoid passing invalid values
+ val, err := strconv.ParseUint(maxReceive, 10, 32)
+ if err != nil {
+ return nil, fmt.Errorf("value [%v] of 'max_receive_size' could not be understood", maxReceiveStr)
+ }
+ cfg.MaxCallRecvMsgSize = int(val)
+ }
+
etcd, err := clientv3.New(cfg)
if err != nil {
return nil, err |
|
Vault uses |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment:
Note we are running inside docker. Both host OS and docker base image is ubuntu 16.04.
Also happens with Vault version v0.8.3
Vault Config File:
Startup Log Output:
Expected Behavior:
After creating 8000 tokens, vault should restart normally.
Actual Behavior:
After creating 8000 tokens, vault cannot be restarted.
Steps to Reproduce:
Create 8000 tokens:
Restart vault server.
Important Factoids:
Large numbers of tokens (approx 6000-8000+) with an etcd backend cause Vault to fail to start up due to the default limit of 4MiB receive size in gRPC.
This does not prevent Vault from working once running and unsealed - if the number of tokens increases while Vault is running, nothing happens, but if restarted, Vault will not unseal again unless:
a) the 4MiB limit is increased by editing the vendored Vault dependencies (specifically vendor/google.golang.org/grpc/clientconn.go L96) and rebuilding, or
b) etcd is wiped and/or the tokens are deleted from etcd
The token counts we're seeing may be caused something we're doing wrong in how we use Vault, but even so I'd hope for a graceful degradation rather than a hard stops-working-at-4MiB - perhaps the lease scan should be paginated or similar?
Critical error is:
References:
n/a
The text was updated successfully, but these errors were encountered: