Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config value that gives users options to skip calculating role for each lease #22651

Merged
merged 24 commits into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions changelog/22651.txt
elliesterner marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:improvement
core/quotas: Add configuration to allow skipping of expensive role calculations
```
1 change: 1 addition & 0 deletions command/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -2953,6 +2953,7 @@ func createCoreConfig(c *ServerCommand, config *server.Config, backend physical.
LogicalBackends: c.LogicalBackends,
Logger: c.logger,
DetectDeadlocks: config.DetectDeadlocks,
ImpreciseLeaseRoleTracking: config.ImpreciseLeaseRoleTracking,
DisableSentinelTrace: config.DisableSentinelTrace,
DisableCache: config.DisableCache,
DisableMlock: config.DisableMlock,
Expand Down
9 changes: 9 additions & 0 deletions command/server/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ type Config struct {

DetectDeadlocks string `hcl:"detect_deadlocks"`

ImpreciseLeaseRoleTracking bool `hcl:"imprecise_lease_role_tracking"`
Copy link
Contributor

@VioletHynes VioletHynes Aug 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though it's more verbose, how about imprecise_lease_count_quota_role_tracking?

I worry that "lease role" won't mean a lot to customers, and changing the name in the future if it's confusing will be difficult. I also want to be very explicit that it's lease count quota specific, and doesn't affect lease behaviour otherwise.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with this change. Although, I think documentation would help here the most. A lot of vault config names/env vars names are confusing to me until I read the docs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow I keep going back and forth on this one. The only concern I have is that imprecise_lease_count_quota_role_tracking is a bit of a mouthful. @ncabatoff @mpalmi do you have a preference?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to go this route, perhaps role_tracking is too much of an implementation detail and we could get away with skip_pre_quota_lease_tracking, or skip_old_lease_counts_on_quota_creation?

I'm honestly cool with any name we land on. This is a tricky one to pin down and I tend to agree that the documentation is the important part.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm going to stick with what we have. we have pretty good documentation on the option now which makes me feel better about it.


EnableResponseHeaderRaftNodeID bool `hcl:"-"`
EnableResponseHeaderRaftNodeIDRaw interface{} `hcl:"enable_response_header_raft_node_id"`

Expand Down Expand Up @@ -412,6 +414,11 @@ func (c *Config) Merge(c2 *Config) *Config {
result.DetectDeadlocks = c2.DetectDeadlocks
}

result.ImpreciseLeaseRoleTracking = c.ImpreciseLeaseRoleTracking
if c2.ImpreciseLeaseRoleTracking {
result.ImpreciseLeaseRoleTracking = c2.ImpreciseLeaseRoleTracking
}

result.EnableResponseHeaderRaftNodeID = c.EnableResponseHeaderRaftNodeID
if c2.EnableResponseHeaderRaftNodeID {
result.EnableResponseHeaderRaftNodeID = c2.EnableResponseHeaderRaftNodeID
Expand Down Expand Up @@ -1144,6 +1151,8 @@ func (c *Config) Sanitized() map[string]interface{} {
"experiments": c.Experiments,

"detect_deadlocks": c.DetectDeadlocks,

"imprecise_lease_role_tracking": c.ImpreciseLeaseRoleTracking,
}
for k, v := range sharedResult {
result[k] = v
Expand Down
1 change: 1 addition & 0 deletions command/server/config_test_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,7 @@ func testConfig_Sanitized(t *testing.T) {
"add_mount_point_rollback_metrics": false,
},
"administrative_namespace_path": "admin/",
"imprecise_lease_role_tracking": false,
}

addExpectedEntSanitizedConfig(expected, []string{"http"})
Expand Down
1 change: 1 addition & 0 deletions http/sys_config_state_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ func TestSysConfigState_Sanitized(t *testing.T) {
},
"storage": tc.expectedStorageOutput,
"administrative_namespace_path": "",
"imprecise_lease_role_tracking": false,
}

if tc.expectedHAStorageOutput != nil {
Expand Down
2 changes: 2 additions & 0 deletions http/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ func rateLimitQuotaWrapping(handler http.Handler, core *vault.Core) http.Handler
NamespacePath: ns.Path,
ClientAddress: parseRemoteIPAddress(r),
}

// This checks if any role based quota is required (LCQ or RLQ).
requiresResolveRole, err := core.ResolveRoleForQuotas(r.Context(), quotaReq)
if err != nil {
core.Logger().Error("failed to lookup quotas", "path", path, "error", err)
Expand Down
7 changes: 7 additions & 0 deletions vault/core.go
Original file line number Diff line number Diff line change
Expand Up @@ -696,6 +696,9 @@ type Core struct {
// if populated, the callback is called for every request
// for testing purposes
requestResponseCallback func(logical.Backend, *logical.Request, *logical.Response)

// If any role based quota (LCQ or RLQ) is enabled, don't track lease counts by role
impreciseLeaseRoleTracking bool
}

// c.stateLock needs to be held in read mode before calling this function.
Expand Down Expand Up @@ -757,6 +760,9 @@ type CoreConfig struct {
// Use the deadlocks library to detect deadlocks
DetectDeadlocks string

// If no role-based lease count quota or role based rate limiting quota is enabled, don't track lease counts by role
elliesterner marked this conversation as resolved.
Show resolved Hide resolved
ImpreciseLeaseRoleTracking bool

// Disables the trace display for Sentinel checks
DisableSentinelTrace bool

Expand Down Expand Up @@ -1029,6 +1035,7 @@ func CreateCore(conf *CoreConfig) (*Core, error) {
pendingRemovalMountsAllowed: conf.PendingRemovalMountsAllowed,
expirationRevokeRetryBase: conf.ExpirationRevokeRetryBase,
rollbackMountPathMetrics: conf.MetricSink.TelemetryConsts.RollbackMetricsIncludeMountPoint,
impreciseLeaseRoleTracking: conf.ImpreciseLeaseRoleTracking,
}

c.standbyStopCh.Store(make(chan struct{}))
Expand Down
10 changes: 6 additions & 4 deletions vault/request_handling.go
Original file line number Diff line number Diff line change
Expand Up @@ -1687,10 +1687,12 @@ func (c *Core) handleLoginRequest(ctx context.Context, req *logical.Request) (re
// Attach the display name, might be used by audit backends
req.DisplayName = auth.DisplayName

// If this is not a role-based quota, we still need to associate the
// login role with this lease for later lease-count quotas to be
// accurate.
if reqRole == nil && resp.Auth.TokenType != logical.TokenTypeBatch {
loginRequest := resp.Auth.TokenType != logical.TokenTypeBatch
elliesterner marked this conversation as resolved.
Show resolved Hide resolved

// If role was not already determined by http.rateLimitQuotaWrapping
// and this is a login request
// and imprecise_lease_role_tracking is not set to true, calculate the role
if reqRole == nil && loginRequest && !c.impreciseLeaseRoleTracking {
role = c.DetermineRoleFromLoginRequest(ctx, req.MountPoint, req.Data)
}

Expand Down
2 changes: 2 additions & 0 deletions vault/testing.go
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,7 @@ func TestCoreWithSealAndUINoCleanup(t testing.T, opts *CoreConfig) *Core {
conf.CensusAgent = opts.CensusAgent
conf.AdministrativeNamespacePath = opts.AdministrativeNamespacePath
conf.AllLoggers = logger.AllLoggers
conf.ImpreciseLeaseRoleTracking = opts.ImpreciseLeaseRoleTracking

if opts.Logger != nil {
conf.Logger = opts.Logger
Expand Down Expand Up @@ -1553,6 +1554,7 @@ func NewTestCluster(t testing.T, base *CoreConfig, opts *TestClusterOptions) *Te
coreConfig.DisableAutopilot = base.DisableAutopilot
coreConfig.AdministrativeNamespacePath = base.AdministrativeNamespacePath
coreConfig.ServiceRegistration = base.ServiceRegistration
coreConfig.ImpreciseLeaseRoleTracking = base.ImpreciseLeaseRoleTracking

if base.BuiltinRegistry != nil {
coreConfig.BuiltinRegistry = base.BuiltinRegistry
Expand Down
4 changes: 4 additions & 0 deletions website/content/docs/configuration/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,10 @@ a negative effect on performance due to the tracking of each lock attempt.
the `VAULT_EXPERIMENTS` environment variable as a comma-separated list, or via the
[`-experiment`](/vault/docs/commands/server#experiment) flag.

- `imprecise_lease_role_tracking` `(bool: "false")` - If no role based quota exists for mount, do not keep track of lease roles.
Role based quotas include rate limit quotas or a lease count quotas that have a role specified role configured.
The purpose of this configuration is to reduce expensive role calculations if role base quotas are not being used.
elliesterner marked this conversation as resolved.
Show resolved Hide resolved

### High availability parameters

The following parameters are used on backends that support [high availability][high-availability].
Expand Down