-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cleaning of stale leases #2452
Merged
+497
−54
Merged
Cleaning of stale leases #2452
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit
Hold shift + click to select a range
1e5d6e3
Added sys/tidy-leases endpoint
vishalnayak 239bd1c
Add locking where possible while doing auth/token/tidy
vishalnayak 14aaa0a
Merge branch 'oss' into clean-stale-leases
vishalnayak 3477038
Added atomic lock to ensure a single tidy operation is in progress
vishalnayak e52625d
Revoke lease that has empty token; added logs
vishalnayak dca0d70
Added logger to token store and logs to tidy function
vishalnayak de1a2a0
Added caching of looked up tokens
vishalnayak 65c63b4
Fix the log statements
vishalnayak b036478
Fix logging levels
vishalnayak 711153d
Fix logging suggestions; put the policyStore nil check back in
vishalnayak 0d629ff
Cache only valid tokens
vishalnayak 0c65cd4
Some more logging updates
vishalnayak 785177a
Merge branch 'oss' into sys-tidy-leases
vishalnayak 3fdf38a
Distinguish valid and invalid tokens using bool value in cache
vishalnayak 98cdb68
Use an atomic lock for tidy operation in token store
vishalnayak 2ef62fe
refactor lock handling in token tidy function
vishalnayak a8ef2c0
Refactor locking code in lease tidy; add ending debug statements
vishalnayak 0892102
Merge branch 'oss' into sys-tidy-leases
vishalnayak 8c7b175
Skip checking the validity of an empty client token
vishalnayak 853233a
Added a test for tidying of empty token
vishalnayak d07d3cb
Added steps to check if invalid token is properly cleaned up
vishalnayak 79fc0d8
Check if multiple leases with same invalid token is getting cleaned up
vishalnayak 497bebe
Do not duplicate log lines for invalid leases
vishalnayak aa08e5c
Added test to check the atomicity of the lease tidy operation
vishalnayak a3c2a42
Test to check that leases with valid tokens are not being cleaned up
vishalnayak b6843ec
Added summary logs to help better understand the consequence
vishalnayak 0c02540
Less scary debugging
jefferai b3c6a56
change some logging output
jefferai 415b0a2
Two things:
jefferai 0bda5a7
Adhere to tainted status in salted accessor lookup
jefferai 8d35f92
consistent logging
vishalnayak a2e431b
Added logs when deletion fails so we can rely on server logs
vishalnayak 2d21bf6
logging updates
vishalnayak 5bc47b0
Add taint flag for looking up by accessor
jefferai b0c4a7e
Add more cleanup if a lease fails to register and revoke tokens if re…
jefferai 265b4cd
Merge remote-tracking branch 'oss/master' into sys-tidy-leases
jefferai 2f6e924
Fix substitution of index/child in delete call
jefferai f1d2fc3
Merge branch 'master-oss' into sys-tidy-leases
jefferai 5320da0
Move tidy-leases to leases/tidy
jefferai cacf072
Update commenting
jefferai 4de09fb
Update Tidy function comment
vishalnayak 106f08a
Fix up the tests
vishalnayak 7829107
Update comments
jefferai 1378dd5
Move client token check in exp register to top
jefferai 5dde45d
Address feedback
jefferai e61298e
Update debugging around tidy
jefferai 0e10477
Merge branch 'master-oss' into sys-tidy-leases
jefferai File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Distinguish valid and invalid tokens using bool value in cache
- v1.18.3
- v1.18.2
- v1.18.1
- v1.18.0
- v1.18.0-rc1
- v1.17.6
- v1.17.5
- v1.17.4
- v1.17.3
- v1.17.2
- v1.17.1
- v1.17.0
- v1.17.0-rc1
- v1.16.3
- v1.16.2
- v1.16.1
- v1.16.0
- v1.16.0-rc3
- v1.16.0-rc2
- v1.16.0-rc1
- v1.15.8+ent
- v1.15.7+ent
- v1.15.6
- v1.15.5
- v1.15.4
- v1.15.3
- v1.15.2
- v1.15.1
- v1.15.0
- v1.15.0-rc1
- v1.14.12+ent
- v1.14.11+ent
- v1.14.10
- v1.14.9
- v1.14.8
- v1.14.7
- v1.14.6
- v1.14.5
- v1.14.4
- v1.14.3
- v1.14.2
- v1.14.1
- v1.14.0
- v1.14.0-rc1
- v1.13.13
- v1.13.12
- v1.13.11
- v1.13.10
- v1.13.9
- v1.13.8
- v1.13.7
- v1.13.6
- v1.13.5
- v1.13.4
- v1.13.3
- v1.13.2
- v1.13.1
- v1.13.0
- v1.13.0-rc1
- v1.12.11
- v1.12.10
- v1.12.9
- v1.12.8
- v1.12.7
- v1.12.6
- v1.12.5
- v1.12.4
- v1.12.3
- v1.12.2
- v1.12.1
- v1.12.0
- v1.12.0-rc1
- v1.11.12
- v1.11.11
- v1.11.10
- v1.11.9
- v1.11.8
- v1.11.7
- v1.11.6
- v1.11.5
- v1.11.4
- v1.11.3
- v1.11.2
- v1.11.1
- v1.11.0
- v1.11.0-rc1
- v1.10.11
- v1.10.10
- v1.10.9
- v1.10.8
- v1.10.7
- v1.10.6
- v1.10.5
- v1.10.4
- v1.10.3
- v1.10.2
- v1.10.1
- v1.10.0
- v1.10.0-rc1
- v1.9.10
- v1.9.9
- v1.9.8
- v1.9.7
- v1.9.6
- v1.9.5
- v1.9.4
- v1.9.3
- v1.9.2
- v1.9.1
- v1.9.0
- v1.9.0-rc1
- v1.8.12
- v1.8.11
- v1.8.10
- v1.8.9
- v1.8.8
- v1.8.7
- v1.8.6
- v1.8.5
- v1.8.4
- v1.8.3
- v1.8.2
- v1.8.1
- v1.8.0
- v1.8.0-rc2
- v1.8.0-rc1
- v1.7.10
- v1.7.9
- v1.7.8
- v1.7.7
- v1.7.6
- v1.7.5
- v1.7.4
- v1.7.3
- v1.7.2
- v1.7.1
- v1.7.0
- v1.7.0-rc2
- v1.7.0-rc1
- v1.6.7
- v1.6.6
- v1.6.5
- v1.6.4
- v1.6.3
- v1.6.2
- v1.6.1
- v1.6.0
- v1.6.0-rc
- v1.5.9
- v1.5.8
- v1.5.7
- v1.5.6
- v1.5.5
- v1.5.4
- v1.5.3
- v1.5.2
- v1.5.1
- v1.5.0
- v1.5.0-rc
- v1.4.7
- v1.4.6
- v1.4.5
- v1.4.4
- v1.4.3
- v1.4.2
- v1.4.1
- v1.4.0
- v1.4.0-rc1
- v1.4.0-beta1
- v1.3.10
- v1.3.9
- v1.3.8
- v1.3.7
- v1.3.6
- v1.3.5
- v1.3.4
- v1.3.3
- v1.3.2
- v1.3.1
- v1.3.0
- v1.3.0-beta1
- v1.2.7
- v1.2.6
- v1.2.5
- v1.2.4
- v1.2.3
- v1.2.2
- v1.2.1
- v1.2.0
- v1.2.0-rc1
- v1.2.0-beta2
- v1.2.0-beta1
- v1.1.5
- v1.1.4
- v1.1.3
- v1.1.2
- v1.1.1
- v1.1.0
- v1.1.0-beta2
- v1.1.0-beta1
- v1.0.3
- v1.0.2
- v1.0.1
- v1.0.0
- v1.0.0-rc1
- v1.0.0-beta2
- v1.0.0-beta1
- v0.11.6
- v0.11.5
- v0.11.4
- v0.11.3
- v0.11.2
- v0.11.1
- v0.11.0
- v0.11.0-beta1
- v0.10.4
- v0.10.3
- v0.10.2
- v0.10.1
- v0.10.0
- v0.10.0-rc1
- v0.9.6
- v0.9.5
- v0.9.4
- v0.9.3
- v0.9.2
- v0.9.1
- v0.9.0
- v0.8.3
- v0.8.2
- v0.8.1
- v0.8.0
- v0.8.0-rc1
- v0.8.0-beta1
- v0.7.3
- v0.7.2
- v0.7.1
- sdk/v0.14.0
- sdk/v0.13.0
- sdk/v0.12.0
- sdk/v0.11.1
- sdk/v0.11.0
- sdk/v0.10.2
- sdk/v0.10.1
- sdk/v0.10.0
- sdk/v0.9.2
- sdk/v0.9.1
- sdk/v0.9.0
- sdk/v0.8.1
- sdk/v0.8.0
- sdk/v0.7.0
- sdk/v0.6.2
- sdk/v0.6.1
- sdk/v0.6.0
- sdk/v0.5.3
- sdk/v0.5.2
- sdk/v0.5.1
- sdk/v0.5.0
- sdk/v0.4.1
- sdk/v0.4.0
- sdk/v0.3.0
- sdk/v0.2.1
- sdk/v0.2.0
- sdk/v0.1.13
- sdk/v0.1.12
- sdk/v0.1.11
- sdk/v0.1.10
- sdk/v0.1.9
- sdk/v0.1.8
- old-stable-website-20210728
- old-stable-website
- main-creation
- last-go-modable
- ent-changelog-1.15.9
- ent-changelog-1.15.8
- ent-changelog-1.15.7
- ent-changelog-1.14.13
- ent-changelog-1.14.12
- ent-changelog-1.14.11
- api/v1.15.0
- api/v1.14.0
- api/v1.13.0
- api/v1.12.2
- api/v1.12.1
- api/v1.12.0
- api/v1.11.0
- api/v1.10.0
- api/v1.9.2
- api/v1.9.1
- api/v1.9.0
- api/v1.8.3
- api/v1.8.2
- api/v1.8.1
- api/v1.8.0
- api/v1.7.2
- api/v1.7.1
- api/v1.7.0
- api/v1.6.0
- api/v1.5.0
- api/v1.4.1
- api/v1.4.0
- api/v1.3.1
- api/v1.3.0
- api/v1.2.0
- api/v1.1.1
- api/v1.1.0
- api/v1.0.4
- api/v1.0.3
- api/v1.0.2
- api/v1.0.1
- api/auth/userpass/v0.8.0
- api/auth/userpass/v0.7.0
- api/auth/userpass/v0.6.0
- api/auth/userpass/v0.5.0
- api/auth/userpass/v0.4.1
- api/auth/userpass/v0.4.0
- api/auth/userpass/v0.3.0
- api/auth/userpass/v0.2.0
- api/auth/userpass/v0.1.0
- api/auth/ldap/v0.8.0
- api/auth/ldap/v0.7.0
- api/auth/ldap/v0.6.0
- api/auth/ldap/v0.5.0
- api/auth/ldap/v0.4.1
- api/auth/ldap/v0.4.0
- api/auth/ldap/v0.3.0
- api/auth/ldap/v0.2.0
- api/auth/ldap/v0.1.0
- api/auth/kubernetes/v0.8.0
- api/auth/kubernetes/v0.7.0
- api/auth/kubernetes/v0.6.0
- api/auth/kubernetes/v0.5.0
- api/auth/kubernetes/v0.4.1
- api/auth/kubernetes/v0.4.0
- api/auth/kubernetes/v0.3.0
- api/auth/kubernetes/v0.2.0
- api/auth/kubernetes/v0.1.0
- api/auth/gcp/v0.8.0
- api/auth/gcp/v0.7.0
- api/auth/gcp/v0.6.0
- api/auth/gcp/v0.5.0
- api/auth/gcp/v0.4.1
- api/auth/gcp/v0.4.0
- api/auth/gcp/v0.3.0
- api/auth/gcp/v0.2.0
- api/auth/gcp/v0.1.0
- api/auth/azure/v0.7.0
- api/auth/azure/v0.6.0
- api/auth/azure/v0.5.0
- api/auth/azure/v0.4.1
- api/auth/azure/v0.4.0
- api/auth/azure/v0.3.0
- api/auth/azure/v0.2.0
- api/auth/azure/v0.1.0
- api/auth/aws/v0.8.0
- api/auth/aws/v0.7.0
- api/auth/aws/v0.6.0
- api/auth/aws/v0.5.0
- api/auth/aws/v0.4.1
- api/auth/aws/v0.4.0
- api/auth/aws/v0.3.0
- api/auth/aws/v0.2.0
- api/auth/aws/v0.1.0
- api/auth/approle/v0.8.0
- api/auth/approle/v0.7.0
- api/auth/approle/v0.6.0
- api/auth/approle/v0.5.0
- api/auth/approle/v0.4.1
- api/auth/approle/v0.4.0
- api/auth/approle/v0.3.0
- api/auth/approle/v0.2.0
- api/auth/approle/v0.1.1
- api/auth/approle/v0.1.0
commit 3fdf38a58a990d4a9cd7d148c8bc3178e55fb4d3
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -124,7 +124,7 @@ func (m *ExpirationManager) Tidy() error { | |
var tidyErrors *multierror.Error | ||
|
||
// Create a cache to keep track of looked up tokens | ||
validTokenCache := make(map[string]struct{}) | ||
tokenCache := make(map[string]bool) | ||
i := 0 | ||
|
||
tidyFunc := func(leaseID string) { | ||
|
@@ -150,26 +150,33 @@ func (m *ExpirationManager) Tidy() error { | |
revokeLease = true | ||
} | ||
|
||
if _, ok := validTokenCache[le.ClientToken]; ok { | ||
return | ||
} | ||
isValid, ok := tokenCache[le.ClientToken] | ||
if !ok { | ||
saltedID := m.tokenStore.SaltID(le.ClientToken) | ||
lock := locksutil.LockForKey(m.tokenStore.tokenLocks, le.ClientToken) | ||
lock.RLock() | ||
te, err := m.tokenStore.lookupSalted(saltedID, true) | ||
lock.RUnlock() | ||
|
||
saltedID := m.tokenStore.SaltID(le.ClientToken) | ||
lock := locksutil.LockForKey(m.tokenStore.tokenLocks, le.ClientToken) | ||
lock.RLock() | ||
te, err := m.tokenStore.lookupSalted(saltedID, true) | ||
lock.RUnlock() | ||
|
||
if err != nil { | ||
tidyErrors = multierror.Append(tidyErrors, fmt.Errorf("failed to lookup token: %v", err)) | ||
return | ||
} | ||
if err != nil { | ||
tidyErrors = multierror.Append(tidyErrors, fmt.Errorf("failed to lookup token: %v", err)) | ||
return | ||
} | ||
|
||
if te == nil { | ||
m.logger.Debug("expiration: lease has an invalid token", "lease_id", leaseID) | ||
revokeLease = true | ||
if te == nil { | ||
m.logger.Debug("expiration: lease has an invalid token", "lease_id", leaseID) | ||
revokeLease = true | ||
tokenCache[le.ClientToken] = false | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not necessary, but I think you might as well add a |
||
} else { | ||
tokenCache[le.ClientToken] = true | ||
} | ||
} else { | ||
validTokenCache[le.ClientToken] = struct{}{} | ||
if isValid { | ||
return | ||
} else { | ||
m.logger.Debug("expiration: lease has an invalid token", "lease_id", leaseID) | ||
revokeLease = true | ||
} | ||
} | ||
|
||
if revokeLease { | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add some progress logging so people can be aware things are still happening.