Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backports for ActivityLog and Reporting 1.11.x #21210

Merged
merged 29 commits into from
Jun 15, 2023

Conversation

mpalmi
Copy link
Contributor

@mpalmi mpalmi commented Jun 13, 2023

This PR encapsulates the backports for Census and ActivityLog changes to get the release branches in sync and should resolve VAULT-14702 and VAULT-14765 along with the backports for the new client count testing library.

It was easier to backport the changes en masse, since they're all intertwined. Teasing apart feature-specific changes would have been more difficult.

Commits were prepared using (with fish shell, $() is just ()):

git cherry-pick -v -x (git log origin/release/1.11.x..origin/main --oneline --reverse -- "*activity*" | awk '{ print $1 }')

Diff between backport and main

## NOTE: This diff does not include changes to core. 
git diff origin/activitylog-backports-1.11.x..origin/main -- "*activity*" "*census*" ':!ui/*'

https://gist.github.com/mpalmi/9b83c19a45002a045b216d6d6446f357

@mpalmi mpalmi changed the title Activitylog backports 1.11.x backports for ActivityLog and Reporting 1.13.x Jun 13, 2023
@mpalmi mpalmi changed the title backports for ActivityLog and Reporting 1.13.x backports for ActivityLog and Reporting 1.11.x Jun 13, 2023
@mpalmi mpalmi force-pushed the activitylog-backports-1.11.x branch from 1eea0e7 to 2ef457e Compare June 14, 2023 12:59
@mpalmi mpalmi changed the base branch from release/1.11.x to hll-backport-1.11.x June 14, 2023 13:00
@mpalmi
Copy link
Contributor Author

mpalmi commented Jun 14, 2023

While going through the backport, there were 4 classes of commits (marked below). The backports for 1.11.x also pull in HLL estimate changes, which will be marked with ⏫ to show where this PR begins:

❌ empty commits, which had already been backported but weren't attributed by git
❌ superfluous changes which rejected the entire commit (these got the 'ol --skip during the cherry-pick)
[no modifier] changes that ported over (sometimes with amendment)
[no modifier] clean cherry-picks, which ported over with no modifications
⏫ HLL estimate backports

2a8a8a8 fix off by one error in activity log nil padding for month data (#15731)
802ba78 iterate through all available logs for precomputation and query gets (#15768)
1575454 refactor some code in modifyResponseMonths and ensure that the last mo… (#15767)
c88c73e Activity Log Filtering Limit Parameter (#16000)
cf60460 ActivityLog Implement HyperLogLog Store Functionality During Precomputation (#16146)
2d12226 activity log refactoring port (#16162)
7e72219 Port: Use Stored Hll to Compute New Clients For Current Month (#16184)
614cee3 VAULT-6614 Enable role based quotas for lease-count quotas (OSS) (#16157)
7140730 Current Month Activity Estimate, Probabilistic Tests, And Bugfixes (#16447)
ff504ad re-add namespace attribution to current month (#16473)
687b3d1 oss port of vault-7225-bugfix (#16745)
89f08de HCP link integration (#16939)
8dedb31 Make some activity log tests less flaky (#17028)
8258b36 make proto (#17120)
72799f7 Update protos to match update of protobuf go library (#17215)
50a1f9e fix off by one err in current month client count computation (#17457)
0423ffb update protoc version to 3.21.7 oss (#17499)
483ef19 Eslint prefer-const (#17864)
840abfb Fix activity log end time (#17856)
87aa644 Don't return a 204 if there's no historical data (#17935)
186ee31 Link OSS (#18228)
b8de2c2 De-duplicate namespaces when historical and current month data are mixed (#18452)
cb61488 Account for mount counts when de-duplicating current and historical month data (#18598)
d641bbc Add new clients into the monthly breakdown (#18629)
2a5a07e Revert "Add new clients into the monthly breakdown (#18629)" (#18726)
771bd8b Add new clients into the monthly breakdown (#18766)
46b9921 Allow Token Create Requests To Be Replicated (#18689)
fbed961 UI: combine current + history client count tabs into one dashboard (#17575)
b5d7d47 VAULT-13061: Fix mount path discrepancy in activity log (#18916)
a9e17c2 VAULT-13763 normalize activity log mount paths (#19343)
7ef7297 UI: Use specific date in clients activity test (#19419)
9f7f8d5 VAULT-13729 activity log test godocs (#19433)
e55c18e adding copyright header (#19555)
e3c5977 Add no-op CensusAgent (#19625)
e9d6dbc activitylog: Fix pq.Get trace logger output (#19650)
bf2f1a8 UI: Fix flaky time-related tests (#19521)
b4fab6a VAULT-13191: OSS changes (#19891)
54904e4 VAULT-14733: Refactor processClientRecord in activity log (#19933)
d70c17f VAULT-14733: SegmentReader interface for reading activity log segments (#19934)
4b6ec40 Require activity log retention months at least the minimum (#20078)
05ba6bb api: Add reporting fields to activitylog config endpoint (#20086)
c95d4fb VAULT-14734: activity log write endpoint (#20019)
75c903f openapi: Add display attributes for /sys (p2) (#19707)
002a59a Add minimum_retention_months to config endpoint (#20150)
77f83d9 Refactor reporter for unseal setup (#20296)
730d0e2 VAULT-14733: Split logic of precomputedQueryWorker (#20073)
d234111 Start counting ACME certificate issuance as client activity (#20520)
35e2c16 VAULT-15703: Reload automated reporting (#20680)
6d95f8c Add client_type field to EntityRecord protobuf (#20626)
810d504 Add current_billing_period activity endpoint param (#20694)
5b23dd5 VAULT-14735: generate mock clients for activity log (#20252)
018ea84 VAULT-15395: Support mocking time functions in the activity log (#20720)
541f18e VAULT-14735: repeated and segmented activity log clients (#20699)
b4e2751 VAULT-14735: write mock activity log entity files (#20702)
dc5dd71 Deflake TestActivityLog_MultipleFragmentsAndSegments (#20930)
5002489 VAULT-15394: Generate activity log precomputed queries (#20778)

@mpalmi mpalmi force-pushed the activitylog-backports-1.11.x branch 2 times, most recently from 3cc42d4 to 512332b Compare June 14, 2023 16:10
@mpalmi mpalmi force-pushed the hll-backport-1.11.x branch from 4588aec to 25fcbcf Compare June 14, 2023 16:11
@mpalmi mpalmi force-pushed the activitylog-backports-1.11.x branch from 512332b to 7beb94a Compare June 14, 2023 16:13
@mpalmi mpalmi force-pushed the hll-backport-1.11.x branch from 25fcbcf to 35d8952 Compare June 14, 2023 20:03
@mpalmi mpalmi force-pushed the activitylog-backports-1.11.x branch from 7beb94a to e5888aa Compare June 14, 2023 21:27
command/server/config_test_helpers.go Show resolved Hide resolved
command/server/config_test_helpers_util.go Show resolved Hide resolved
vault/core.go Show resolved Hide resolved
vault/logical_system_test.go Show resolved Hide resolved
vault/testing.go Show resolved Hide resolved
Base automatically changed from hll-backport-1.11.x to release/1.11.x June 14, 2023 23:35
@mpalmi mpalmi force-pushed the activitylog-backports-1.11.x branch from e5888aa to 2ccb340 Compare June 14, 2023 23:59
@mpalmi mpalmi marked this pull request as ready for review June 15, 2023 00:06
@mpalmi mpalmi requested a review from raskchanky June 15, 2023 00:48
@mpalmi mpalmi merged commit ae23611 into release/1.11.x Jun 15, 2023
@mpalmi mpalmi deleted the activitylog-backports-1.11.x branch June 15, 2023 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants