Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catchpoints: Enrich catchpoint generation and status with KV metadata #4808

Merged
merged 32 commits into from
Nov 18, 2022
Merged

Catchpoints: Enrich catchpoint generation and status with KV metadata #4808

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
6af364f
always process KvMods to add OldData to mods when building StateDeltas
cce Nov 16, 2022
df4dc56
remove guard clause
cce Nov 16, 2022
d4a9408
Merge remote-tracking branch 'origin/master' into fix-kvmods-deltas
cce Nov 16, 2022
de58702
Test that old data is populated by roundCowState.deltas()
jannotti Nov 16, 2022
f3d88dd
Merge pull request #11 from jannotti/fix-kvmods-deltas-test
cce Nov 16, 2022
f3b226c
Add KVs count to catchpoint generation
michaeldiamant Nov 16, 2022
a8e6cfa
Try to wire in support for counting accounts vs KVs
michaeldiamant Nov 16, 2022
28969e9
squash box changes that become no-ops
jannotti Nov 16, 2022
06555df
Plumb kv stats back to status endpoint
michaeldiamant Nov 17, 2022
07ee088
Plumb kv stats back through goal
michaeldiamant Nov 17, 2022
7e206f5
Merge pull request #12 from jannotti/squash-optimization
cce Nov 17, 2022
277dc19
Fix codegen_verification failure notification condition
michaeldiamant Nov 17, 2022
de8d3d7
Make minimal changes to make the build pass
michaeldiamant Nov 17, 2022
1bf1861
Fix reviewdog lint error
michaeldiamant Nov 17, 2022
aeb4f19
Make HashKind an explicit concept
michaeldiamant Nov 17, 2022
166d941
Fix lint errors
michaeldiamant Nov 17, 2022
4c4dba0
Fix msgp generation
michaeldiamant Nov 17, 2022
7e1f45c
Merge remote-tracking branch 'cce/fix-kvmods-deltas' into kvs_catchpo…
michaeldiamant Nov 17, 2022
d27d3a4
Remove extraneous assignments
michaeldiamant Nov 17, 2022
85e25e9
Fix counting logic
michaeldiamant Nov 17, 2022
c5e14df
Emit KVs count during catchpointdump file
michaeldiamant Nov 17, 2022
c480d17
Merge branch 'master' into kvs_catchpoint_stats
michaeldiamant Nov 17, 2022
e41f3ec
Add a test defending against hashing breaking changes
michaeldiamant Nov 18, 2022
465b954
Confirm expected hash kind
michaeldiamant Nov 18, 2022
015199a
Reverse argument ordering to match prior convention
michaeldiamant Nov 18, 2022
beff392
Make hashKind members private
michaeldiamant Nov 18, 2022
168026a
Merge branch 'master' into kvs_catchpoint_stats
michaeldiamant Nov 18, 2022
9d685c8
Merge branch 'master' into kvs_catchpoint_stats
michaeldiamant Nov 18, 2022
351f9be
Make hashKindEncodingIndex private
michaeldiamant Nov 18, 2022
c9a8b11
Fix comment spacing per code review
michaeldiamant Nov 18, 2022
5fd5770
Shorten method name per code review
michaeldiamant Nov 18, 2022
56b4a54
Pass resourcesData as pointer because it's a large struct
michaeldiamant Nov 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions catchup/catchpointService.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,15 @@ type CatchpointCatchupStats struct {
TotalAccounts uint64
ProcessedAccounts uint64
VerifiedAccounts uint64
TotalKVs uint64
ProcessedKVs uint64
VerifiedKVs uint64
TotalBlocks uint64
AcquiredBlocks uint64
VerifiedBlocks uint64
ProcessedBytes uint64
TotalAccountHashes uint64
TotalKVHashes uint64
StartTime time.Time
}

Expand Down Expand Up @@ -303,7 +307,7 @@ func (cs *CatchpointCatchupService) processStageLedgerDownload() (err error) {
if err == nil {
cs.log.Infof("ledger downloaded in %d seconds", time.Since(start)/time.Second)
start = time.Now()
err = cs.ledgerAccessor.BuildMerkleTrie(cs.ctx, cs.updateVerifiedAccounts)
err = cs.ledgerAccessor.BuildMerkleTrie(cs.ctx, cs.updateVerifiedCounts)
if err == nil {
cs.log.Infof("built merkle trie in %d seconds", time.Since(start)/time.Second)
break
Expand Down Expand Up @@ -335,12 +339,17 @@ func (cs *CatchpointCatchupService) processStageLedgerDownload() (err error) {
return nil
}

// updateVerifiedAccounts update the user's statistics for the given verified accounts
func (cs *CatchpointCatchupService) updateVerifiedAccounts(addedTrieHashes uint64) {
// updateVerifiedCounts update the user's statistics for the given verified hashes
func (cs *CatchpointCatchupService) updateVerifiedCounts(accountCount, kvCount uint64) {
cs.statsMu.Lock()
defer cs.statsMu.Unlock()

if cs.stats.TotalAccountHashes > 0 {
cs.stats.VerifiedAccounts = cs.stats.TotalAccounts * addedTrieHashes / cs.stats.TotalAccountHashes
cs.stats.VerifiedAccounts = cs.stats.TotalAccounts * accountCount / cs.stats.TotalAccountHashes
}

if cs.stats.TotalKVs > 0 {
cs.stats.VerifiedKVs = kvCount
}
}

Expand Down Expand Up @@ -756,6 +765,8 @@ func (cs *CatchpointCatchupService) updateLedgerFetcherProgress(fetcherStats *le
defer cs.statsMu.Unlock()
cs.stats.TotalAccounts = fetcherStats.TotalAccounts
cs.stats.ProcessedAccounts = fetcherStats.ProcessedAccounts
cs.stats.TotalKVs = fetcherStats.TotalKVs
cs.stats.ProcessedKVs = fetcherStats.ProcessedKVs
cs.stats.ProcessedBytes = fetcherStats.ProcessedBytes
cs.stats.TotalAccountHashes = fetcherStats.TotalAccountHashes
}
Expand Down
2 changes: 2 additions & 0 deletions cmd/catchpointdump/file.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,7 @@ func printAccountsDatabase(databaseName string, fileHeader ledger.CatchpointFile
"Block Header Digest: %s",
"Catchpoint: %s",
"Total Accounts: %d",
"Total KVs: %d",
"Total Chunks: %d",
}
var headerValues = []interface{}{
Expand All @@ -275,6 +276,7 @@ func printAccountsDatabase(databaseName string, fileHeader ledger.CatchpointFile
fileHeader.BlockHeaderDigest.String(),
fileHeader.Catchpoint,
fileHeader.TotalAccounts,
fileHeader.TotalKVs,
fileHeader.TotalChunks,
}
// safety check
Expand Down
2 changes: 1 addition & 1 deletion cmd/goal/messages.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ const (
infoNodeStatus = "Last committed block: %d\nTime since last block: %s\nSync Time: %s\nLast consensus protocol: %s\nNext consensus protocol: %s\nRound for next consensus protocol: %d\nNext consensus protocol supported: %v"
catchupStoppedOnUnsupported = "Last supported block (%d) is committed. The next block consensus protocol is not supported. Catchup service is stopped."
infoNodeCatchpointCatchupStatus = "Last committed block: %d\nSync Time: %s\nCatchpoint: %s"
infoNodeCatchpointCatchupAccounts = "Catchpoint total accounts: %d\nCatchpoint accounts processed: %d\nCatchpoint accounts verified: %d"
infoNodeCatchpointCatchupAccounts = "Catchpoint total accounts: %d\nCatchpoint accounts processed: %d\nCatchpoint accounts verified: %d\nCatchpoint total KVs: %d\nCatchpoint KVs processed: %d\nCatchpoint KVs verified: %d"
infoNodeCatchpointCatchupBlocks = "Catchpoint total blocks: %d\nCatchpoint downloaded blocks: %d"
nodeLastCatchpoint = "Last Catchpoint: %s"
errorNodeCreationIPFailure = "Parsing passed IP %v failed: need a valid IPv4 or IPv6 address with a specified port number"
Expand Down
3 changes: 2 additions & 1 deletion cmd/goal/node.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -467,7 +467,8 @@ func makeStatusString(stat model.NodeStatusResponse) string {

if stat.CatchpointTotalAccounts != nil && (*stat.CatchpointTotalAccounts > 0) && stat.CatchpointProcessedAccounts != nil {
statusString = statusString + "\n" + fmt.Sprintf(infoNodeCatchpointCatchupAccounts, *stat.CatchpointTotalAccounts,
*stat.CatchpointProcessedAccounts, *stat.CatchpointVerifiedAccounts)
*stat.CatchpointProcessedAccounts, *stat.CatchpointVerifiedAccounts,
*stat.CatchpointTotalKvs, *stat.CatchpointProcessedKvs, *stat.CatchpointVerifiedKvs)
}
if stat.CatchpointAcquiredBlocks != nil && stat.CatchpointTotalBlocks != nil && (*stat.CatchpointAcquiredBlocks+*stat.CatchpointTotalBlocks > 0) {
statusString = statusString + "\n" + fmt.Sprintf(infoNodeCatchpointCatchupBlocks, *stat.CatchpointTotalBlocks,
Expand Down
2 changes: 1 addition & 1 deletion components/mocks/mockCatchpointCatchupAccessor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ func (m *MockCatchpointCatchupAccessor) ProcessStagingBalances(ctx context.Conte
}

// BuildMerkleTrie inserts the account hashes into the merkle trie
func (m *MockCatchpointCatchupAccessor) BuildMerkleTrie(ctx context.Context, progressUpdates func(uint64)) (err error) {
func (m *MockCatchpointCatchupAccessor) BuildMerkleTrie(ctx context.Context, progressUpdates func(uint64, uint64)) (err error) {
return nil
}

Expand Down
12 changes: 12 additions & 0 deletions daemon/algod/api/algod.oas2.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3523,6 +3523,18 @@
"description": "The number of accounts from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchpoint-total-kvs": {
"description": "The total number of key-values (KVs) included in the current catchpoint",
"type": "integer"
},
"catchpoint-processed-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-verified-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchpoint-total-blocks": {
"description": "The total number of blocks that are required to complete the current catchpoint catchup",
"type": "integer"
Expand Down
36 changes: 36 additions & 0 deletions daemon/algod/api/algod.oas3.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -525,6 +525,10 @@
"description": "The number of accounts from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-processed-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-total-accounts": {
"description": "The total number of accounts included in the current catchpoint",
"type": "integer"
Expand All @@ -533,10 +537,18 @@
"description": "The total number of blocks that are required to complete the current catchpoint catchup",
"type": "integer"
},
"catchpoint-total-kvs": {
"description": "The total number of key-values (KVs) included in the current catchpoint",
"type": "integer"
},
"catchpoint-verified-accounts": {
"description": "The number of accounts from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchpoint-verified-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchup-time": {
"description": "CatchupTime in nanoseconds",
"type": "integer"
Expand Down Expand Up @@ -4101,6 +4113,10 @@
"description": "The number of accounts from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-processed-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-total-accounts": {
"description": "The total number of accounts included in the current catchpoint",
"type": "integer"
Expand All @@ -4109,10 +4125,18 @@
"description": "The total number of blocks that are required to complete the current catchpoint catchup",
"type": "integer"
},
"catchpoint-total-kvs": {
"description": "The total number of key-values (KVs) included in the current catchpoint",
"type": "integer"
},
"catchpoint-verified-accounts": {
"description": "The number of accounts from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchpoint-verified-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchup-time": {
"description": "CatchupTime in nanoseconds",
"type": "integer"
Expand Down Expand Up @@ -4232,6 +4256,10 @@
"description": "The number of accounts from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-processed-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been processed so far as part of the catchup",
"type": "integer"
},
"catchpoint-total-accounts": {
"description": "The total number of accounts included in the current catchpoint",
"type": "integer"
Expand All @@ -4240,10 +4268,18 @@
"description": "The total number of blocks that are required to complete the current catchpoint catchup",
"type": "integer"
},
"catchpoint-total-kvs": {
"description": "The total number of key-values (KVs) included in the current catchpoint",
"type": "integer"
},
"catchpoint-verified-accounts": {
"description": "The number of accounts from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchpoint-verified-kvs": {
"description": "The number of key-values (KVs) from the current catchpoint that have been verified so far as part of the catchup",
"type": "integer"
},
"catchup-time": {
"description": "CatchupTime in nanoseconds",
"type": "integer"
Expand Down
9 changes: 9 additions & 0 deletions daemon/algod/api/server/v2/generated/model/types.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading