Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding comments to all the database columns #6068

Merged
merged 4 commits into from
Jan 18, 2022
Merged

Adding comments to all the database columns #6068

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
44b74d2
Adding comments to addtional db columns
mm-near Dec 27, 2021
7140fd1
Added documentation for all the storage columns
mm-near Jan 14, 2022
29aceac
Merge branch 'master' into 1227_more_readme
mm-near Jan 14, 2022
adbad9a
Merge master into 1227_more_readme
near-bulldozer[bot] Jan 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 83 additions & 15 deletions core/store/src/db.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,27 +110,60 @@ pub enum DBCol {
ColChunks = 13,
/// Storage for PartialEncodedChunk.
/// - *Rows*: ChunkHash (CryptoHash)
/// - *Content type*: [near_primitives::sharding::PartialEncodedChunkV1]
/// - *Content type*: [near_primitives::sharding::PartialEncodedChunk]
ColPartialChunks = 14,
/// Blocks for which chunks need to be applied after the state is downloaded for a particular epoch
/// TODO: describe what is exactly inside the rows/cells.
/// - *Rows*: BlockHash (CryptoHash)
/// - *Content type*: Vec of BlockHash (CryptoHash)
ColBlocksToCatchup = 15,
/// Blocks for which the state is being downloaded
/// - *Rows*: First block of the epoch (CryptoHash)
/// - *Content type*: StateSyncInfo
ColStateDlInfos = 16,
/// Blocks that were ever challenged.
/// - *Rows*: BlockHash (CryptoHash)
/// - *Content type*: 'true' (bool)
ColChallengedBlocks = 17,
/// Contains all the Shard State Headers.
/// - *Rows*: StateHeaderKey (ShardId || BlockHash)
/// - *Content type*: ShardStateSyncResponseHeader
ColStateHeaders = 18,
/// Contains all the invalid chunks (that we had trouble decoding or verifying).
/// - *Rows*: ShardChunkHeader object
/// - *Content type*: EncodedShardChunk
ColInvalidChunks = 19,
/// Contains 'BlockExtra' information that is computed after block was processed.
/// Currently it stores only challenges results.
/// - *Rows*: BlockHash (CryptoHash)
/// - *Content type*: BlockExtra
ColBlockExtra = 20,
/// Store hash of a block per each height, to detect double signs.
/// Store hash of all block per each height, to detect double signs.
/// - *Rows*: int (height of the block)
/// - *Content type*: Map: EpochId -> Set of BlockHash(CryptoHash)
ColBlockPerHeight = 21,
/// Contains State parts that we've received.
/// - *Rows*: StatePartKey (BlockHash || ShardId || PartId (u64))
/// - *Content type*: state part (bytes)
ColStateParts = 22,
/// Contains mapping from epoch_id to epoch start (first block height of the epoch)
/// - *Rows*: EpochId (CryptoHash) -- TODO: where does the epoch_id come from? it looks like blockHash..
/// - *Content type*: BlockHeight (int)
ColEpochStart = 23,
/// Map account_id to announce_account
/// Map account_id to announce_account (which peer has announced which account in the current epoch). // TODO: explain account annoucement
/// - *Rows*: AccountId (str)
/// - *Content type*: AnnounceAccount
ColAccountAnnouncements = 24,
/// Next block hashes in the sequence of the canonical chain blocks
/// Next block hashes in the sequence of the canonical chain blocks.
/// - *Rows*: BlockHash (CryptoHash)
/// - *Content type*: next block: BlockHash (CryptoHash)
ColNextBlockHashes = 25,
/// `LightClientBlock`s corresponding to the last final block of each completed epoch
/// `LightClientBlock`s corresponding to the last final block of each completed epoch.
/// - *Rows*: EpochId (CryptoHash)
/// - *Content type*: LightClientBlockView
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ... feels fishy. My understanding that *View types are for RPC, not for storing directly in DB...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this code:
https://github.com/near/nearcore/blob/master/chain/chain/src/store.rs#L2603

And this type is BorshSerializable:
https://github.com/near/nearcore/blob/master/core/primitives/src/views.rs#L1433

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's not a statement about this particular change, it's a statement about pre-existing state of code.

My understanding is that *View-types are intended to be API types we use in JSON RPC. So, eg, we use Action as internal representation of what we store in the database, and we use ActionView for the RPC (which is reachable from the chunk method).

Some of the *View types are not only JSON-encodable, but also borsh-encodable. My understanding that this bottoms out in RoutedMessageBody::FinalExecutionOutcomeView. I am not sure if that should've use *View struct or not: it feels weird that here we have both *View and not *View struct. So, my current understanding is "some view types are borsh because we use then in p2p messages, which I am not sure why we do".

What I am noticing here is that there's also at least one *View type which we store in the database. And that's surprising, because that means that we store both , eg, ValidatorStakeView (through ColEpochLightClientBlocks) and ValidatorStake (through ColEpochValidatorInfo) in the database.

Overall, it seems that the codebase as a whole is confused about what *View really means, and that some things are miss-classified as *View.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc #5516

ColEpochLightClientBlocks = 26,
/// Mapping from Receipt id to Shard id
/// - *Rows*: ReceiptId (CryptoHash)
/// - *Content type*: Shard Id || ref_count (u64 || u64)
ColReceiptIdToShardId = 27,
// Deprecated.
_ColNextBlockWithNewChunk = 28,
Expand Down Expand Up @@ -171,33 +204,68 @@ pub enum DBCol {
/// - *Rows*: transaction hash
/// - *Column type*: SignedTransaction
ColTransactions = 33,
/// Mapping from a given (Height, ShardId) to the Chunk hash.
/// - *Rows*: (Height || ShardId) - (u64 || u64)
/// - *Column type*: ChunkHash (CryptoHash)
ColChunkPerHeightShard = 34,
/// Changes to key-values that we have recorded.
/// Changes to state (Trie) that we have recorded.
/// - *Rows*: BlockHash || TrieKey (TrieKey is written via custom to_vec)
/// - *Column type*: TrieKey, new value and reason for change (RawStateChangesWithTrieKey)
ColStateChanges = 35,
/// Mapping from Block to its refcount. (Refcounts are used in handling chain forks)
/// - *Rows*: BlockHash (CryptoHash)
/// - *Column type*: refcount (u64)
ColBlockRefCount = 36,
/// Changes to Trie that we recorded during given block/shard processing.
/// - *Rows*: BlockHash || ShardId
/// - *Column type*: old root, new root, list of insertions, list of deletions (TrieChanges)
ColTrieChanges = 37,
/// Merkle tree of block hashes
/// Mapping from a block hash to a merkle tree of block hashes that are in the chain before it.
/// - *Rows*: BlockHash
/// - *Column type*: PartialMerkleTree - MerklePath to the leaf + number of leaves in the whole tree.
ColBlockMerkleTree = 38,
/// Mapping from height to the set of Chunk Hashes that were included in the block at that height.
/// - *Rows*: height (u64)
/// - *Column type*: Vec<ChunkHash (CryptoHash)>
ColChunkHashesByHeight = 39,
/// Block ordinals.
/// Mapping from block ordinal number (number of the block in the chain) to the BlockHash.
/// - *Rows*: ordinal (u64)
/// - *Column type*: BlockHash (CryptoHash)
ColBlockOrdinal = 40,
/// GC Count for each column
/// GC Count for each column - number of times we did the GarbageCollection on the column.
/// - *Rows*: column id (byte)
/// - *Column type*: u64
ColGCCount = 41,
/// All Outcome ids by block hash and shard id. For each shard it is ordered by execution order.
/// TODO: seems that it has only 'transaction ids' there (not sure if intentional)
/// - *Rows*: BlockShardId (BlockHash || ShardId) - 40 bytes
/// - *Column type*: Vec <OutcomeId (CryptoHash)>
ColOutcomeIds = 42,
/// Deprecated
_ColTransactionRefCount = 43,
/// Heights of blocks that have been processed
/// Heights of blocks that have been processed.
/// - *Rows*: height (u64)
/// - *Column type*: empty
ColProcessedBlockHeights = 44,
/// Receipts
/// Mapping from receipt hash to Receipt.
/// - *Rows*: receipt (CryptoHash)
/// - *Column type*: Receipt
ColReceipts = 45,
/// Precompiled machine code of the contract
/// Precompiled machine code of the contract, used by StoreCompiledContractCache.
/// - *Rows*: ContractCacheKey or code hash (not sure)
/// - *Column type*: near-vm-runner CacheRecord
ColCachedContractCode = 46,
/// Epoch validator information used for rpc purposes
/// Epoch validator information used for rpc purposes.
/// - *Rows*: epoch id (CryptoHash)
/// - *Column type*: EpochSummary
ColEpochValidatorInfo = 47,
/// Header Hashes indexed by Height
/// Header Hashes indexed by Height.
/// - *Rows*: height (u64)
/// - *Column type*: Vec<HeaderHashes (CryptoHash)>
ColHeaderHashesByHeight = 48,
/// State changes made by a chunk, used for splitting states
/// - *Rows*: BlockShardId (BlockHash || ShardId) - 40 bytes
/// - *Column type*: StateChangesForSplitStates
ColStateChangesForSplitStates = 49,
}

Expand Down