updated ec spc and weight fn

hugo/data/sref_locations.yaml

@@ -14,12 +14,20 @@ chainsync: [systems, filecoin_blockchain, chainsync]
 
 ### SPC
 storage_power_consensus: [systems, filecoin_blockchain, storage_power_consensus]
-power_table: [systems, filecoin_blockchain, storage_power_consensus]
-expected_consensus: [algorithms, expected_consensus]
+ticket_chain: [systems, filecoin_blockchain, storage_power_consensus, ticket_chain]
+
+### Storage Market Actor
+power_table: [systems, filecoin_blockchain, storage_power_consensus, storage_power_actor, power_table]
+pledge_collateral: [systems, filecoin_blockchain, storage_power_consensus, storage_power_actor, pledge_collateral]
+
+### EC
 tickets: [algorithms, expected_consenus, tickets]
+expected_consensus: [algorithms, expected_consensus]
 leader_election: [algorithms, expected_consensus, leader_election]
 chain_selection: [algorithms, expected_consensus, chain_selection]
 finality: [algorithms, expected_consensus, finality]
+consensus_faults: [algorithms, expected_consensus, consensus_faults]
+
 ### Message Pool
 message_pool: [systems, filecoin_blockchain, message_pool]
 message_syncer: [systems, filecoin_blockchain, message_syncer]
@@ -30,6 +38,8 @@ message_storage: [systems, filecoin_blockchain, message_storage]
 # markets
 retrieval_market: [systems, filecoin_markets, retrieval_market]
 storage_market: [systems, filecoin_markets, storage_market]
+storage_deal_collateral: [systems, filecoin_markets, storage_market, storage_deal_collateral]
+storage_faults: [systems, filecoin_markets, storage_market, faults]
 
 # mining
 
@@ -41,4 +51,4 @@ post: [algorithms, proof_of_spacetime]
 
 # nodes
 
-# token
+# token

hugo/data/sref_titles.yaml

@@ -10,12 +10,15 @@ sys_vm: "VM"
 block_producer: "Block Producer"
 ### SPC
 storage_power_consensus: "Storage Power Consensus"
+ticket_chain: "Ticket Chain"
 power_table: "Power Table"
 expected_consensus: "Expected Consensus"
 tickets: "Tickets"
 leader_election: "Secret Leader Election"
 chain_selection: "Chain Selection"
 finality: "EC Finality"
+consensus_faults: "Consensus Faults"
+
 ### Message Pool
 message_pool: "Message Pool"
 message_syncer: "Message Syncer"
@@ -26,6 +29,8 @@ message_storage: "Message Storage"
 # markets
 retrieval_market: "Retrieval Market"
 storage_market: "Storage Market"
+storage_deal_collateral: "Storage Deal Collateral"
+storage_faults: "Storage Faults"
 
 # mining
 
@@ -38,4 +43,4 @@ post: "Proof of Spacetime"
 
 # nodes
 
-# token
+# token

src/libraries/filcrypto/filecoin_crypto_lib.go

@@ -9,11 +9,3 @@ func (self *VRFResult_I) Verify(input util.Bytes, pk VRFPublicKey) bool {
 func (self *VRFResult_I) ValidateSyntax() bool {
 	panic("TODO")
 }
-
-func (self *VDFResult_I) Verify(input util.Bytes) bool {
-	panic("TODO")
-}
-
-func (self *VDFResult_I) ValidateSyntax() bool {
-	panic("TODO")
-}

src/libraries/filcrypto/filecoin_crypto_lib.id

@@ -20,21 +20,6 @@ type VRFResult struct {
     ValidateSyntax() bool
 }
 
-type VDF struct {
-    evalKey            Bytes      @(internal)
-    VerifKey           Bytes      @(internal)
-
-    Eval(input Bytes)  VDFResult  @(static)
-}
-
-type VDFResult struct {
-    Output  Bytes  @(internal)
-    Proof   Bytes  @(internal)
-
-    Verify(input Bytes) bool @(static)
-    ValidateSyntax() bool
-}
-
 type SigKeyPair struct {
     PublicKey  SigPublicKey
     SecretKey  SigSecretKey

src/listings/data_structures.md

@@ -20,8 +20,9 @@ For more detail, see the full {{<sref app_address "address spec">}}.
 A block header contains information relevant to a particular point in time over which the network may achieve consensus. The block header contains:
 
 - The address of the miner that mined the block
-- An array of the tickets that led to this particular miner being selected as the leader for this round (see
-{{<sref leader_election>}} for more details), as well as a signature on the winning ticket
+- A ticket associated to this block's creation to be used as randomness elsewhere in the protocol (see
+{{<sref leader_election>}} for more details)
+- An electionProof showing this miner was eligible to mine, as well as a Nonce relating the number of rounds over which the block was mined (on expectation 1)
 - The set of parent blocks and aggregate {{<sref chain_selection "chain weight">}} of the parents
 - This block's height
 - Merkle root of the state tree (after applying the messages -- state transitions -- included in this block)
@@ -57,10 +58,7 @@ We define VRF personalizations as follow, to enable domain separation across ope
 # Ticket
 
 A ticket contains a shared random value referenced by a particular `Block` in the Filecoin blockchain.
-Every miner must produce a new `Ticket` each time they run a leader election attempt.
-In that sense, every new block produced will have one or more associated tickets
-(specifically, the block may contain more than one ticket if it corresponds to
-one or more zero-winner epochs of {{<sref expected_consensus>}}).
+Every miner must produce a new `Ticket` for each ticket used in a leader election attempt.
 
 To produce the ticket values,
 we use an [EC-VRF per Goldberg et al.](https://tools.ietf.org/html/draft-irtf-cfrg-vrf-04#page-10)
@@ -73,9 +71,9 @@ with Secp256k1 and SHA-256 to obtain a deterministic, pseudorandom output.
 The ticket is represented concretely by the `Ticket` data structure.
 Whenever the Filecoin protocol refers to ticket values
 (notably in crafting {{<sref post "PoSTs">}} or running leader election),
-what is meant is that the bytes of the `VDFResult` field in the `Ticket` struct are used.
+what is meant is that the bytes of the `VRFResult` field in the `Ticket` struct are used.
 Specifically, tickets are compared lexicographically,
-interpreting the bytes of the `VDFResult` as an unsigned integer value (little-endian).
+interpreting the bytes of the `VRFResult.Output` as an unsigned integer value (little-endian).
 
 # ElectionProof
 
@@ -84,7 +82,9 @@ by a miner during the leader election process.
 Its output value determines whether the miner is elected as one of the leaders,
 and hence is eligible to produce a block for the current epoch.
 The inclusion of the `ElectionProof` in the block allows other network participants
-to verify that the block was mined by a valid leader.
+to verify that the block was mined by a valid leader. With every leader election attempt for a given ticket,
+(in cases where no blocks are found in a round) a miner increments that ElectionProof's associated `Nonce`, marking
+increased block height.
 
 {{<goFile ElectionProof>}}
 

src/systems/filecoin_blockchain/blockchain.id

@@ -1,7 +1,6 @@
 import clock "github.com/filecoin-project/specs/systems/filecoin_nodes/clock"
 import st "github.com/filecoin-project/specs/systems/filecoin_vm/state_tree"
 import block "github.com/filecoin-project/specs/systems/filecoin_blockchain/struct/block"
-import chain "github.com/filecoin-project/specs/systems/filecoin_blockchain/struct/chain"
 
 type SectorID struct {}
 
@@ -18,8 +17,8 @@ type BlockchainSubsystem struct @(mutable) {
     Clock              &clock.WallClock
 
     LatestEpoch()      block.ChainEpoch
-    BestChain()        chain.Chain
-    CandidateChains()  [chain.Chain]
+    BestChain()        block.Chain
+    CandidateChains()  [block.Chain]
 
     // // Receiving blocks and tipset
     // should be channels or notifications
@@ -52,7 +51,4 @@ type BlockchainSubsystem struct @(mutable) {
 
     // call by BlockchainSubsystem itself in BlockReception after Round Cutoff
     ChooseTipset(tipsets [block.Tipset]) block.Tipset
-
-    // call by StorageMining
-    GetPostRandomness() chain.Randomness
 }

src/systems/filecoin_blockchain/blockchain/block_producer.go

@@ -0,0 +1 @@
+package block_producer

src/systems/filecoin_blockchain/chainsync/block_syncer.go

@@ -16,6 +16,8 @@ func (self *BlockSyncer_I) OnNewBlock(block block.Block) error {
 // The syntactic stage may be validated without reference to additional data (see block)
 func (bs *BlockSyncer_I) validateBlockSyntax(block block.Block) error {
 	panic("TODO")
+
+	// if !block.Epoch().WithinFinality
 	// if !block.MinerAddress().VerifySyntax(StorageMinerActor.Address.Protocol()) {
 	// 	return ErrInvalidBlockSyntax("bad miner address syntax")
 	// }

src/systems/filecoin_blockchain/storage_power_consensus/_index.md

@@ -11,9 +11,9 @@ The Storage Power Consensus subsystem is the main interface which enables Fileco
 Succinctly, the SPC subsystem offers the following services:
 - Access to the _Power Table_ for every subchain, accounting for individual storage miner power and total power on-chain.
 - Access to {{<sref expected_consensus>}} for individual storage miners, enabling:
-    - Access to verifiable randomness {{<sref tickets>}}as needed in the rest of the protocol. 
+    - Access to verifiable randomness {{<sref tickets>}} as needed in the rest of the protocol.
     - Running  {{<sref leader_election>}} to produce new blocks.
-    - Running {{<sref chain_selection>}} across subchains using EC's weighting function. 
+    - Running {{<sref chain_selection>}} across subchains using EC's weighting function.
     - Identification of {{<sref finality "the most recently finalized tipset">}}, for use by all protocol participants.
 
 Much of the Storage Power Consensus' subsystem functionality is detailed in the code below but we touch upon some of its behaviors in more detail.
@@ -22,34 +22,122 @@ Much of the Storage Power Consensus' subsystem functionality is detailed in the
 
 ## Distinguishing between storage miners and block miners
 
-There are two ways to earn Filecoin tokens in the Filecoin network: 
+There are two ways to earn Filecoin tokens in the Filecoin network:
 - By participating in the {{<sref storage_market>}} as a storage provider and being paid by clients for file storage deals.
 - By mining new blocks on the network, helping modify system state and secure the Filecoin consensus mechanism.
 
 We must distinguish between both types of "miners" (storage and block miners). {{<sref leader_election>}} in Filecoin is predicated on a miner's storage power. Thus, while all block miners will be storage miners, the reverse is not necessarily true.
 
 However, given Filecoin's "useful Proof-of-Work" is achieved through file storage (PoRep and PoSt), there is little overhead cost for storage miners to participate in leader election. Such a {{<sref storage_miner_actor>}} need only register with the {{<sref storage_power_actor>}} in order to participate in Expected Consensus and mine blocks.
 
-{{<label power_table>}}
-## The Power Table
+## Repeated leader election attempts
 
-The portion of blocks a given miner generates through leader election in EC (and so the block rewards they earn) is proportional to their `Power Fraction` over time. That is, a miner whose storage represents 1% of total storage on the network should mine 1% of blocks on expectation.
+In the case that no miner is eligible to produce a block in a given round of EC, the storage power consensus subsystem will be called by the block producer to attempt another leader election by incrementing the nonce appended to the ticket drawn from the past in order to attempt to craft a new valid `ElectionProof` and trying again.
 
-SPC provides a power table abstraction which tracks miner power (i.e. miner storage in relation to network storage) over time. The power table is updated for new sector commitments (incrementing miner power), when PoSts fail to be put on-chain (decrementing miner power) or for other storage and consensus faults.
+{{<label ticket_chain>}}
+### The Ticket chain and randomness on-chain
 
-An invariant of the storage power consensus subsystem is that all storage in the power table must be verified. That is, miners can only derive power from storage they have already proven to the network. 
+While each Filecoin block header contains a ticket field (see {{<sref tickets>}}), it is useful to provide nodes with a ticket chain abstraction.
 
-In order to achieve this, Filecoin delays updating power for new sector commitments until the first valid PoSt in the next proving period corresponding to that sector.
-Conversely, storage faults only lead to power loss once they are detected (up to one proving period after the fault) so miners will mine with no more power than they have used to store data over time.
+Namely, tickets are used throughout the Filecoin system as sources of on-chain randomness. For instance,
+- The {{<sref sector_sealer>}} uses tickets as SealSeeds to bind sector commitments to a given subchain.
+- The {{<sref post_generator>}} likewise uses tickets as PoStChallenges to prove sectors remain committed as of a given block.
+- They are drawn by the Storage Power subsystem as randomness in {{<leader_election>}} to determine their eligibility to mine a block
+- They are drawn by the Storage Power subsystem in order to generate new tickets for future use.
 
-Put another way, power accounting in the SPC is delayed between storage being proven or faulted, and power being updated in the power table (and so for leader election). This ensures fairness over time.
+Each of these ticket uses may require drawing tickets at different chain heights, according to the security requirements of the particular protocol making use of tickets. Due to the nature of Filecoin's Tipsets and the possibility of using losing tickets (that did not yield leaders in leader election) for randomness at a given height, tracking the canonical ticket of a subchain at a given height can be arduous to reason about in terms of blocks. To that end, it is helpful to create a ticket chain abstraction made up of only those tickets to be used for randomness at a given height.
 
-To illustrate this, an example:
+This ticket chain will track one-to-one with a block at each height in a given subchain, but omits certain details including other blocks mined at that height.
 
-Miner M1 has a provingPeriod of 30. M1 submits a PoST at height 39. Their next `provingPeriodEnd` will be 69, but M1 can submit a new PoST at any height X, for X in (39, 69]. Let's assume X is 67.
+It is composed inductively as follows. For a given chain:
 
-At height Y in (39, 67], M1 will attempt to generate an `ElectionProof` using the storage market actor from height 39 for their own power (and an actor from Y for total network power); at height 68, M1 will use the storage market actor from height 67 for their own power, and the storage market actor from height 68 for total power and so on.
+- At height 0, take the genesis block, return its ticket
+- At height n+1, take the heaviest tipset in our chain at height n.
+    - select the block in that tipset with the smallest final ticket, return its ticket
 
-## Repeated leader election attempts
+Because a Tipset can contain multiple blocks, the smallest ticket in the Tipset must be drawn otherwise the block will be invalid.
+
+```
+   ┌──────────────────────┐
+   │                      │
+   │                      │
+   │┌────┐                │
+   ││ TA │              A │
+   └┴────┴────────────────┘
+
+   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+                          │
+   │
+    ┌────┐                │       TA < TB < TC
+   ││ TB │              B
+    ┴────┘─ ─ ─ ─ ─ ─ ─ ─ ┘
+
+   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+                          │
+   │
+    ┌────┐                │
+   ││ TC │              C
+    ┴────┘─ ─ ─ ─ ─ ─ ─ ─ ┘
+```
+
+In the above diagram, a miner will use block A's Ticket to generate a new ticket (or an election proof farther in the future) since it is the smallest in the Tipset.
+
+## Drawing randomness for sector commitments
+
+Tickets are used as input to the SEAL above in order to tie Proofs-of-Replication to a given chain, thereby preventing long-range attacks (from another miner in the future trying to reuse SEALs).
+
+The ticket has to be drawn from a finalized block in order to prevent the miner from potential losing storage (in case of a chain reorg) even though their storage is intact.
+
+Verification should ensure that the ticket was drawn no farther back than necessary by the miner. We note that tickets can uniquely be associated to a given round in the protocol (lest a hash collision be found), but that the round number is explicited by the miner in `commitSector`.
+
+We present precisely how ticket selection and verification should work. In the below, we use the following notation:
+
+- `F`-- Finality (number of rounds)
+- `X`-- round in which SEALing starts
+- `Z`-- round in which the SEAL appears (in a block)
+- `Y`-- round announced in the SEAL `commitSector` (should be X, but a miner could use any Y <= X), denoted by the ticket selection
+ - `T`-- estimated time for SEAL, dependent on sector size
+ - `G = T + variance`-- necessary flexibility to account for network delay and SEAL-time variance.
+
+We expect Filecoin will be able to produce estimates for sector commitment time based on sector sizes, e.g.:
+`(estimate, variance) <--- SEALTime(sectors)`
+G and T will be selected using these.
+
+#### Picking a Ticket to Seal
+
+When starting to prepare a SEAL in round X, the miner should draw a ticket from X-F with which to compute the SEAL.
+
+#### Verifying a Seal's ticket
+
+When verifying a SEAL in round Z, a verifier should ensure that the ticket used to generate the SEAL is found in the range of rounds [Z-T-F-G, Z-T-F+G].
+
+#### In Detail
+
+```
+                               Prover
+           ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+          │
+
+          ▼
+         X-F ◀───────F────────▶ X ◀──────────T─────────▶ Z
+     -G   .  +G                 .                        .
+  ───(┌───────┐)───────────────( )──────────────────────( )────────▶
+      └───────┘                 '                        '        time
+ [Z-T-F-G, Z-T-F+G]
+          ▲
+
+          └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+                              Verifier
+```
+
+Note that the prover here is submitting a message on chain (i.e. the SEAL). Using an older ticket than necessary to generate the SEAL is something the miner may do to gain more confidence about finality (since we are in a probabilistically final system). However it has a cost in terms of securing the chain in the face of long-range attacks (specifically, by mixing in chain randomness here, we ensure that an attacker going back a month in time to try and create their own chain would have to completely regenerate any and all sectors drawing randomness since to use for their fork's power).
+
+We break this down as follows:
 
-In the case that no miner is eligible to produce a block in a given round of EC, the storage power consensus subsystem will be called by the block producer to attempt another leader election.
+- The miner should draw from `X-F`.
+- The verifier wants to find what `X-F` should have been (to ensure the miner is not drawing from farther back) even though Y (i.e. the round of the ticket actually used) is an unverifiable value.
+- Thus, the verifier will need to make an inference about what `X-F` is likely to have been based on:
+  - (known) round in which the message is received (Z)
+  - (known) finality value (F)
+  - (approximate) SEAL time (T)
+- Because T is an approximate value, and to account for network delay and variance in SEAL time across miners, the verifier allows for G offset from the assumed value of `X-F`: `Z-T-F`, hence verifying that the ticket is drawn from the range `[Z-T-F-G, Z-T-F+G]`.