Skip to content

Commit

Permalink
Godoc improvements (#299)
Browse files Browse the repository at this point in the history
  • Loading branch information
arajasek authored Jun 26, 2020
1 parent 3ab7900 commit ecce3a1
Show file tree
Hide file tree
Showing 11 changed files with 65 additions and 56 deletions.
2 changes: 1 addition & 1 deletion pieceio/pieceio.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ type CarIO interface {
// WriteCar writes a given payload to a CAR file and into the passed IO stream
WriteCar(ctx context.Context, bs ReadStore, payloadCid cid.Cid, node ipld.Node, w io.Writer, userOnNewCarBlocks ...car.OnNewCarBlockFunc) error

// PrepareCar prepares a car so that it's total size can be calculated without writing it to a file.
// PrepareCar prepares a car so that its total size can be calculated without writing it to a file.
// It can then be written with PreparedCar.Dump
PrepareCar(ctx context.Context, bs ReadStore, payloadCid cid.Cid, node ipld.Node) (PreparedCar, error)

Expand Down
24 changes: 12 additions & 12 deletions retrievalmarket/doc.go
Original file line number Diff line number Diff line change
@@ -1,40 +1,40 @@
/*
Package retrievalmarket implements the filecoin retrieval protocol.
Package retrievalmarket implements the Filecoin retrieval protocol.
An overview of the retrieval protocol can be found in the filecoin specification:
An overview of the retrieval protocol can be found in the Filecoin specification:
https://filecoin-project.github.io/specs/#systems__filecoin_markets__retrieval_market
The follow architectural components provide a brief overview of the design of
The following architectural components provide a brief overview of the design of
the retrieval market module:
Public Interfaces And Node Dependencies
While retrieval deals primarily happen off chain, there are some chain operations
that must be performed by a Filecoin node implementation. The module is intended to seperate
While retrieval deals primarily happen off-chain, there are some chain operations
that must be performed by a Filecoin node implementation. The module is intended to separate
the primarily off-chain retrieval deal flow from the on-chain operations related primarily
to payment channels, the mechanism for getting paid for retrieval deals.
As such for both the client and the provider in the retrieval market, the module defines a top level
public interface which it provides an implementation for, and a node interface that must be implemented
by the filecoin node itself, and provided as a dependency. These node interfaces provide a universal way to
talk to potentially multiple different filecoin node implementations, and can be implemented as using HTTP
by the Filecoin node itself, and provided as a dependency. These node interfaces provide a universal way to
talk to potentially multiple different Filecoin node implementations, and can be implemented as using HTTP
or other interprocess communication to talk to a node implementation running in a different process.
The top level interfaces this package implements are RetrievalClient & RetrievalProvider. The dependencies the filecoin
The top level interfaces this package implements are RetrievalClient & RetrievalProvider. The dependencies the Filecoin
node is expected to implement are RetrievalClientNode & RetrievalProviderNode. Further documentation of exactly what those
dependencies should do can be found in the readme.
Finite State Machines
While retrieval deals in general should be fairly fast, making a retrieval deal is still an asynchronous process.
As documented in the filecoin spec, the basic architecture of the filecoin retrieval protocol is incremental payments.
As documented in the Filecoin spec, the basic architecture of the Filecoin retrieval protocol is incremental payments.
Because neither client nor provider trust each other, we bootstrap trust by essentially paying in small increments as we receive
data. The client only sends payment when it verifies data and the provider only sends more data when it receives payment.
Not surprisingly, many things can go wrong along the way. To manage this back and forth asynchronous process,
we use finite state machines that update deal state when discrete events occur. State updates
always persist state to disk. This means we have a permaneant record of exactly what's going on with deals at any time,
and we can ideally survive our filecoin processes shutting down and restarting.
always persist state to disk. This means we have a permanent record of exactly what's going on with deals at any time,
and we can ideally survive our Filecoin processes shutting down and restarting.
The following diagrams visualize the statemachine flows for the client and the provider:
Expand Down Expand Up @@ -70,7 +70,7 @@ or by simply calling `ListDeals` to get all deal statuses.
The FSMs implement every remaining step in deal negotiation. Importantly, the RetrievalProvider delegates unsealing sectors
back to the node via the `UnsealSector` method (the node itself likely delegates management of sectors and sealing to an
implementation of the Storage Mining subsystem of the filecoin spec). Sectors are unsealed on an as needed basis using
implementation of the Storage Mining subsystem of the Filecoin spec). Sectors are unsealed on an as needed basis using
the `PieceStore` to locate sectors that contain data related to the deal.
Major Dependencies
Expand Down
8 changes: 4 additions & 4 deletions retrievalmarket/impl/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ func (c *Client) FindProviders(payloadCID cid.Cid) []retrievalmarket.RetrievalPe

/*
Query sends a retrieval query to a specific retrieval provider, to determine
if the provider can serve a retrieval request and what it's specific parameters for
if the provider can serve a retrieval request and what its specific parameters for
the request are.
The client a new `RetrievalQueryStream` for the chosen peer ID,
Expand All @@ -119,15 +119,15 @@ func (c *Client) Query(_ context.Context, p retrievalmarket.RetrievalPeer, paylo
}

/*
Retrieve initiates the retrieval deal flow, which involes multiple requests and responses
Retrieve initiates the retrieval deal flow, which involves multiple requests and responses
To start this processes, the client creates a new `RetrievalDealStream`. Currently, this connection is
kept open through the entire deal until completion or failure. Make deals pauseable as well as surviving
a restart is a planned future feature.
Retrieve should be called after using FindProviders and Query are used to identify an appropriate provider to
retrieve the deal from. The parameters identified in Query should be passed to Retrieve to insure the
greatest likelyhood the provider will accept the deal
retrieve the deal from. The parameters identified in Query should be passed to Retrieve to ensure the
greatest likelihood the provider will accept the deal
When called, the client takes the following actions:
Expand Down
2 changes: 1 addition & 1 deletion retrievalmarket/impl/provider.go
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ A Provider handling a retrieval `Query` does the following:
2. Look in its piece store for determine if it can serve the given payload CID.
3. Combine these results with it's existing parameters for retrieval deals to construct a `retrievalmarket.QueryResponse` struct.
3. Combine these results with its existing parameters for retrieval deals to construct a `retrievalmarket.QueryResponse` struct.
4.0 Writes this response to the `Query` stream.
Expand Down
8 changes: 4 additions & 4 deletions retrievalmarket/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ const (
// ClientEventPaymentSent indicates a payment was sent to the provider
ClientEventPaymentSent

// ClientEventConsumeBlockFailed indicates an error occurrect while trying to
// ClientEventConsumeBlockFailed indicates an error occurred while trying to
// read a block from the provider
ClientEventConsumeBlockFailed

Expand Down Expand Up @@ -227,7 +227,7 @@ type RetrievalClientNode interface {
GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error)

// GetOrCreatePaymentChannel sets up a new payment channel if one does not exist
// between a client and a miner and insures the client has the given amount of funds available in the channel
// between a client and a miner and ensures the client has the given amount of funds available in the channel
GetOrCreatePaymentChannel(ctx context.Context, clientAddress, minerAddress address.Address,
clientFundsAvailable abi.TokenAmount, tok shared.TipSetToken) (address.Address, cid.Cid, error)

Expand Down Expand Up @@ -292,7 +292,7 @@ const (
// ProviderEventDecisioningError means the Deciding function returned an error
ProviderEventDecisioningError

// ProviderEventWriteResponseFailed happens when a network error ocurrs writing a deal response
// ProviderEventWriteResponseFailed happens when a network error occurs writing a deal response
ProviderEventWriteResponseFailed

// ProviderEventReadPaymentFailed happens when a network error occurs trying to read a
Expand All @@ -318,7 +318,7 @@ const (
// trying to read the next block from the piece
ProviderEventBlockErrored

// ProviderEventBlocksCompleted happeds when the provider reads the last block
// ProviderEventBlocksCompleted happens when the provider reads the last block
// in the piece
ProviderEventBlocksCompleted

Expand Down
6 changes: 3 additions & 3 deletions storagemarket/dealstatus.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ const (
// StorageDealUnknown means the current status of a deal is undefined
StorageDealUnknown = StorageDealStatus(iota)

// StorageDealProposalNotFound is a status returned in responses when the deal itself cannont
// StorageDealProposalNotFound is a status returned in responses when the deal itself cannot
// be located
StorageDealProposalNotFound

Expand Down Expand Up @@ -37,10 +37,10 @@ const (
StorageDealSlashed

// StorageDealFailing means something has gone wrong in a deal. Once data is cleaned up the deal will
// finalize on StorateDealError
// finalize on StorageDealError
StorageDealFailing

// StorageDealFundsEnsured means we've deposited funds as neccesary to create a deal, ready to move forward
// StorageDealFundsEnsured means we've deposited funds as necessary to create a deal, ready to move forward
StorageDealFundsEnsured

// StorageDealWaitingForDataRequest means the client is waiting for a request for the deal data from the provider
Expand Down
55 changes: 32 additions & 23 deletions storagemarket/doc.go
Original file line number Diff line number Diff line change
@@ -1,37 +1,37 @@
/*
Package storagemarket implements the filecoin storage protocol.
Package storagemarket implements the Filecoin storage protocol.
An overview of the storage protocol can be found in the filecoin specification:
An overview of the storage protocol can be found in the Filecoin specification:
https://filecoin-project.github.io/specs/#systems__filecoin_markets__storage_market
The follow architectural components provide a brief overview of the design of
The following architectural components provide a brief overview of the design of
the storagemarket module:
Public Interfaces And Node Dependencies
A core goal of this module is to isolate the negotation of deals from the actual chain operations
A core goal of this module is to isolate the negotiation of deals from the actual chain operations
performed by the node to put the deal on chain. The module primarily orchestrates the storage deal
flow, rather than performing specific chain operations which are delegated to the node.
As such for both the client and the provider in the storage market, the module defines a top level
As such, for both the client and the provider in the storage market, the module defines a top level
public interface which it provides an implementation for, and a node interface that must be implemented
by the filecoin node itself, and provided as a dependency. These node interfaces provide a universal way to
talk to potentially multiple different filecoin node implementations, and can be implemented using HTTP
or other interprocess communication to talk to a node implementation running in a different process.
by the Filecoin node itself, and provided as a dependency. These node interfaces provide a universal way to
talk to potentially multiple different Filecoin node implementations, and can be implemented using HTTP
or some other interprocess communication to talk to a node implementation running in a different process.
The top level interfaces this package implements are StorageClient & StorageProvider. The dependencies the filecoin
The top level interfaces this package implements are StorageClient & StorageProvider. The dependencies the Filecoin
node is expected to implement are StorageClientNode & StorageProviderNode. Further documentation of exactly what those
dependencies should do can be found in the readme.
Finite State Machines and Resumability
Making deals in filecoin is a highly asynchronous process. For a large piece of data, it's possible from start
to finish the proposing a deal, transferring data, publishing the deal, putting the data in a sector and sealing it
Making deals in Filecoin is a highly asynchronous process. For a large piece of data, it's possible that the entire
process of proposing a deal, transferring data, publishing the deal, putting the data in a sector and sealing it
could take hours or even days. Not surprisingly, many things can go wrong along the way. To manage the process
of orchestrating deals, we use finite state machines that update deal state when discrete events occur. State updates
always persist state to disk. This means we have a permanent record of exactly what's going on with deals at any time,
and we can ideally survive our filecoin processes shutting down and restarting.
and we can ideally survive our Filecoin processes shutting down and restarting.
The following diagrams visualize the statemachine flows for the client and the provider:
Expand All @@ -41,13 +41,13 @@ Provider FSM - https://raw.githubusercontent.com/filecoin-project/go-fil-markets
Identifying Providers For A Deal
The StorageClient provides two functions locate a provider with which to store a deal:
The StorageClient provides two functions to locate a provider with whom to make a deal:
`ListProviders` returns a list of storage providers on the filecoin network. This list is assembled through
a query to chain state for active storage miners
`ListProviders` returns a list of storage providers on the Filecoin network. This list is assembled by
querying the chain state for active storage miners.
`QueryAsk` queries a single provider for more specific details about the kinds of deals they accept, as
expressed through a `StorageAsk`
expressed through a `StorageAsk`.
Deal Flow
Expand All @@ -66,11 +66,11 @@ From this point forward, deal negotiation is completely asynchronous and runs in
A user of the modules can monitor deal progress through `SubscribeToEvents` methods on StorageClient and StorageProvider,
or by simply calling `ListLocalDeals` to get all deal statuses.
The FSM implement every step in deal negotiation up to deal publishing. However, adding the deal to a sector and sealing
The FSMs implement every step in deal negotiation up to deal publishing. However, adding the deal to a sector and sealing
it is handled outside this module. When a deal is published, the StorageProvider calls `OnDealComplete` on the StorageProviderNode
interface (the node itself likely delegates management of sectors and sealing to an implementation of the Storage Mining subsystem
of the filecoin spec). At this point, the markets implementations essentially shift to being monitors of deal progression,
waiting to see and record when the deal becomes active and later expired or slashed.
of the Filecoin spec). At this point, the markets implementations essentially shift to being monitors of deal progression:
they wait to see and record when the deal becomes active and later expired or slashed.
When a deal becomes active on chain, the provider records the location of where it's stored in a sector in the PieceStore,
so that it's available for retrieval.
Expand All @@ -80,30 +80,39 @@ Major Dependencies
Other libraries in go-fil-markets:
https://github.com/filecoin-project/go-fil-markets/tree/master/filestore - used to store pieces and other
temporary data before its transferred to either a sector or the PieceStore
temporary data before it's transferred to either a sector or the PieceStore.
https://github.com/filecoin-project/go-fil-markets/tree/master/pieceio - used to convert back and forth between raw
payload data and pieces that fit in sector. Also provides utilites for generating CommP
payload data and pieces that fit in sector. Also provides utilities for generating CommP.
https://github.com/filecoin-project/go-fil-markets/tree/master/piecestore - used to write information about where data
lives in sectors so that it can later be retrieved
lives in sectors so that it can later be retrieved.
https://github.com/filecoin-project/go-fil-markets/tree/master/shared - types and utility functions shared with
retrievalmarket package
retrievalmarket package.
Other Filecoin Repos:
https://github.com/filecoin-project/go-data-transfer - for transferring data, via go-graphsync
https://github.com/filecoin-project/go-statemachine - a finite state machine that tracks deal state
https://github.com/filecoin-project/go-storedcounter - for generating and persisting unique deal IDs
https://github.com/filecoin-project/specs-actors - the Filecoin actors
IPFS Project Repos:
https://github.com/ipfs/go-graphsync - used by go-data-transfer
https://github.com/ipfs/go-datastore - for persisting statemachine state for deals
https://github.com/ipfs/go-ipfs-blockstore - for storing and retrieving block data for deals
Other Repos:
https://github.com/libp2p/go-libp2p) the network over which retrieval deal data is exchanged.
https://github.com/hannahhoward/go-pubsub - for pub/sub notifications external to the statemachine
Root Package
Expand Down
4 changes: 2 additions & 2 deletions storagemarket/impl/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -250,11 +250,11 @@ func (c *Client) GetProviderDealState(ctx context.Context, info storagemarket.St
}

/*
ProposeStorageDeal initiates the retrieval deal flow, which involes multiple requests and responses.
ProposeStorageDeal initiates the retrieval deal flow, which involves multiple requests and responses.
This function is called after using ListProviders and QueryAs are used to identify an appropriate provider
to store data. The parameters passed to ProposeStorageDeal should matched those returned by the miner from
QueryAsk to insure the greatest likelyhood the provider will accept the deal.
QueryAsk to ensure the greatest likelihood the provider will accept the deal.
When called, the client takes the following actions:
Expand Down
2 changes: 1 addition & 1 deletion storagemarket/nodes.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ type StorageProviderNode interface {
// GetMinerWorkerAddress returns the worker address associated with a miner
GetMinerWorkerAddress(ctx context.Context, addr address.Address, tok shared.TipSetToken) (address.Address, error)

// LocatePieceForDealWithinSector looks up a given dealID in the miners sectors, and returns it's sectorID and location
// LocatePieceForDealWithinSector looks up a given dealID in the miners sectors, and returns its sectorID and location
LocatePieceForDealWithinSector(ctx context.Context, dealID abi.DealID, tok shared.TipSetToken) (sectorID uint64, offset uint64, length uint64, err error)
}

Expand Down
2 changes: 1 addition & 1 deletion storagemarket/testnodes/testnodes.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ func (n *FakeCommonNode) AddFunds(ctx context.Context, addr address.Address, amo
return n.AddFundsCid, nil
}

// EnsureFunds adds funds to the given actor in the storage market state to insure it has at least the given amount
// EnsureFunds adds funds to the given actor in the storage market state to ensure it has at least the given amount
func (n *FakeCommonNode) EnsureFunds(ctx context.Context, addr, wallet address.Address, amount abi.TokenAmount, tok shared.TipSetToken) (cid.Cid, error) {
if n.EnsureFundsError == nil {
balance := n.SMState.Balance(addr)
Expand Down
Loading

0 comments on commit ecce3a1

Please sign in to comment.