Standardize Storage AskProtocol v2

[jennijuju]

^{Originally posted by @brendalee, edited by @jennijuju December 13, 2021}
Starting this issue to get additional feedback from the community on proposed updates to the storage AsksProtocol v1.1.0. Currently, the ask protocol does not give the client / SP enough information on the likelihood of a storage deal being accepted. Note that there have already been a handful of discussions around storage ask protocol changes with the community (notably in Filecoin Slack #arg channel and other ad-hoc discussions with storage providers).

The goal of AskProtocol v2 is to help with awareness between the client and the storage provider before a deal proposal. Before a client attempts to make a deal, they can send details of their proposed deal, and the storage provider can respond to the proposal (with accept or reject and other details such as price). Note that the ask protocol is not designed to handle back and forth deal negotiations.

Proposed Storage AskProtocol v2 Standard

This proposal is to request change to the current ask protocol and update the version of AskProtocolID to /fil/storage/ask/1.1.0, and

Storage AskProtocol v2 Sketch

We need to design a new storage ask protocol. Let's brainstorm what it might look like here.
I think we should keep it very simple still, with a client requesting, and the storage provider responding.
To get started, i'm just going to define a request and a response type, and then we can assume thats the protocol.

Request

type StorageAskRequest struct {
  ClientAddress Address
  PeerId  *abi.PeerID //nullable
  PieceSize abi.PaddedPieceSize
  Duration int64
  //Deal start epoch
  StartEpoch abi.ChainEpoch 
  Verified bool
  TransferProtocol string
  Message string

Signature: The request should be signed by the client.

Open question:

• Should we have `fast retrieval for the request, given that FVM may enable retrievalbility reaasurance?
• Do we need a Message field as string as a free-form field set by the client to the storage provider?
• Do we need the client to provide the payload CID?

Response

The response should be a signed object containing the request, for context

type StorageAskResponse struct {
  Query StorageAskRequest
  
  // WillAccept denotes whether or not the storage provider will accept the proposed deal
  WillAccept bool
  
  // RejectionCode should projects the reection reason
  RejectionCode uint
  
  // Price per GB per epoch. This is a single price field, verified vs unverified is specified by the Query
  Price abi.TokenAmount
  
  // Expiry denotes how long this offer is valid for
  // This AskResponse is not binding in any way, this is just for the clients convenience.
  Expiry time.Time
  
  //DealStartEpoch is how soon miner offers to get the deal fully onchain (provecommited) 
  DealStartEpoch abi.ChainEpoch
  
  // Message is a free-form field set by the storage provider to the client
  Message string
}

Open Questions:

• RejectionCode: Should we keep the code simple and have a predefined list of error codes, or should it to be a string for customized rejection reason in here. Or should it be a structured field? or left open for people to do whatever they like?
• Miner response can contain an epoch that they will try to get the published deal on chain

[jennijuju]

Initial FIP draft created at #224

Simple Summary

The goal of AskProtocol v2 is to help with the discovery between the client and storage providers before making a storage deal proposal. As the first step of the SP discovery, clients can send details of their proposed deal to the storage provider, and the storage provider should be able to respond to the proposal (with accept or reject, price and additional service offer). Note that the ask protocol is not designed to handle back and forth deal negotiations.

Abstract

AskProtocol v2 aims to improve the existing AskProtocol to help the client understand whether or not a storage provider is likely to accept their deal before they send a signed proposal.

The proposed flow is simple. The client sends an ask request with information about the storage deal they would like to make. The ask response by the storage provider indicates whether or not the deal will be accepted with specifics on conditions (for example, how long the offer is valid for, and what the price to store is). More details about the AskProtocol v2 request/response can be found in the specification section.

Change Motivation

Clients need to be able to share details of the storage deal they want to make. Furthermore, they need to know what capabilities and features a Storage Provider supports, so as to understand what Storage Providers they can successfully make a deal with.

Specification

Client Request of AskProtocol v2

For the AskProtocol v2 request, the client provides the following information:

type StorageAskRequest struct {
  ClientAddress Address
  PeerId  *abi.PeerID 
  PieceSize abi.PaddedPieceSize
  Duration int64
  StartEpoch abi.ChainEpoch 
  Verified bool
  TransferProtocol string
  Message string

where:

• ClientAddress: is the wallet/account address that the client will be used for signing the deal proposal and making/paying the storage deal
• PeerId: is the peerID of the client's node, this is an optional field
• PieceSize: is the padded size of the piece in the deal
• Duration: is the deal duration
• StartEpoch: is the start epoch of the deal
• Verified: if the deal is a verified deal
• Transfer protocol: is the protocol that will be used for data transfer, for example, graphsync, physical. Aside from a list of defined the protocols , the client/storage provider may have the option to specify/add additional transfer protocols
• Message: is a free-form field set by the client to the storage provider
  Signature: the request should be signed by the client address

Storage Provider Response of AskProtocol v2

For the response, the provider will respond with the following information:

type StorageAskResponse struct {
  Query StorageAskRequest
  WillAccept bool
  RejectionCode uint
  Price abi.TokenAmount
  Expiry time.Time
  DealStartEpoch abi.ChainEpoch
  Message string
}

where:

• Query: is the StorageAskRequest that was sent by the client
• WillAccept: is whether the storage provider will accept or reject the deal
• RejectionCode: is codes for rejection reasons
  • Need clearly defined rejection codes
  • Start with HTTP codes and build from there
  • Option for storage provider to specify additional rejection codes
• Price: is the offering price per GB per epoch.
• Expiry: is how long the offer is valid for, this is not a binding information but just for client's information
• DealStartEpoch: is how the storage provider is offering to the the deal fully committed to the network with in a sector
• `Message: is a free-form field set by the storage provider to the client

Design Rationale

TBD

Backwards Compatibility

TBD

Test Cases

TBD

Security Considerations

TBD

Incentive Considerations

N/A

Product Considerations

One of the main experiences we want to improve is storage dealmaking and overall Filecoin storage reliability. AskProtocol v2 improvements aim to help with the following:

• Better information on Storage Provider capabilities before client attempts to make a storage deal
• Easier to filter which Storage Providers will accept a client’s storage deals
• Increase storage deal acceptance and success rate

Implementation

TBD

Copyright

Copyright and related rights waived via CC0.

[brendalee]

Some discussion points -

Ask request

• Flagging that fast retrieval has been removed, any concerns with this?
• PeerID - part of the libp2p protocol, SP should already have this information (is this redundant?)

Ask response

• DealStartEpoch - this field is for the SP to share when the deal start epoch could be, and should also be able to encompass if the SP wants the client to come back at a later time. How would the SP calculate this (we should have a standard that we push to SPs so it’s not interpreted differently)? - @whyrusleeping thoughts on this?
• RejectionCode - @dirkmc feedback
  HTTP doesn’t map very well, would be good for us to delinate a list of error codes and use those.
  Currently this is a number, would it be better if this is a string
• Thoughts about including option for SP to include client collateral requirement (https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/market/deal.go#L23)

[hunjixin]

In venus-market, we have gathered the needs of many miners and miners. access the venus-market address directly during the ask process (market url obtained by accessing a fixed contract/config in the future), and return a series of miners and the miner’s demand, and then let the client choose a miner to issue an order.

[steven004]

I would like to put @hunjixin 's thought into more general.

In Filecoin ecosystem, the storage provider just part of the network infrastructure, above that, there could be higher layer services, e.g. deal broker, gateway or proxy, to help on storing data to Filecoin and retrieval from Filecoin network too. In addition, when we consider retrieval, there could be services particularly for retrieval too, that means, the fast retrieval could be done by other service nodes.

And all these services bridging between the clients and storage providers could be possibly provided by smart contracts when FVM is available.

In the v2 design, should we make it more flexible to not limit to storage provide nodes and end clients, or we design a delegate or trust protocol for the market in another FIP, while having this protocol support it?

[brendalee]

Thanks for the feedback, @steven004 and @hunjixin, the aggregation idea is a good one, and is definitely something that we can consider at a later stage. For the current proposed AskProtocol v2, it's intentionally scoped down so we can quickly enable storage providers to provide information on why they would reject a deal.

[anorth]

I would love to evaluate this proposal, but can't do much without much more description of the perceived problems, motivation for specific changes, and design rationale.

[brendalee]

Thanks! Adding some more information on change motivation -

With the existing StorageAsk, only storage providers are able to share details on what parameters by which they will accept or reject a deal. In the existing state, it is a precondition for clients to adhere to these parameters, but storage providers can still reject deals. Clients are not able to share details of their proposed storage deal, nor can they get additional information about why their storage deal was rejected.

With AskProtocol v2, clients can initiate to share details of the storage deal they want to make. Storage providers then have detailed information about potential storage deals, and can specify for a particular proposed deal, if it will be accepted, and if not, provide a rejection reason. In addition, clients can better understand what capabilities and features a Storage Provider supports, and see what Storage Providers they can successfully make a deal with.

For design rationale, the goal was to make the update lightweight and enable a simple request/response from a client to a storage provider. @whyrusleeping @jennijuju @ribasushi @dirkmc for additional details on design rationale.

[s0nik42]

@anorth , Two of the main reasons of this proposal are to allow SPs to provide tailor price/acceptance criteria per client AND more granular pricing.

The current Ask protocol can be consider as a public price (same conditions for all clients) with a very limited structure.

1/ Granular pricing :
With AskV1, a 2MiB deal for 6 months is charged at the same price/size/epoch than a 32GiB for 18 months.
=> So miners tend to only accept large deals. With AskV2, miners can give a price based on any attributes of a deal and should be able to accept small deals.

2/ Per client pricing :
What we observe is that SPs need to : set price per client; prioratize their client; set hourly deal rate, etc...
AskV2 solve that problem as well, allowing client to be inform of the deal price and acceptance prior placing the deal and in a standerdize way.

[anorth]

Thanks these are good context, and should appear in the proposal.

[anorth]

Another thing we should add into a revised ask protocol is an identifier (address) of the market contract for which the ask is intended.

The FVM and related protocol changes will eventually support the development of alternative storage markets – contracts that play the same role as, and compete with, the built-in storage market. In the near term those markets may rely on some other out-of-band channel for communicating asks/offers, but to the extend that the Filecoin protocol specifies a standard protocol, we should expand it to support arbitrary alternative market protocols.

The very minimum for this would be to add the address of the contract implementing the storage market protocol which the client wishes to use for the deal. Today that would be the address of the built-in storage market actor.

There is lots more we could do to better support alternative markets. One possibility would be to add a string field which could contain arbitrary JSON describing market-specific properties of the intended deal (e.g. required collateral, or whether the client intends to pay upfront, ...). We can probably do better than that, but don't have enough experience with alternative markets to know concretely how.

Note that we'll need to make a similar change to ClientDealProposal in order that a deal proposal intended for the built-in market cannot be "replayed" to a different market contract.

[jennijuju]

I think that could be ask protocol v3 once user deployed contract interfaces are standardized later on - I don’t think extra field for potential changes brought in by fvm should be considered in this iteration necessarily.
The smart contract address in proposal is kinda useless without specifc information for the individual market interface.

Cc @brendalee

[momack2]

Another thing to think about for the client deal proposal - does the client want their data to be publicly accessible to other network participants (and therefore if it's FIL+ data, anyone can audit the SP's responsiveness to public retrieval requests), or do they want their data to be private (and they're likely encrypting their data / asking SPs to only serve retrievals to a subset of addresses). SPs might offer different prices based on this (whether they're going to pre-pay the cost of retrievals).

[anorth]

After better understanding the ask and the deal request protocols, I think that either (a) these should become the same thing, or (b) the existing model of advertising a public price might be fine, and the more detailed negotiation is better left to the deal protocol.

The proposed StorageAskRequest is very close to a DealProposal. The fields that it omits (primarily collateral) could be argued should be included for the same reason that the other fields have been: yet another term which the provider might reject. The proposal looks like "pretend to do the deal to see if it works, then do the deal".

Rather than add a duplicative proposal ahead of the deal request, why not just add some better information to the deal proposal response, much like the proposed StorageAskResponse? Then a client which makes a proposal with some term that the miner rejects will have the information to make a new, more acceptable proposal. But in the case the miner would accept the deal... boom, they already have! What role does the ask protocol have at all?

One objection I can forsee to this proposal is that teams have metrics around improving "deal success rate", and by offloading a rejection to the ask protocol it could be claimed that these are not deal rejections. I don't think clients would care much for this distinction if the upshot is they have to shop their deal around anyway, and now it has two ways to fail. A simpler protocol will be easier for all parties to implement and understand, and we should not let goals for particular metrics wag the dog into a more complex protocol.

In light of this I think the ask protocol is intended to provide exactly that information that does not depend on client specifics like the piece CID, and would be suitable for aggregation so that many clients could consult one source for a view of the market. I expect there is a ton of room to improve this protocol by allowing the provider to specify much more detail about timeframes, sizes, collateral bounds, and different prices they might offer for different values thereof. But not in response to a specific client proposal – that's just a deal request and we have a protocol for that.

[anorth]

Another possible objection to my proposal of just enhancing the deal protocol itself with better response information is that maybe clients need to understand whether a provider will accept a deal without committing to it. That is, that the provider could respond "yes" but the client change its mind and pull out. I can't think of a strong case for that, but would welcome more information about why that's a significant need.

[dirkmc]

@anorth I'm sympathetic to your point that it's better not to have two slightly different ways to get the same information.

Having said that your second comment is correct, there is a a cost to the client when making a deal proposal: the client must reserve funds for the deal before making the deal proposal.

[brendalee]

I think the main scenario where this might be an issue is if a client is looking at pricing options / restrictions from various storage providers, and then selecting their top x SPs that they want to work with.

[dirkmc]

Is there a strong use case for a client wanting to get relatively high assurance about the possibility of a deal but then back away from actually doing it?

To me it seems strange to ask the client to commit funds for a deal at the informational stage where they just want to know how much it's going to cost them. The typical flow today is that the client checks the price right before they want to make a deal, but it's not hard to imagine other use cases where clients want to know the price without the immediate need to make a deal.

[anorth]

Sure, I totally get both of those points. But can we make the Ask protocol better, providing the pricing options/restrictions, without making it interactive and specific to the client? Quoting myself: "allowing the provider to specify much more detail about timeframes, sizes, collateral bounds, and different prices they might offer for different values thereof". Can we format an Ask response that answers the client's specific question for most reasonable values of the size, duration etc?

If we make the response client-specific, we can never create an aggregator that will save the client the work of discovering and querying a ton of providers. I think the ask protocol was design with aggregation in mind, it just hasn't been built yet.

[dirkmc]

Right that makes sense 👍

[s0nik42]

If we make the response client-specific, we can never create an aggregator that will save the client the work of discovering and querying a ton of providers. I think the ask protocol was design with aggregation in mind, it just hasn't been built yet.

Comments on aggregators :

1. Aggregator should aggregate what we can call Storage-provider Public Price. We cannot avoid that tailor pricing exist in every business and will exist on Filecoin as well and will be out of aggregator's scope. But we should adress it anyway this is a strong request.

2. With the current lotus implementation storage providers as no way to get tailor pricing and public pricing coexisting on the same miner, because the lotus pricing (get-ask) apply before the deal filter is triggered (used to manage tailor pricing).

3. The current get-ask is not granular enough to provide a acceptable public pricing. As an exemple, we will never find a Storage-provider accepting deals at the same price for a small or a large deal.

Rather than add a duplicative proposal ahead of the deal request, why not just add some better information to the deal proposal response, much like the proposed StorageAskResponse?

To me, I'm fine avoiding a duplicative proposal and provide more information in StorageAskResponse. This solution will solve a lot issues and help building reputation system in the short term.
I also want to add that requesting a "quote" prior "ordering" feel more natural to me when buying goods, but I understand that it's more complex to manage.

[brendalee]

Some notes from discussion with @whyrusleeping and @jacobheun

Observations from Estuary and Estuary needs:

• Currently SPs are using things outside the ask protocol to make decisions on whether or not they will accept a deal. It is frustrating to get an ask response, propose a deal that meets those requirements, then still not get a deal accepted.
• A deal proposal includes a signed statement, and since deal rejection is not on chain, the signature is not invalidated. Ideally, if I propose a deal that is perfectly valid, an SP rejecting that deal which matches the ask should be considered a significant error.
• For backoffs, we can have asks be frequently updated with timing information. For example, an SP could specify that the ask is valid for 1 hour, and what the start date will be if a deal is proposed within that hour.

[anorth]

Ideally, if I propose a deal that is perfectly valid, an SP rejecting that deal which matches the ask should be considered a significant error

I sort of get this, but also not sure. SPs need to be able to, e.g., reject content according to specific deny lists etc.

One party has to commit first, and we chose the client. If we changed things so the provider committed first, they would need all the information in a DealProposal anyway (just without the signature), and so I don't think the flow would change very much. Clients would still use an Ask protocol to discover SPs, then send concrete proposals to them, and be rejected sometimes (and then have to sign and post on chain). But rejections would be much rarer if Ask protocol provided richer requirements, which is the totally valid motivation behind this proposal.

I think mechanisms where the SP commits to storage without knowing the data will be quite plausible in the future, probably in alternative markets on the FVM. We do want to enable this as a "capacity deal" or "storage future".

Time-bounds on the ask information sounds good.

[flyworker]

Recently we are suffered from SP change price after get query, it takes us about 10m ~2h to update the sp's price, so if they change the deal during the timeslot, our deal sending will be failed.

We also noticed that if SP use CID Gravity, they might setup different policy for different file size which make they behavior unpredictable.

I think we need a way to sync up with this dynamic price change behavior.