Support Retrieval By Any CID, Not Just Root

[hannahhoward]

Goals

Provide the option for storage providers to maintain references to all CIDs in a payload and their corresponding sectors, so that the provider can serve retrievals for any CID in the payload, not just root.

Implementation

• Add configuration options to the provider constructor for optional configurations
• Add an option to "Enable Universal Retrieval" - this means allowing retrieval deals that do not just start at the root of the payload, but rather any CID
• Plumb the CAR file writers configurable hooks up through PieceIO
• Write a block recorder hook that uses the data from the CAR writer to record block locations, and write that to a temporary piece metadata file
• Add a function to record this metadata while generating CommP
• Modify provider code for commP generation to record metadata based on whether universal retrieval is enabled
• Transfer metadata if present to the PieceStore
• Generally improve deletion of temp files in the filestore -- there were several getting left around
• Also cleanup failure code to include temp file cleanup
• Add a TestFileStore to support filestore testing
• Refactor ProviderDealEnvironment to have less functions, provide direct access to filestore & piecestore

For Discussion

• Why record a second temp file? Well, the data needs to be persisted, and it seemed like the best way to keep it around while not consuming resources on deal steps it's not involved in. I though about putting the metadata into the deal itself -- but this mean it'd get loaded off disk on every deal step, which felt less than idea
• There's a lot of complicated interface plumbing here, and I wonder if there's a better way to do all this down the line, but for now, this is as straightforward as I could make it.

fixes #61

[rvagg]

Any security-related implications worth considering here? I'm thinking mainly of DoS style attacks where I make deals with you to store my CAR files where the blocks are small enough to just make unique CIDs, so your metadata storage takes up nearly as much space as the data itself. Would that be a reasonable attack scenario against a miner (what are the incentives for this?) and how would miners react against such attacks--would it mean that they would (could?) turn off indexing and we'd end up with fewer indexing miners for retrieval?
I'm sure the economics of such an "attack" are difficult to model but I reckon there are others that have more of a clue than I about it to think it through.

[hannahhoward]

@rvagg this PR is as much technical proof-of-concept as working system to be frank. It's something neither implementation (lotus or gfc) will likely turn on by default. There are actually many many product features to work out before it becomes widespread and something we make easy to enable. For example, how do I know as a client which miners support indexing? Especially if I want to make a deal with one that does. How do I know my miner is actually gonna build the index (I don't). Part of the system design is to intentionally separate storage and retrieval markets, so we need to figure out how this index is created and maintained, and by whom. Anyway, this is all to say, we needed to show it could be done, but making the feature usable at a product level is still a separate chunk of work. (in fairness, this is not spelled out in the PR description, and we've only been reminded of this as we began to socialize this new feature outside this team). Anyway though, rather than try to get it all right, we'd like to just implement the feature as is as a base for ongoing discussions.

[ingar]

Cool! One question: if a Provider turns on universal retrieval at some point in the future, do we just error out if we don't have the metadata for their retrieval?

[rvagg]

@hannahhoward sorry, not intending to indicate I have a blocking objection, just interested in the general discussion and 👍 to the path forward of just getting something implemented as part of the path forward.

[hannahhoward]

@rvagg no it's totally valid, we've realized during discussions that there are a lot of product questions surrounding this feature, and that we need to escalate those concerns to the wider team :)