dagstore: quota allocation for transients directory and watermark-based GC

[raulk]

Context

The dagstore fetches unsealed pieces from the storage subsystem of Lotus into the transients directory whenever it needs to serve a retrieval. Once the retrieval completes, the refcount on the transient decreases, and when it's down to 0, it becomes eligible for reclaim (i.e. deletion) by the GC algorithm.

GC currently runs every minute by default (frequency can be configured through the [Dagstore] section of config.toml), effectively pruning away unreferenced files almost immediately.

This strategy is fail-safe, but it's rather inefficient:

1. if a new retrieval deal for the same piece comes shortly thereafter, it'll need to be refetched from the storage cluster.
2. makes inefficient use of local storage.
3. it produces more filesystem churn than necessary.
4. the transients directory can grow unbounded when faced with high retrieval concurrency.

Proposal

1. Turn the transients directory into an LRU/LFU cache, with an explicit quota assignment (e.g. 100GiB) so that storage providers can place a limit on how much the transients directory is allowed to take up.
2. Manage the usage through a mechanism of watermarks, e.g. prune LRU/LFU transients when usage goes above N threshold.
3. Park/queue shard acquisitions with insufficient quota.

References

filecoin-project/dagstore#65

[rjan90]

Closing this issue as outdated on the lotus-miner side of things. This issue is related to the legacy markets code in the Lotus-Miner which will be deprecated.

Boost has been working on designing, building, and testing a new Local Index Directory which will replace the dagstore which this issue is related to. X-posting the message from their announcement channel:

📣 Coming soon: Yugabyte database for the new Local Index Directory (LID) 📣 **
As you may know, we have been working on designing, building, and testing a new Local Index Directory which will replace the dagstore. Many of you have reported problems with the dagstore, and migrating to use LID will help resolve these issues. The LID introduces a horizontally scalable backend database for storing the data - YugabyteDB. We would love to get your input and feedback on these changes - feel free to raise them in Boost discussion or reach out in this slack channel.

How this benefits SPs!

• We are no longer relying on the dagstore!
  • Increased visibility into the state of your sectors and indexes
• Improved coordination between Lotus and Boost (Lotus will also leverage YugabyteDB)
  • built-in UI and admin panel for YugabyteDB for easy management
• Enables scalability for SPs storing client data.
  • Read more details on how this works: https://boost.filecoin.io/experimental-features/local-index-directory
  • Ensuring your data can be reliably retrieved will become increasingly more important as the ecosystem continues pushing on retrieval incentives and reputation systems, both of which have teams actively working on them.

What’s next?

• The Boost team has completed migration of our two production miners to LID, and we will be monitoring for any issues that come up. 📊
  • We will work with other early adopter SPs and Lotus TSEs to try this out! 🎉
• We are working on documentation and migration tools, and will announce when it is ready for all SPs to migrate to LID.
• Again, we’d love to hear directly from you. Please let us know your input and feedback in Boost discussion, or reach out in #boost-help.