Extract Lotus CommP writer and use when calculating CommP in markets #4899
Labels
area/markets
Area: Markets
dif/medium
Prior development experience with lotus is likely helpful
kind/enhancement
Kind: Enhancement
P0
P0: Critical Blocker
What
Currently when we calculate commP in go-fil-markets we pass an io.Reader of the whole piece to the FFI. The FFI seems to write a temp file the size of the whole piece to the OS temp directory during the process of calculating CommP. Instead of using the FFI directly, we can use Lotus's commP writer: https://github.com/filecoin-project/lotus/blob/master/lib/commp/writer.go which chunks up calls to the FFI when writing tempfiles so that it doesn't write a file the size of the whole piece to OS temp directory, which may not have a lot of space in a miner setup for large pieces.
How
extract lib/commp writer from lotus -- I think it would be nice to make a new shared repo that both go-fil-markets and lotus use -- also it may make sense to actually extract pieceIO from go-fil-markets as well and combine it. Though perhaps just extracting the commP writer as a start from lotus is a good place to be.
on top of Remove temp files go-fil-markets#449, rewrite GeneratePieceCommitment in PieceIO to use the commP writer. This will probably entail not re-using the GeneratePieceReader function in GeneratePieceCommitment any more, and instead calling PrepareCar directly and passing the extract commP writer directly to PreparedCar.Dump (then calling writer.Sum)
The text was updated successfully, but these errors were encountered: