Batch published deals do not get packed into sectors as expected

[f8-ptrk]

regarding:

https://filecoinproject.slack.com/archives/G01KU7G441Y/p1618134221295100

when batch publishing deals MaxWaitDealsSectors needs to be set to 1 if a miner wants all published deals in the same sector. as soon as MaxWaitDealsSectors > 1 the deals get spread out over [up to] MaxWaitDealsSectors sectors.

Expected/wanted behavior:

lotus-miner packing deals into sectors as it can find space without creating [up to] MaxWaitDealsSectors sectors.

[edit]

as reference about how this behaves:

https://filecoinproject.slack.com/archives/G01KU7G441Y/p1616110446225200?thread_ts=1615892104.004200&cid=G01KU7G441Y

[neondragon]

Steps to reproduce:

Set MaxWaitDealsSectors to 2
Receive 4 1GiB storage deals
lotus-miner version 1.6.0+mainnet+git.3cfb6b09d.dirty

Expected

1 WaitDeals sector created containing all four deals. Efficient packing.

WaitDeals [ 1GiB, 1GiB, 1GiB, 1GiB ]

Actual

2 WaitDeals sectors created. Inefficient use of sector capacity.

WaitDeals [ 1GiB, 1GiB ]
WaitDeals [ 1GiB, 1GiB ]

[rjan90]

I'm experiencing the same thing on v1.6.0. Ideally it would package as many deals as possible into one sector before it spawns up the the next WaitDealsSectors.

One problem with spreading the batch-publishing deals out over the MaxWaitDealsSectors-amount, is that you could end up with sectors in wait deals-state with a couple of smaller deal sizes (e.g 1GiB) spread across them. And when you get a full-sector deal, it will be stuck until you start manually sealing one of the sectors.

Lotus version

Daemon:  1.6.0+mainnet+git.3fc23a785+api1.1.0
Local: lotus version 1.6.0+mainnet+git.3fc23a785

[jennijuju]

@f8-ptrk can you please summarize the slack conversation instead of linking a thread?

[magik6k]

Ok, this is a bug.

The fix should be relatively easy - storage fsm input should be able to queue pieces to be added to a sector - right now if a piece is being added to a sector (the sector is in the AddPiece state), we'll always start a new sector if we can, which results in this behavior.

Not sure if this should be configurable, I think that packing as much data as we can into sectors is always better - and if, for example, we have 2 32G pieces coming to a 32G miner at once, we'll still start 2 sectors to process them in parallel

[ribasushi]

@f8-ptrk can you please confirm this got resolved from your perspective?

[f8-ptrk]

have some deals?

@ribasushi maybe ask in minerX for people who have a bunch of deals going on. we are basically dry for a month now on the deal side

[edit]

already forwarded it

[rjan90]

Just sent Reiers 20+ small deals, and he can confirm that it works: https://filecoinproject.slack.com/archives/C022ZR4JA1M/p1623237962391300?thread_ts=1623157725.356700&channel=C022ZR4JA1M&message_ts=1623237962.391300