Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sector that had unsealedInfoMap problem now stuck in RecoverDealIDs #3446

Closed
damonhung opened this issue Sep 1, 2020 · 4 comments · Fixed by #5658
Closed

Sector that had unsealedInfoMap problem now stuck in RecoverDealIDs #3446

damonhung opened this issue Sep 1, 2020 · 4 comments · Fixed by #5658

Comments

@damonhung
Copy link

damonhung commented Sep 1, 2020 •
edited
Loading

Continuing the thread starting with:

  1. Stuck in WaitDeals with sector not in unsealedInfoMap.infos (https://filecoinproject.slack.com/archives/C0179RNEMU4/p1598730762404500)
  2. Workaround committed here for v0.5.7

Looks like v0.5.7 didn't get me completely healthy. I ran sectors seal on both of my not in unsealedInfoMap sectors. One sector went through packing and then proceeded to Removed but the other got stuck in RecoverDealIDs. But it's not actually recovering deal IDs. It's endlessly retrying:

35.     2020-08-31 21:49:32 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}

which produces:

WARN    sectors storage-sealing/states_failed.go:297    getting deal 45203 for piece 0: deal 45203 not found

for each deal in that sector. That makes sense, but then it doesn't get any further. Running sectors remove has no effect.

I yolo'ed it with update-state ... Removing, which understandably freaked out the FSM (ERROR: normal shutdown of state machine) but seemingly resolved upon restarting the miner.

Here is the sector log:

Status:         Removed
CIDcommD:       <nil>
CIDcommR:       <nil>
Ticket:
TicketH:        0
Seed:
SeedH:          0
Precommit:      <nil>
Commit:         <nil>
Proof:
Deals:          [45203 45354 0 0 0 0 0 0 0 0 0 45713 45816 46117 46203 0 0 0 0 0 0 0 0 0 0 46420 0 0 0 0 0 0]
Retries:        0
--------
Event Log:
0.      2020-08-29 11:10:30 -0400 EDT:  [event;sealing.SectorStart]     {"User":{"ID":8,"SectorType":3}}
1.      2020-08-29 11:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqhfrmjhppizsdqy7wbyrrtz7u2hr6lc6cq6f6l4vcgbysivm4mmmq"}},"DealInfo":{"PublishCid":{"/":"bafy2bzaceaklw5p5jmqdxqyvssl7ohodvw5bjzvhcmy2kf5qibejfmbx3icku"},"DealID":45203,"DealSchedule":{"StartEpoch":16441,"EndEpoch":717164},"KeepUnsealed":true}}}}
2.      2020-08-29 11:23:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqkps6ngwwjp5c6ehspwbnygnvqnpp3wpti6gi2xmpluzd7hjep6ay"}},"DealInfo":{"PublishCid":{"/":"bafy2bzacedmqifjw2fudbevivoaapvkllfyi3nd7ekscn3zp3g5rcfsyuowok"},"DealID":45354,"DealSchedule":{"StartEpoch":16474,"EndEpoch":717164},"KeepUnsealed":true}}}}
3.      2020-08-29 12:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":262144,"PieceCID":{"/":"baga6ea4seaqhklmwsp5bm5jehfkhnyyxvgcyb4aji6x3piyfidlclkjjdtasuby"}},"DealInfo":null}}}
4.      2020-08-29 12:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":524288,"PieceCID":{"/":"baga6ea4seaqhaixwb57pnlp2c4ixuutbtyym5kbmnadvvxy4mz3yn3cqn3xs2gi"}},"DealInfo":null}}}
5.      2020-08-29 12:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":1048576,"PieceCID":{"/":"baga6ea4seaqntgehxfzvoouw4ejzgzcsg3axwh2moa2noi6hvgpxbg5u3jqrmky"}},"DealInfo":null}}}
6.      2020-08-29 12:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":2097152,"PieceCID":{"/":"baga6ea4seaqnbnjq3oylj4s4luxsukg752aiwu2bfibjghyyysm7lisubbvrgjq"}},"DealInfo":null}}}
7.      2020-08-29 12:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":4194304,"PieceCID":{"/":"baga6ea4seaqijqccdoqgqwqbx54vui2eazh6ijf5kku5eq3xwokp6tclivuoqei"}},"DealInfo":null}}}
8.      2020-08-29 12:10:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":8388608,"PieceCID":{"/":"baga6ea4seaqgl4u6lwmnerwdrm4iz7ag3mpwwaqtapc2fciabpooqmvjypweeha"}},"DealInfo":null}}}
9.      2020-08-29 12:10:31 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":16777216,"PieceCID":{"/":"baga6ea4seaqkejdvbaufquewln7dgszre6ymaqvr2bdnyvcaee3we7gypgoocoq"}},"DealInfo":null}}}
10.     2020-08-29 12:10:32 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":33554432,"PieceCID":{"/":"baga6ea4seaqnv7nlnwutmrctyjwtg4tlt7x6gq56r6awjhwabgvnh6x7kbqxkca"}},"DealInfo":null}}}
11.     2020-08-29 12:10:33 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":67108864,"PieceCID":{"/":"baga6ea4seaqnsqov4dldcsuzlqz77pkpxzurddlt2ts72lgtd4hxzbxl3ukoobq"}},"DealInfo":null}}}
12.     2020-08-29 12:10:36 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":134217728,"PieceCID":{"/":"baga6ea4seaqawdng4p7xmrn6q6i4h4ikzrd4sd7ncvofdoqrl66tl522uoqjagy"}},"DealInfo":{"PublishCid":{"/":"bafy2bzacea4mwejs5tqusuv4yepzzjsb36h6vkrwvoyphgibj3d6yvadsdi6a"},"DealID":45713,"DealSchedule":{"StartEpoch":16568,"EndEpoch":717164},"KeepUnsealed":true}}}}
13.     2020-08-29 12:28:03 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":134217728,"PieceCID":{"/":"baga6ea4seaqak2qiybfx7zpezpn6xhi4gjuf42dy7bz3fizri4nqalqj5g5v2ia"}},"DealInfo":{"PublishCid":{"/":"bafy2bzacedyy5oqbglreskk5qasuez3qbo2vmgpd5lgelf73th4wx3c4uwqdc"},"DealID":45816,"DealSchedule":{"StartEpoch":16594,"EndEpoch":717164},"KeepUnsealed":true}}}}
14.     2020-08-29 13:13:02 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":134217728,"PieceCID":{"/":"baga6ea4seaqfk3yukuzihv5vmxchfm2itfvfe5b6dyiaqzlwz2sfglh4it7aidy"}},"DealInfo":{"PublishCid":{"/":"bafy2bzacedpekrb5hkbcobsr322coumjwhby7odyqjk77mennkccxe5anemsq"},"DealID":46117,"DealSchedule":{"StartEpoch":16686,"EndEpoch":717164},"KeepUnsealed":true}}}}
15.     2020-08-29 13:24:00 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqdihqz5ahlocl57gpi5c5h3gwb3o4f7ncph6rnjhadk5uexo2aeay"}},"DealInfo":{"PublishCid":{"/":"bafy2bzaceb7c5jcznkczvnynmiiboabmyijnds22fxsv6zmztszpv2x7wzr5c"},"DealID":46203,"DealSchedule":{"StartEpoch":16714,"EndEpoch":717164},"KeepUnsealed":true}}}}
16.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqi5hreap5iqthwen7wbxzf7a7oidokt3mht23pmnjncuee6wwq2py"}},"DealInfo":null}}}
17.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":262144,"PieceCID":{"/":"baga6ea4seaqhklmwsp5bm5jehfkhnyyxvgcyb4aji6x3piyfidlclkjjdtasuby"}},"DealInfo":null}}}
18.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":524288,"PieceCID":{"/":"baga6ea4seaqhaixwb57pnlp2c4ixuutbtyym5kbmnadvvxy4mz3yn3cqn3xs2gi"}},"DealInfo":null}}}
19.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":1048576,"PieceCID":{"/":"baga6ea4seaqntgehxfzvoouw4ejzgzcsg3axwh2moa2noi6hvgpxbg5u3jqrmky"}},"DealInfo":null}}}
20.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":2097152,"PieceCID":{"/":"baga6ea4seaqnbnjq3oylj4s4luxsukg752aiwu2bfibjghyyysm7lisubbvrgjq"}},"DealInfo":null}}}
21.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":4194304,"PieceCID":{"/":"baga6ea4seaqijqccdoqgqwqbx54vui2eazh6ijf5kku5eq3xwokp6tclivuoqei"}},"DealInfo":null}}}
22.     2020-08-29 14:08:37 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":8388608,"PieceCID":{"/":"baga6ea4seaqgl4u6lwmnerwdrm4iz7ag3mpwwaqtapc2fciabpooqmvjypweeha"}},"DealInfo":null}}}
23.     2020-08-29 14:08:38 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":16777216,"PieceCID":{"/":"baga6ea4seaqkejdvbaufquewln7dgszre6ymaqvr2bdnyvcaee3we7gypgoocoq"}},"DealInfo":null}}}
24.     2020-08-29 14:08:38 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":33554432,"PieceCID":{"/":"baga6ea4seaqnv7nlnwutmrctyjwtg4tlt7x6gq56r6awjhwabgvnh6x7kbqxkca"}},"DealInfo":null}}}
25.     2020-08-29 14:08:39 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":67108864,"PieceCID":{"/":"baga6ea4seaqnsqov4dldcsuzlqz77pkpxzurddlt2ts72lgtd4hxzbxl3ukoobq"}},"DealInfo":null}}}
26.     2020-08-29 14:08:41 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":134217728,"PieceCID":{"/":"baga6ea4seaqjamx67n44o2oonsdtqfjqwgtawfzyqubzfvtlk4z65prqreagugq"}},"DealInfo":{"PublishCid":{"/":"bafy2bzaceai6iijep3ay6da2soyulaeadgdfelxelaxidyhp2jeernscueykc"},"DealID":46420,"DealSchedule":{"StartEpoch":16803,"EndEpoch":717164},"KeepUnsealed":true}}}}
27.     2020-08-29 16:25:49 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
28.     2020-08-29 19:41:29 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
29.     2020-08-29 19:52:22 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
30.     2020-08-31 21:12:12 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
31.     2020-08-31 21:24:06 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
32.     2020-08-31 21:24:29 -0400 EDT:  [event;sealing.SectorStartPacking]      {"User":{}}
33.     2020-08-31 21:32:42 -0400 EDT:  [event;sealing.SectorPacked]    {"User":{"FillerPieces":[{"Size":268435456,"PieceCID":{"/":"baga6ea4seaqk2bufhfu5g7ju74eobh2wsmfevum2rhppmdf75z7b2m4byhtryny"}},{"Size":1073741824,"PieceCID":{"/":"baga6ea4seaqmzq6acl23axubdiv37xipnaz3qqtvwr57ekoaauviescphqnfwpi"}},{"Size":2147483648,"PieceCID":{"/":"baga6ea4seaqh34u3nf3tdgpi6k2aw54rtucikcpo25uofrzjpmprinydj7b4mla"}},{"Size":4294967296,"PieceCID":{"/":"baga6ea4seaqgntqfunthkuwpixacxtcoqojjdg66vq254l7vmjyyjdu7pntvcby"}},{"Size":8589934592,"PieceCID":{"/":"baga6ea4seaqnqyicdbbfvnpjlmokmi45fgroiigxa2uw6nz6f6ojveoxlhizwai"}},{"Size":17179869184,"PieceCID":{"/":"baga6ea4seaqg2nsld34emra2ljfgrbrdcswmbjdpaftrpzjuipudt3w7qpbikpa"}}]}}
34.     2020-08-31 21:32:42 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
35.     2020-08-31 21:49:32 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
36.     2020-08-31 21:49:32 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
37.     2020-08-31 22:05:13 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
38.     2020-08-31 22:05:13 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
39.     2020-08-31 22:21:19 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
40.     2020-08-31 22:21:19 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
41.     2020-08-31 22:37:46 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
42.     2020-08-31 22:37:46 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
43.     2020-08-31 22:54:36 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
44.     2020-08-31 22:54:36 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
45.     2020-08-31 23:11:15 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
46.     2020-08-31 23:11:15 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
47.     2020-08-31 23:28:25 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
48.     2020-08-31 23:28:25 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
49.     2020-08-31 23:45:24 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
50.     2020-08-31 23:45:24 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
51.     2020-09-01 00:02:57 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
52.     2020-09-01 00:02:57 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
53.     2020-09-01 00:19:43 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
54.     2020-09-01 00:19:43 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
55.     2020-09-01 00:36:42 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
56.     2020-09-01 00:36:42 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
57.     2020-09-01 00:45:28 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
58.     2020-09-01 01:03:12 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
59.     2020-09-01 01:03:12 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
60.     2020-09-01 01:21:24 -0400 EDT:  [event;sealing.SectorRemove]    {"User":{}}
61.     2020-09-01 01:21:24 -0400 EDT:  [event;sealing.SectorUpdateDealIDs]     {"User":{"Updates":{"0":45203,"1":45354,"11":45713,"12":45816,"13":46117,"14":46203,"25":46420}}}
62.     2020-09-01 01:58:43 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
63.     2020-09-01 01:58:43 -0400 EDT:  [event;sealing.SectorRemoved]   {"User":{}}

Here is the sector log of the other not in unsealedInfoMap sector, which went through Packing and then to Removed:

Status:         Removed
CIDcommD:       <nil>
CIDcommR:       <nil>
Ticket:
TicketH:        0
Seed:
SeedH:          0
Precommit:      <nil>
Commit:         <nil>
Proof:
Deals:          [39410 39775 0 0 0 0 0 0 0 0 0 39846 40150 0 0 0 0 0 0 0 0 0 0 40239 0 0 0 0 0 0]
Retries:        0
--------
Event Log:
0.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorStart]     {"User":{"ID":6,"SectorType":3}}
1.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqdoyrf23xgtapdbms27dgvzfd2wfn22nklpeqd2nhxwokspn7qsmi"}},"DealInfo":{"DealID":39410,"DealSchedule":{"StartEpoch":15001,"EndEpoch":717164},"KeepUnsealed":true}}}}
2.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqk7imla4wu6wuq7q7zyfs62wx3q23nqh7pd3ioodfy56j2yzwj2bi"}},"DealInfo":{"DealID":39775,"DealSchedule":{"StartEpoch":15118,"EndEpoch":717164},"KeepUnsealed":true}}}}
3.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":262144,"PieceCID":{"/":"baga6ea4seaqhklmwsp5bm5jehfkhnyyxvgcyb4aji6x3piyfidlclkjjdtasuby"}},"DealInfo":null}}}
4.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":524288,"PieceCID":{"/":"baga6ea4seaqhaixwb57pnlp2c4ixuutbtyym5kbmnadvvxy4mz3yn3cqn3xs2gi"}},"DealInfo":null}}}
5.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":1048576,"PieceCID":{"/":"baga6ea4seaqntgehxfzvoouw4ejzgzcsg3axwh2moa2noi6hvgpxbg5u3jqrmky"}},"DealInfo":null}}}
6.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":2097152,"PieceCID":{"/":"baga6ea4seaqnbnjq3oylj4s4luxsukg752aiwu2bfibjghyyysm7lisubbvrgjq"}},"DealInfo":null}}}
7.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":4194304,"PieceCID":{"/":"baga6ea4seaqijqccdoqgqwqbx54vui2eazh6ijf5kku5eq3xwokp6tclivuoqei"}},"DealInfo":null}}}
8.      2020-08-29 00:37:57 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":8388608,"PieceCID":{"/":"baga6ea4seaqgl4u6lwmnerwdrm4iz7ag3mpwwaqtapc2fciabpooqmvjypweeha"}},"DealInfo":null}}}
9.      2020-08-29 00:37:58 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":16777216,"PieceCID":{"/":"baga6ea4seaqkejdvbaufquewln7dgszre6ymaqvr2bdnyvcaee3we7gypgoocoq"}},"DealInfo":null}}}
10.     2020-08-29 00:37:58 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":33554432,"PieceCID":{"/":"baga6ea4seaqnv7nlnwutmrctyjwtg4tlt7x6gq56r6awjhwabgvnh6x7kbqxkca"}},"DealInfo":null}}}
11.     2020-08-29 00:37:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":67108864,"PieceCID":{"/":"baga6ea4seaqnsqov4dldcsuzlqz77pkpxzurddlt2ts72lgtd4hxzbxl3ukoobq"}},"DealInfo":null}}}
12.     2020-08-29 00:38:03 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":134217728,"PieceCID":{"/":"baga6ea4seaqk5mfjwhwbpcuoom27xexlwee4d77egnixkrv3jd5g5pyhji2a2gy"}},"DealInfo":{"DealID":39846,"DealSchedule":{"StartEpoch":15125,"EndEpoch":717164},"KeepUnsealed":true}}}}
13.     2020-08-29 01:03:30 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqff7ueoq7jvhjpun642eyuglaaviqnmtdkbrkaote3teaomkerqby"}},"DealInfo":{"DealID":40150,"DealSchedule":{"StartEpoch":15240,"EndEpoch":717164},"KeepUnsealed":true}}}}
14.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":131072,"PieceCID":{"/":"baga6ea4seaqi5hreap5iqthwen7wbxzf7a7oidokt3mht23pmnjncuee6wwq2py"}},"DealInfo":null}}}
15.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":262144,"PieceCID":{"/":"baga6ea4seaqhklmwsp5bm5jehfkhnyyxvgcyb4aji6x3piyfidlclkjjdtasuby"}},"DealInfo":null}}}
16.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":524288,"PieceCID":{"/":"baga6ea4seaqhaixwb57pnlp2c4ixuutbtyym5kbmnadvvxy4mz3yn3cqn3xs2gi"}},"DealInfo":null}}}
17.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":1048576,"PieceCID":{"/":"baga6ea4seaqntgehxfzvoouw4ejzgzcsg3axwh2moa2noi6hvgpxbg5u3jqrmky"}},"DealInfo":null}}}
18.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":2097152,"PieceCID":{"/":"baga6ea4seaqnbnjq3oylj4s4luxsukg752aiwu2bfibjghyyysm7lisubbvrgjq"}},"DealInfo":null}}}
19.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":4194304,"PieceCID":{"/":"baga6ea4seaqijqccdoqgqwqbx54vui2eazh6ijf5kku5eq3xwokp6tclivuoqei"}},"DealInfo":null}}}
20.     2020-08-29 01:14:59 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":8388608,"PieceCID":{"/":"baga6ea4seaqgl4u6lwmnerwdrm4iz7ag3mpwwaqtapc2fciabpooqmvjypweeha"}},"DealInfo":null}}}
21.     2020-08-29 01:15:00 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":16777216,"PieceCID":{"/":"baga6ea4seaqkejdvbaufquewln7dgszre6ymaqvr2bdnyvcaee3we7gypgoocoq"}},"DealInfo":null}}}
22.     2020-08-29 01:15:00 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":33554432,"PieceCID":{"/":"baga6ea4seaqnv7nlnwutmrctyjwtg4tlt7x6gq56r6awjhwabgvnh6x7kbqxkca"}},"DealInfo":null}}}
23.     2020-08-29 01:15:02 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":67108864,"PieceCID":{"/":"baga6ea4seaqnsqov4dldcsuzlqz77pkpxzurddlt2ts72lgtd4hxzbxl3ukoobq"}},"DealInfo":null}}}
24.     2020-08-29 01:15:04 -0400 EDT:  [event;sealing.SectorAddPiece]  {"User":{"NewPiece":{"Piece":{"Size":134217728,"PieceCID":{"/":"baga6ea4seaqh3n2tu2oamb6exrxov3wso5mgbuylqi4jmnsrya6cdd2zftf22mi"}},"DealInfo":{"DealID":40239,"DealSchedule":{"StartEpoch":15253,"EndEpoch":717164},"KeepUnsealed":true}}}}
25.     2020-08-29 01:54:55 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
26.     2020-08-29 16:25:49 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
27.     2020-08-29 19:41:29 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
28.     2020-08-29 19:52:22 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
29.     2020-08-31 21:12:12 -0400 EDT:  [event;sealing.SectorRestart]   {"User":{}}
30.     2020-08-31 21:13:03 -0400 EDT:  [event;sealing.SectorStartPacking]      {"User":{}}
31.     2020-08-31 21:21:19 -0400 EDT:  [event;sealing.SectorPacked]    {"User":{"FillerPieces":[{"Size":536870912,"PieceCID":{"/":"baga6ea4seaqdsvqopmj2soyhujb72jza76t4wpq5fzifvm3ctz47iyytkewnubq"}},{"Size":1073741824,"PieceCID":{"/":"baga6ea4seaqmzq6acl23axubdiv37xipnaz3qqtvwr57ekoaauviescphqnfwpi"}},{"Size":2147483648,"PieceCID":{"/":"baga6ea4seaqh34u3nf3tdgpi6k2aw54rtucikcpo25uofrzjpmprinydj7b4mla"}},{"Size":4294967296,"PieceCID":{"/":"baga6ea4seaqgntqfunthkuwpixacxtcoqojjdg66vq254l7vmjyyjdu7pntvcby"}},{"Size":8589934592,"PieceCID":{"/":"baga6ea4seaqnqyicdbbfvnpjlmokmi45fgroiigxa2uw6nz6f6ojveoxlhizwai"}},{"Size":17179869184,"PieceCID":{"/":"baga6ea4seaqg2nsld34emra2ljfgrbrdcswmbjdpaftrpzjuipudt3w7qpbikpa"}}]}}
32.     2020-08-31 21:21:19 -0400 EDT:  [event;sealing.SectorInvalidDealIDs]    {"User":{"Return":"PreCommit1"}}
33.     2020-08-31 21:21:19 -0400 EDT:  [event;sealing.SectorRemove]    {"User":{}}
34.     2020-08-31 21:21:19 -0400 EDT:  [event;sealing.SectorRemoved]   {"User":{}}

Lotus miner logs

I lost some logs due to a typo, but here is what I have:

2020-08-31T21:24:09.660-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false
 cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:09.660-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:24:29.424-0400    INFO    sectors storage-sealing/sealing.go:236  Starting packing sector 8
2020-08-31T21:24:29.424-0400    INFO    sectors storage-sealing/sealing.go:241  send Starting packing event success sector 8
2020-08-31T21:24:29.425-0400    INFO    sectors storage-sealing/states_sealing.go:22    performing filling up rest of the sector...     {"sector":
"8"}
2020-08-31T21:24:29.425-0400    WARN    sectors storage-sealing/states_sealing.go:41    Creating 6 filler pieces for sector 8
2020-08-31T21:24:29.425-0400    INFO    sectors storage-sealing/garbage.go:16   Pledge {9737 8}, contains [130048 130048 260096 520192 1040384 2080768 4161536 8323072 16646144 33292288 66584576 133169152 133169152 133169152 130048 130048 260096 520192 1040384 2080768 4161536 8323072 16646144 33292288 66584576 133169152]
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 1 queued; 2 open windows
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: [[0 1]]
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:430     SCHED try assign sqi:0 sector 8 to window 0
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:437     SCHED ASSIGNED sqi:0 sector 8 task seal/v0/addpiece to window 0
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:598     assign worker sector 8
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 0 queued; 2 open windows
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/localworker.go:75        acquired sector {9737 8} (e:1; a:0): {{0 0} /scratch/unsealed/s-t09737-8  }
2020-08-31T21:24:29.425-0400    DEBUG   advmgr  sector-storage/sched.go:557     task done       {"workerid": 0}
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 1 queued; 2 open windows
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: [[0 1]]
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:430     SCHED try assign sqi:0 sector 8 to window 0
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:437     SCHED ASSIGNED sqi:0 sector 8 task seal/v0/addpiece to window 0
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:598     assign worker sector 8
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 0 queued; 2 open windows
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/localworker.go:75        acquired sector {9737 8} (e:1; a:0): {{0 0} /scratch/unsealed/s-t09737-8  }
2020-08-31T21:24:33.612-0400    DEBUG   advmgr  sector-storage/sched.go:557     task done       {"workerid": 0}
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 1 queued; 2 open windows
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: [[0 1]]
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:430     SCHED try assign sqi:0 sector 8 to window 0
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:437     SCHED ASSIGNED sqi:0 sector 8 task seal/v0/addpiece to window 0
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:598     assign worker sector 8
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 0 queued; 2 open windows
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/localworker.go:75        acquired sector {9737 8} (e:1; a:0): {{0 0} /scratch/unsealed/s-t09737-8  }
2020-08-31T21:24:48.629-0400    DEBUG   advmgr  sector-storage/sched.go:557     task done       {"workerid": 0}
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 1 queued; 2 open windows
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: [[0 1]]
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:430     SCHED try assign sqi:0 sector 8 to window 0
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:437     SCHED ASSIGNED sqi:0 sector 8 task seal/v0/addpiece to window 0
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:598     assign worker sector 8
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 0 queued; 2 open windows
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/localworker.go:75        acquired sector {9737 8} (e:1; a:0): {{0 0} /scratch/unsealed/s-t09737-8  }
2020-08-31T21:25:19.664-0400    DEBUG   advmgr  sector-storage/sched.go:557     task done       {"workerid": 0}
2020-08-31T21:26:00.601-0400    INFO    storageminer    storage/wdpost_sched.go:208     at 20572, doPost for P 20205, dd 6
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 1 queued; 2 open windows
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: [[0 1]]
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:430     SCHED try assign sqi:0 sector 8 to window 0
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:437     SCHED ASSIGNED sqi:0 sector 8 task seal/v0/addpiece to window 0
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:598     assign worker sector 8
2020-08-31T21:26:20.340-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 0 queued; 2 open windows
2020-08-31T21:26:20.341-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:26:20.341-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:26:20.341-0400    DEBUG   advmgr  sector-storage/sched.go:557     task done       {"workerid": 0}
2020-08-31T21:26:20.341-0400    DEBUG   advmgr  sector-storage/localworker.go:75        acquired sector {9737 8} (e:1; a:0): {{0 0} /scratch/unsealed/s-t09737-8  }
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 1 queued; 2 open windows
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: [[0 1]]
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:430     SCHED try assign sqi:0 sector 8 to window 0
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:437     SCHED ASSIGNED sqi:0 sector 8 task seal/v0/addpiece to window 0
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:598     assign worker sector 8
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:324     SCHED 0 queued; 2 open windows
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:415     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:<nil>} todo:[]}]
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:416     SCHED Acceptable win: []
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/localworker.go:75        acquired sector {9737 8} (e:1; a:0): {{0 0} /scratch/unsealed/s-t09737-8  }
2020-08-31T21:28:29.365-0400    DEBUG   advmgr  sector-storage/sched.go:557     task done       {"workerid": 0}
2020-08-31T21:32:42.287-0400    WARN    sectors storage-sealing/states_sealing.go:89    invalid deals in sector 8: getting deal 45203 for piece 0: deal 45203 not found
2020-08-31T21:32:42.289-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45203 for piece 0: deal 45203 not found
2020-08-31T21:32:42.289-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45354 for piece 1: deal 45354 not found
2020-08-31T21:32:42.290-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45713 for piece 11: deal 45713 not found
2020-08-31T21:32:42.290-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45816 for piece 12: deal 45816 not found
2020-08-31T21:32:42.290-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46117 for piece 13: deal 46117 not found
2020-08-31T21:32:42.291-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46203 for piece 14: deal 46203 not found
2020-08-31T21:32:42.291-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46420 for piece 25: deal 46420 not found
2020-08-31T21:49:32.652-0400    WARN    sectors storage-sealing/states_sealing.go:89    invalid deals in sector 8: getting deal 45203 for piece 0: deal 45203 not found
2020-08-31T21:49:32.654-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45203 for piece 0: deal 45203 not found
2020-08-31T21:49:32.654-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45354 for piece 1: deal 45354 not found
2020-08-31T21:49:32.655-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45713 for piece 11: deal 45713 not found
2020-08-31T21:49:32.655-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45816 for piece 12: deal 45816 not found
2020-08-31T21:49:32.656-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46117 for piece 13: deal 46117 not found
2020-08-31T21:49:32.656-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46203 for piece 14: deal 46203 not found
2020-08-31T21:49:32.657-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46420 for piece 25: deal 46420 not found
2020-08-31T22:05:13.927-0400    WARN    sectors storage-sealing/states_sealing.go:89    invalid deals in sector 8: getting deal 45203 for piece 0: deal 45203 not found
2020-08-31T22:05:13.929-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45203 for piece 0: deal 45203 not found
2020-08-31T22:05:13.929-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45354 for piece 1: deal 45354 not found
2020-08-31T22:05:13.930-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45713 for piece 11: deal 45713 not found
2020-08-31T22:05:13.930-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 45816 for piece 12: deal 45816 not found
2020-08-31T22:05:13.931-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46117 for piece 13: deal 46117 not found
2020-08-31T22:05:13.932-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46203 for piece 14: deal 46203 not found
2020-08-31T22:05:13.932-0400    WARN    sectors storage-sealing/states_failed.go:297    getting deal 46420 for piece 25: deal 46420 not found
   <repeats forever>

Lotus miner diagnostic info

allinfo.txt

Version

lotus version 0.5.7+git.49d64f7f
@jennijuju
Copy link
Member

fixed by #3401

@damonhung
Copy link
Author

@jennijuju Thanks for taking a look! But this issue wasn't fixed by #3401. This is the state I got into after #3401. Specifically:

  1. I got the unsealedMapInfo problem, which was unrecoverable before Rebuild unsealed infos on miner restart #3401
  2. I applied Rebuild unsealed infos on miner restart #3401 (in the form of v0.5.7), which allowed those sectors to get unstuck.
  3. As you can see in the original report, the sectors made progress and then got stuck again. This was recoverable by using the dangerous update-state command plus a miner reboot. I reported this in the Slack and was asked to file this ticket.

@jennijuju
Copy link
Member

@jennijuju Thanks for taking a look! But this issue wasn't fixed by #3401. This is the state I got into after #3401. Specifically:

  1. I got the unsealedMapInfo problem, which was unrecoverable before Rebuild unsealed infos on miner restart #3401
  2. I applied Rebuild unsealed infos on miner restart #3401 (in the form of v0.5.7), which allowed those sectors to get unstuck.
  3. As you can see in the original report, the sectors made progress and then got stuck again. This was recoverable by using the dangerous update-state command plus a miner reboot. I reported this in the Slack and was asked to file this ticket.

Ah sorry - will reopen the issue.

@jennijuju jennijuju reopened this Oct 2, 2020
@magik6k magik6k mentioned this issue Feb 22, 2021
Merged
@Reiers
Copy link

Reiers commented Apr 18, 2022 •
edited
Loading

Hi @damonhung

Thanks for the report. The fix seems to be merged, and I have not seen any reports of it recently.

Closing ticket.

If you are still experiencing this issue or have this problem in the current release - please open a new ticket, and reference this issue.
Thank you !

@Reiers Reiers closed this as completed Apr 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove sectors with all deals expired in RecoverDealIDs filecoin-project/lotus
3 participants
@damonhung @jennijuju @Reiers