Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bor finality whitelist race at startup #10479

Closed
AskAlexSharov opened this issue May 26, 2024 · 0 comments · Fixed by #10493
Closed

Bor finality whitelist race at startup #10479

AskAlexSharov opened this issue May 26, 2024 · 0 comments · Fixed by #10493
Labels
imp1 High importance polygon
Milestone
2.60.1-fixes

Comments

@AskAlexSharov
Copy link
Collaborator 

==================
WARNING: DATA RACE
Read at 0x00c3cd8fae10 by goroutine 25015:
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).RemoveMilestoneID()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:171 +0xd9
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleNoAckMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:221 +0x12c
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:143 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:101 +0x57
  github.com/ledgerwatch/erigon/polygon/bor/finality.startNoAckMilestoneService()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:86 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap3()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:49 +0x17

Previous write at 0x00c3cd8fae10 by goroutine 25014:
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).purgeMilestoneIDsList()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:216 +0x13d
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).ProcessFutureMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:253 +0x123
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:192 +0x1e5
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:143 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:101 +0x5c
  github.com/ledgerwatch/erigon/polygon/bor/finality.startMilestoneWhitelistService()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:77 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap2()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:48 +0x17

Goroutine 25015 (running) created at:
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:49 +0x3b0
  github.com/ledgerwatch/erigon/polygon/bor.(*Bor).Start()
      github.com/ledgerwatch/erigon/polygon/bor/bor.go:1283 +0x30c
  github.com/ledgerwatch/erigon/eth.(*Ethereum).Start()
      github.com/ledgerwatch/erigon/eth/backend.go:1545 +0xd64
  github.com/ledgerwatch/erigon/node.(*Node).Start()
      github.com/ledgerwatch/erigon/node/node.go:125 +0x3b3
  github.com/ledgerwatch/erigon/node.StartNode()
      github.com/ledgerwatch/erigon/node/node.go:412 +0x2e
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).run()
      github.com/ledgerwatch/erigon/turbo/node/node.go:52 +0x9b
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve()
      github.com/ledgerwatch/erigon/turbo/node/node.go:32 +0x86
  main.runErigon()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:74 +0x645
  github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1()
      github.com/ledgerwatch/erigon/turbo/app/make_app.go:54 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      github.com/urfave/cli/[email protected]/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      github.com/urfave/cli/[email protected]/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      github.com/urfave/cli/[email protected]/app.go:307 +0xc8
  main.main()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x8d

Goroutine 25014 (running) created at:
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:48 +0x345
  github.com/ledgerwatch/erigon/polygon/bor.(*Bor).Start()
      github.com/ledgerwatch/erigon/polygon/bor/bor.go:1283 +0x30c
  github.com/ledgerwatch/erigon/eth.(*Ethereum).Start()
      github.com/ledgerwatch/erigon/eth/backend.go:1545 +0xd64
  github.com/ledgerwatch/erigon/node.(*Node).Start()
      github.com/ledgerwatch/erigon/node/node.go:125 +0x3b3
  github.com/ledgerwatch/erigon/node.StartNode()
      github.com/ledgerwatch/erigon/node/node.go:412 +0x2e
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).run()
      github.com/ledgerwatch/erigon/turbo/node/node.go:52 +0x9b
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve()
      github.com/ledgerwatch/erigon/turbo/node/node.go:32 +0x86
  main.runErigon()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:74 +0x645
  github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1()
      github.com/ledgerwatch/erigon/turbo/app/make_app.go:54 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      github.com/urfave/cli/[email protected]/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      github.com/urfave/cli/[email protected]/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      github.com/urfave/cli/[email protected]/app.go:307 +0xc8
  main.main()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x8d
==================
==================
WARNING: DATA RACE
Write at 0x00c276bf0d80 by goroutine 25015:
  runtime.mapdelete_faststr()
      runtime/map_faststr.go:301 +0x0
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).RemoveMilestoneID()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:171 +0x104
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleNoAckMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:221 +0x12c
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:143 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:101 +0x57
  github.com/ledgerwatch/erigon/polygon/bor/finality.startNoAckMilestoneService()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:86 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap3()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:49 +0x17

Previous read at 0x00c276bf0d80 by goroutine 25014:
  reflect.maplen()
      runtime/map.go:1406 +0x0
  reflect.Value.lenNonSlice()
      reflect/value.go:1785 +0x1e9
  reflect.Value.Len()
      reflect/value.go:1774 +0x426
  encoding/json.mapEncoder.encode()
      encoding/json/encode.go:741 +0x408
  encoding/json.mapEncoder.encode-fm()
      <autogenerated>:1 +0x84
  encoding/json.structEncoder.encode()
      encoding/json/encode.go:704 +0x2bd
  encoding/json.structEncoder.encode-fm()
      <autogenerated>:1 +0xe4
  encoding/json.(*encodeState).reflectValue()
      encoding/json/encode.go:321 +0x83
  encoding/json.(*encodeState).marshal()
      encoding/json/encode.go:297 +0xea
  encoding/json.Marshal()
      encoding/json/encode.go:163 +0x12b
  github.com/ledgerwatch/erigon/polygon/bor/finality/rawdb.WriteLockField()
      github.com/ledgerwatch/erigon/polygon/bor/finality/rawdb/milestone.go:142 +0x10b
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).ProcessFutureMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:255 +0x265
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:192 +0x1e5
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:143 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:101 +0x5c
  github.com/ledgerwatch/erigon/polygon/bor/finality.startMilestoneWhitelistService()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:77 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap2()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:48 +0x17

Goroutine 25015 (running) created at:
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:49 +0x3b0
  github.com/ledgerwatch/erigon/polygon/bor.(*Bor).Start()
      github.com/ledgerwatch/erigon/polygon/bor/bor.go:1283 +0x30c
  github.com/ledgerwatch/erigon/eth.(*Ethereum).Start()
      github.com/ledgerwatch/erigon/eth/backend.go:1545 +0xd64
  github.com/ledgerwatch/erigon/node.(*Node).Start()
      github.com/ledgerwatch/erigon/node/node.go:125 +0x3b3
  github.com/ledgerwatch/erigon/node.StartNode()
      github.com/ledgerwatch/erigon/node/node.go:412 +0x2e
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).run()
      github.com/ledgerwatch/erigon/turbo/node/node.go:52 +0x9b
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve()
      github.com/ledgerwatch/erigon/turbo/node/node.go:32 +0x86
  main.runErigon()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:74 +0x645
  github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1()
      github.com/ledgerwatch/erigon/turbo/app/make_app.go:54 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      github.com/urfave/cli/[email protected]/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      github.com/urfave/cli/[email protected]/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      github.com/urfave/cli/[email protected]/app.go:307 +0xc8
  main.main()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x8d

Goroutine 25014 (running) created at:
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:48 +0x345
  github.com/ledgerwatch/erigon/polygon/bor.(*Bor).Start()
      github.com/ledgerwatch/erigon/polygon/bor/bor.go:1283 +0x30c
  github.com/ledgerwatch/erigon/eth.(*Ethereum).Start()
      github.com/ledgerwatch/erigon/eth/backend.go:1545 +0xd64
  github.com/ledgerwatch/erigon/node.(*Node).Start()
      github.com/ledgerwatch/erigon/node/node.go:125 +0x3b3
  github.com/ledgerwatch/erigon/node.StartNode()
      github.com/ledgerwatch/erigon/node/node.go:412 +0x2e
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).run()
      github.com/ledgerwatch/erigon/turbo/node/node.go:52 +0x9b
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve()
      github.com/ledgerwatch/erigon/turbo/node/node.go:32 +0x86
  main.runErigon()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:74 +0x645
  github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1()
      github.com/ledgerwatch/erigon/turbo/app/make_app.go:54 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      github.com/urfave/cli/[email protected]/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      github.com/urfave/cli/[email protected]/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      github.com/urfave/cli/[email protected]/app.go:307 +0xc8
  main.main()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x8d
==================
==================
WARNING: DATA RACE
Write at 0x00c3cd8fae08 by goroutine 25015:
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).RemoveMilestoneID()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:174 +0x164
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleNoAckMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:221 +0x12c
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:143 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:101 +0x57
  github.com/ledgerwatch/erigon/polygon/bor/finality.startNoAckMilestoneService()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:86 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap3()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:49 +0x17

Previous write at 0x00c3cd8fae08 by goroutine 25014:
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).ProcessFutureMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist/milestone.go:252 +0x104
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleMilestone()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:192 +0x1e5
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:143 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:101 +0x5c
  github.com/ledgerwatch/erigon/polygon/bor/finality.startMilestoneWhitelistService()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:77 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap2()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:48 +0x17

Goroutine 25015 (running) created at:
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:49 +0x3b0
  github.com/ledgerwatch/erigon/polygon/bor.(*Bor).Start()
      github.com/ledgerwatch/erigon/polygon/bor/bor.go:1283 +0x30c
  github.com/ledgerwatch/erigon/eth.(*Ethereum).Start()
      github.com/ledgerwatch/erigon/eth/backend.go:1545 +0xd64
  github.com/ledgerwatch/erigon/node.(*Node).Start()
      github.com/ledgerwatch/erigon/node/node.go:125 +0x3b3
  github.com/ledgerwatch/erigon/node.StartNode()
      github.com/ledgerwatch/erigon/node/node.go:412 +0x2e
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).run()
      github.com/ledgerwatch/erigon/turbo/node/node.go:52 +0x9b
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve()
      github.com/ledgerwatch/erigon/turbo/node/node.go:32 +0x86
  main.runErigon()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:74 +0x645
  github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1()
      github.com/ledgerwatch/erigon/turbo/app/make_app.go:54 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      github.com/urfave/cli/[email protected]/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      github.com/urfave/cli/[email protected]/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      github.com/urfave/cli/[email protected]/app.go:307 +0xc8
  main.main()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x8d

Goroutine 25014 (running) created at:
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist()
      github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.go:48 +0x345
  github.com/ledgerwatch/erigon/polygon/bor.(*Bor).Start()
      github.com/ledgerwatch/erigon/polygon/bor/bor.go:1283 +0x30c
  github.com/ledgerwatch/erigon/eth.(*Ethereum).Start()
      github.com/ledgerwatch/erigon/eth/backend.go:1545 +0xd64
  github.com/ledgerwatch/erigon/node.(*Node).Start()
      github.com/ledgerwatch/erigon/node/node.go:125 +0x3b3
  github.com/ledgerwatch/erigon/node.StartNode()
      github.com/ledgerwatch/erigon/node/node.go:412 +0x2e
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).run()
      github.com/ledgerwatch/erigon/turbo/node/node.go:52 +0x9b
  github.com/ledgerwatch/erigon/turbo/node.(*ErigonNode).Serve()
      github.com/ledgerwatch/erigon/turbo/node/node.go:32 +0x86
  main.runErigon()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:74 +0x645
  github.com/ledgerwatch/erigon/turbo/app.MakeApp.func1()
      github.com/ledgerwatch/erigon/turbo/app/make_app.go:54 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      github.com/urfave/cli/[email protected]/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      github.com/urfave/cli/[email protected]/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      github.com/urfave/cli/[email protected]/app.go:307 +0xc8
  main.main()
      github.com/ledgerwatch/erigon/cmd/erigon/main.go:34 +0x8d
==================
@AskAlexSharov AskAlexSharov added this to the 2.60.1-fixes milestone May 26, 2024
@yperbasis yperbasis added the imp1 High importance label May 27, 2024
@mh0lt mh0lt mentioned this issue May 27, 2024
Merged
yperbasis pushed a commit that referenced this issue May 28, 2024
@mh0lt
add lock to purgeMilestoneIDsList (#10493)
53b3d23 
Fixes: #10479
yperbasis pushed a commit that referenced this issue May 28, 2024
@mh0lt @yperbasis
add lock to purgeMilestoneIDsList (#10493)
23fb7e4 
Fixes: #10479
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
imp1 High importance polygon
Projects
None yet
Milestone
2.60.1-fixes
Development

Successfully merging a pull request may close this issue.

add lock to purgeMilestoneIDsList erigontech/erigon
2 participants
@yperbasis @AskAlexSharov