Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exp/lighthorizon: mapreduce data race condition #4462

Closed
2opremio opened this issue Jul 18, 2022 · 3 comments
Closed

exp/lighthorizon: mapreduce data race condition #4462

2opremio opened this issue Jul 18, 2022 · 3 comments
Assignees
@Shaptic
Labels
bug horizon

Comments

@2opremio
Copy link
Contributor

The tests at https://github.com/stellar/go/runs/7388078175?check_suite_focus=true detected a race condition

     mapreduce_test.go:45: Building baseline for ledger range [1410048, 1410303]
    mapreduce_test.go:202: 
        	Error Trace:	mapreduce_test.go:202
        	            				mapreduce_test.go:57
        	Error:      	map[uint32]struct {}{} does not contain 0x5614
        	Test:       	TestReduce
        	Messages:   	failed to find 22036 for GBM3LXCLBTCWHWHUZK3UCAVLE5SK3RVPIUIZIWW5V62FLTV4D6NHINCS (found map[])
==================
WARNING: DATA RACE
Write at 0x00c0006ce150 by goroutine 24:
  runtime.mapassign_faststr()
      /opt/hostedtoolcache/go/1.17.11/x64/src/runtime/map_faststr.go:202 +0x0
  github.com/stellar/go/exp/lighthorizon/index.(*store).getCreateIndex()
      /home/runner/work/go/go/exp/lighthorizon/index/store.go:313 +0x5a4
  github.com/stellar/go/exp/lighthorizon/index.(*store).AddParticipantsToIndexes()
      /home/runner/work/go/go/exp/lighthorizon/index/store.go:260 +0x184
  github.com/stellar/go/exp/lighthorizon/index.ProcessAccounts()
      /home/runner/work/go/go/exp/lighthorizon/index/modules.go:33 +0x202
  github.com/stellar/go/exp/lighthorizon/index.(*IndexBuilder).RunModules()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:206 +0x152
  github.com/stellar/go/exp/lighthorizon/index.(*IndexBuilder).Build()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:244 +0x62e
  github.com/stellar/go/exp/lighthorizon/index.BuildIndices.func2()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:123 +0x1e4
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x96
Previous read at 0x00c0006ce150 by goroutine 32:
  runtime.mapiternext()
      /opt/hostedtoolcache/go/1.17.11/x64/src/runtime/map.go:851 +0x0
  github.com/stellar/go/exp/lighthorizon/index/backend.parallelFlush.func1()
      /home/runner/work/go/go/exp/lighthorizon/index/backend/parallel_flush.go:25 +0xa8
Goroutine 24 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0xf0
  github.com/stellar/go/exp/lighthorizon/index.BuildIndices()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:117 +0xef7
  github.com/stellar/go/exp/lighthorizon/index/cmd_test.testSingleProcess()
      /home/runner/work/go/go/exp/lighthorizon/index/cmd/single_test.go:84 +0x274
  github.com/stellar/go/exp/lighthorizon/index/cmd_test.TestSingleProcess.func1()
      /home/runner/work/go/go/exp/lighthorizon/index/cmd/single_test.go:55 +0x46
  testing.tRunner()
      /opt/hostedtoolcache/go/1.17.11/x64/src/testing/testing.go:1259 +0x22f
  testing.(*T).Run·dwrap·21()
      /opt/hostedtoolcache/go/1.17.11/x64/src/testing/testing.go:1306 +0x47
Goroutine 32 (finished) created at:
  github.com/stellar/go/exp/lighthorizon/index/backend.parallelFlush()
      /home/runner/work/go/go/exp/lighthorizon/index/backend/parallel_flush.go:24 +0x13c
  github.com/stellar/go/exp/lighthorizon/index/backend.(*FileBackend).Flush()
      /home/runner/work/go/go/exp/lighthorizon/index/backend/file.go:30 +0xae
  github.com/stellar/go/exp/lighthorizon/index.(*store).Flush()
      /home/runner/work/go/go/exp/lighthorizon/index/store.go:165 +0x158
  github.com/stellar/go/exp/lighthorizon/index.BuildIndices.func2()
      /home/runner/work/go/go/exp/lighthorizon/index/builder.go:135 +0x259
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x96
==================
--- FAIL: TestSingleProcess (73.66s)
    single_test.go:76: Validating single-process builder on ledger range [1410048, 1410367] (320 ledgers)
    single_test.go:81: Storing indices in file:/tmp/TestSingleProcess1447992593/001
    --- FAIL: TestSingleProcess/start-at-checkpoint (37.55s)
        testing.go:1152: race detected during execution of test
    single_test.go:76: Validating single-process builder on ledger range [1410053, 1410367] (315 ledgers)
    single_test.go:81: Storing indices in file:/tmp/TestSingleProcess1447992593/002
    testing.go:1152: race detected during execution of test
FAIL
@2opremio 2opremio mentioned this issue Jul 18, 2022
Merged
@Shaptic
Copy link
Contributor

Shaptic commented Jul 18, 2022

Yep, and it's not even a race condition 😕

This got introduced by #4442 somehow... I'm investigating.

@Shaptic
Copy link
Contributor

Shaptic commented Jul 19, 2022 •
edited
Loading

This should be closed by #4465.

Nope, still present, though that did fix a real test failure. Still investigating...

@Shaptic Shaptic self-assigned this Jul 20, 2022
@Shaptic
Copy link
Contributor

Shaptic commented Jul 21, 2022

Closed by #4470.

@Shaptic Shaptic closed this as completed Jul 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Shaptic Shaptic

Labels
bug horizon
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Shaptic @2opremio