Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix data race in periodic reader tests #4731

Merged
merged 2 commits into from
Nov 28, 2023

Conversation

dashpole
Copy link
Contributor

@dashpole dashpole commented Nov 22, 2023

Fixes #4543

~/go/src/go.opentelemetry.io/opentelemetry-go/sdk/metric$ go test -race -count=1000 -timeout=2m -run "^TestPeriodicReaderRun$|^TestPeriodicReader$"
PASS
ok  	go.opentelemetry.io/otel/sdk/metric	46.140s

The Reader is already being initialized in the individual tests, e.g.

func (ts *readerTestSuite) TestSDKProducer() {
ts.Reader = ts.Factory()

The creation of an additional reader during registration causes problems because the extra reader was being overwritten in the test, and thus never properly Shutdown(). The run() goroutine from the reader was still active during other tests, and could race with them.

Copy link

codecov bot commented Nov 22, 2023

Codecov Report

Merging #4731 (7c38a0e) into main (47ba653) will decrease coverage by 0.1%.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##            main   #4731     +/-   ##
=======================================
- Coverage   81.8%   81.7%   -0.1%     
=======================================
  Files        224     224             
  Lines      18113   18113             
=======================================
- Hits       14817   14815      -2     
- Misses      3000    3002      +2     
  Partials     296     296             

see 1 file with indirect coverage changes

@dashpole dashpole added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Nov 22, 2023
@dashpole dashpole marked this pull request as ready for review November 22, 2023 18:02
@pellared
Copy link
Member

I think that some tests also miss shutdown of the reader which also can bring problems.
Can you also please double-check these tests?

  • TestPeriodicReaderMultipleForceFlush
  • BenchmarkPeriodicReader (it is a benchmark so it is usually not run, but still it can interfere if one runs the benchmarks)

@dashpole
Copy link
Contributor Author

Added shutdowns to the two you mention. Since tests are run serially, it doesn't seem to matter, but good to do anyways.

before:

BenchmarkPeriodicReader/Collect-24 465 2940 ns/op 192 B/op 6 allocs/op

after:

BenchmarkPeriodicReader/Collect-24 546 2229 ns/op 192 B/op 6 allocs/op

@pellared pellared merged commit 204be61 into open-telemetry:main Nov 28, 2023
24 checks passed
@MrAlias MrAlias added this to the v1.22.0 milestone Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Skip Changelog PRs that do not require a CHANGELOG.md entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sdk/metric: Flaky test TestPeriodicReaderRun (data race)
5 participants