Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data races in ingester #2265

Closed
cyriltovena opened this issue Jun 26, 2020 · 1 comment · Fixed by #2327
Closed

Data races in ingester #2265

cyriltovena opened this issue Jun 26, 2020 · 1 comment · Fixed by #2327
Labels
component/loki help wanted We would love help on these issues. Please come help us!

Comments

@cyriltovena
Copy link
Contributor

I've run the following and found some races:

❯ go test -mod=vendor -race  ./pkg/ingester
==================
WARNING: DATA RACE
Read at 0x00c00020c7f0 by goroutine 103:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bounds()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:458 +0x5e
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepStream()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:174 +0x26e
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepInstance()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:157 +0x16b
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepUsers()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:148 +0x90
  github.com/grafana/loki/pkg/ingester.(*Ingester).loop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:238 +0x112

Previous write at 0x00c00020c7f0 by goroutine 101:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:437 +0x378
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a2
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:330 +0x6e9
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Goroutine 103 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:187 +0x366
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 101 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8
==================
==================
WARNING: DATA RACE
Read at 0x00c0000c8020 by goroutine 103:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bounds()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:459 +0x3ba
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepStream()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:174 +0x26e
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepInstance()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:157 +0x16b
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepUsers()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:148 +0x90
  github.com/grafana/loki/pkg/ingester.(*Ingester).loop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:238 +0x112

Previous write at 0x00c0000c8020 by goroutine 101:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:437 +0x337
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a2
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:330 +0x6e9
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Goroutine 103 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:187 +0x366
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 101 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8
==================
==================
WARNING: DATA RACE
Read at 0x00c0000c8028 by goroutine 103:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bounds()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:460 +0x43c
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepStream()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:174 +0x26e
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepInstance()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:157 +0x16b
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepUsers()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:148 +0x90
  github.com/grafana/loki/pkg/ingester.(*Ingester).loop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:238 +0x112

Previous write at 0x00c0000c8028 by goroutine 101:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:437 +0x337
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a2
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:330 +0x6e9
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Goroutine 103 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:187 +0x366
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 101 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8
==================
==================
WARNING: DATA RACE
Read at 0x00c0007e71a0 by goroutine 103:
  github.com/grafana/loki/pkg/chunkenc.(*headBlock).isEmpty()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:98 +0xad
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bounds()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:463 +0x80
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepStream()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:174 +0x26e
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepInstance()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:157 +0x16b
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepUsers()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:148 +0x90
  github.com/grafana/loki/pkg/ingester.(*Ingester).loop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:238 +0x112

Previous write at 0x00c0007e71a0 by goroutine 101:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:448 +0x456
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a2
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:330 +0x6e9
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Goroutine 103 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:187 +0x366
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 101 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8
==================
==================
WARNING: DATA RACE
Write at 0x00c00041c9b0 by goroutine 101:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:449 +0x49a
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a2
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:330 +0x6e9
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c00041c9b0 by goroutine 103:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bounds()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:465 +0x32c
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepStream()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:174 +0x26e
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepInstance()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:157 +0x16b
  github.com/grafana/loki/pkg/ingester.(*Ingester).sweepUsers()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:148 +0x90
  github.com/grafana/loki/pkg/ingester.(*Ingester).loop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:238 +0x112

Goroutine 101 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 103 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:187 +0x366
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8
==================
--- FAIL: TestChunkFlushingIdle (20.61s)
    testing.go:906: race detected during execution of test
==================
WARNING: DATA RACE
Write at 0x00c00010e150 by goroutine 82:
  github.com/grafana/loki/pkg/ingester.(*Ingester).collectChunksToFlush()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:252 +0x676
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:216 +0xbf
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c00010e150 by goroutine 83:
  github.com/grafana/loki/pkg/ingester.(*stream).Iterator()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/stream.go:261 +0x4e5
  github.com/grafana/loki/pkg/ingester.TestTransferOut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/transfer_test.go:90 +0x11d5
  testing.tRunner()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:991 +0x1eb

Goroutine 82 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 83 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1042 +0x660
  testing.runTests.func1()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1284 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:991 +0x1eb
  testing.runTests()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1282 +0x527
  testing.(*M).Run()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1199 +0x2ff
  main.main()
      _testmain.go:80 +0x223
==================
==================
WARNING: DATA RACE
Write at 0x00c00010e120 by goroutine 82:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:283 +0xd6f
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:330 +0x6e9
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c00010e120 by goroutine 83:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Iterator()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:481 +0x28d
  github.com/grafana/loki/pkg/ingester.(*stream).Iterator()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/stream.go:262 +0x621
  github.com/grafana/loki/pkg/ingester.TestTransferOut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/transfer_test.go:90 +0x11d5
  testing.tRunner()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:991 +0x1eb

Goroutine 82 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 83 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1042 +0x660
  testing.runTests.func1()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1284 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:991 +0x1eb
  testing.runTests()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1282 +0x527
  testing.(*M).Run()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1199 +0x2ff
  main.main()
      _testmain.go:80 +0x223
==================
==================
WARNING: DATA RACE
Write at 0x00c00010e158 by goroutine 82:
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:231 +0x39f
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c00010e158 by goroutine 83:
  github.com/grafana/loki/pkg/ingester.(*stream).Iterator()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/stream.go:261 +0x4e5
  github.com/grafana/loki/pkg/ingester.TestTransferOut()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/transfer_test.go:90 +0x11d5
  testing.tRunner()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:991 +0x1eb

Goroutine 82 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /Users/ctovena/go/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /Users/ctovena/go/src/github.com/grafana/loki/vendor/github.com/cortexproject/cortex/pkg/util/services/basic_service.go:134 +0x6f8

Goroutine 83 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1042 +0x660
  testing.runTests.func1()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1284 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:991 +0x1eb
  testing.runTests()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1282 +0x527
  testing.(*M).Run()
      /usr/local/Cellar/go/1.14.2_1/libexec/src/testing/testing.go:1199 +0x2ff
  main.main()
      _testmain.go:80 +0x223
==================
--- FAIL: TestTransferOut (0.31s)
    testing.go:906: race detected during execution of test
FAIL
FAIL	github.com/grafana/loki/pkg/ingester	89.175s
FAIL

I don't think they all are legitimate, and probably generated by the tests, but we should make sure this is the case and solve them all.
@cyriltovena cyriltovena added component/loki help wanted We would love help on these issues. Please come help us! labels Jun 26, 2020
@adityacs
Copy link
Contributor

adityacs commented Jul 3, 2020 •
edited
Loading

We have more data races. I found this while running logcli queries. Looks like we have to protect all chunk head block read and write. This is happening because while we query the chunk, flush loop might be trying to encode the chunk, this involves lot of head block read and write.

Mainly data race is happening in these two blocks. However, all read and write of head block should be protected.

loki/pkg/chunkenc/memchunk.go

Lines 437 to 450 in 3845204

c.blocks = append(c.blocks, block{
readers: c.readers,
b: b,
numEntries: len(c.head.entries),
mint: c.head.mint,
maxt: c.head.maxt,
uncompressedSize: c.head.size,
})
c.cutBlockSize += len(b)
c.head.entries = c.head.entries[:0]
c.head.mint = 0 // Will be set on first append.
c.head.size = 0

loki/pkg/chunkenc/memchunk.go

Lines 458 to 470 in 3845204

if len(c.blocks) > 0 {
from = c.blocks[0].mint
to = c.blocks[len(c.blocks)-1].maxt
}
if !c.head.isEmpty() {
if from == 0 || from > c.head.mint {
from = c.head.mint
}
if to < c.head.maxt {
to = c.head.maxt
} 

> go run -race main.go -config.file=loki-local-config.yaml
WARNING: DATA RACE
Write at 0x00c000243e10 by goroutine 154:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:437 +0x378
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a5
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /loki/pkg/mod/github.com/cortexproject/[email protected]/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:331 +0x76d
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c000243e10 by goroutine 8:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Iterator()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:479 +0x16a
  github.com/grafana/loki/pkg/ingester.(*stream).Iterator()
      /loki/src/github.com/grafana/loki/pkg/ingester/stream.go:262 +0x610
  github.com/grafana/loki/pkg/ingester.(*instance).Query.func1()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:211 +0x258
  github.com/grafana/loki/pkg/ingester.(*instance).forMatchingStreams()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:340 +0x380
  github.com/grafana/loki/pkg/ingester.(*instance).Query()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:207 +0x228
  github.com/grafana/loki/pkg/ingester.(*Ingester).Query()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:288 +0x219
  github.com/grafana/loki/pkg/logproto._Querier_Query_Handler()
      /loki/src/github.com/grafana/loki/pkg/logproto/logproto.pb.go:2222 +0x135
  github.com/grafana/loki/pkg/loki.glob..func3()
      /loki/src/github.com/grafana/loki/pkg/loki/fake_auth.go:29 +0x154
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:44 +0xf0
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
      /loki/pkg/mod/github.com/opentracing-contrib/[email protected]/server.go:114 +0x427
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/weaveworks/common/middleware.StreamServerInstrumentInterceptor.func1()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_instrumentation.go:42 +0xa6
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/weaveworks/common/middleware.GRPCServerLog.StreamServerInterceptor()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_logging.go:49 +0xa8
  github.com/weaveworks/common/middleware.GRPCServerLog.StreamServerInterceptor-fm()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_logging.go:47 +0xa1
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0x135
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:1336 +0xa95
  google.golang.org/grpc.(*Server).handleStream()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:1409 +0x1271
  google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:746 +0xc8

Goroutine 154 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /loki/pkg/mod/github.com/cortexproject/[email protected]/pkg/util/services/basic_service.go:134 +0x701

Goroutine 8 (finished) created at:
  google.golang.org/grpc.(*Server).serveStreams.func1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:744 +0xb8
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /loki/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:442 +0x16ab
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /loki/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:483 +0x459
  google.golang.org/grpc.(*Server).serveStreams()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:742 +0x19a
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:703 +0x4c
==================
==================
WARNING: DATA RACE
Write at 0x00c000742de0 by goroutine 154:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:448 +0x456
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a5
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /loki/pkg/mod/github.com/cortexproject/[email protected]/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:331 +0x76d
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c000742de0 by goroutine 8:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Iterator()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:98 +0x36e
  github.com/grafana/loki/pkg/ingester.(*stream).Iterator()
      /loki/src/github.com/grafana/loki/pkg/ingester/stream.go:262 +0x610
  github.com/grafana/loki/pkg/ingester.(*instance).Query.func1()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:211 +0x258
  github.com/grafana/loki/pkg/ingester.(*instance).forMatchingStreams()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:340 +0x380
  github.com/grafana/loki/pkg/ingester.(*instance).Query()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:207 +0x228
  github.com/grafana/loki/pkg/ingester.(*Ingester).Query()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:288 +0x219
  github.com/grafana/loki/pkg/logproto._Querier_Query_Handler()
      /loki/src/github.com/grafana/loki/pkg/logproto/logproto.pb.go:2222 +0x135
  github.com/grafana/loki/pkg/loki.glob..func3()
      /loki/src/github.com/grafana/loki/pkg/loki/fake_auth.go:29 +0x154
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:44 +0xf0
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
      /loki/pkg/mod/github.com/opentracing-contrib/[email protected]/server.go:114 +0x427
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/weaveworks/common/middleware.StreamServerInstrumentInterceptor.func1()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_instrumentation.go:42 +0xa6
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/weaveworks/common/middleware.GRPCServerLog.StreamServerInterceptor()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_logging.go:49 +0xa8
  github.com/weaveworks/common/middleware.GRPCServerLog.StreamServerInterceptor-fm()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_logging.go:47 +0xa1
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0x135
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:1336 +0xa95
  google.golang.org/grpc.(*Server).handleStream()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:1409 +0x1271
  google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:746 +0xc8

Goroutine 154 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /loki/pkg/mod/github.com/cortexproject/[email protected]/pkg/util/services/basic_service.go:134 +0x701

Goroutine 8 (finished) created at:
  google.golang.org/grpc.(*Server).serveStreams.func1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:744 +0xb8
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /loki/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:442 +0x16ab
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /loki/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:483 +0x459
  google.golang.org/grpc.(*Server).serveStreams()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:742 +0x19a
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:703 +0x4c
==================
==================
WARNING: DATA RACE
Write at 0x00c000742e00 by goroutine 154:
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).cut()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:449 +0x49a
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Bytes()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:256 +0x12a5
  github.com/grafana/loki/pkg/chunkenc.Facade.Marshal()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/facade.go:47 +0x4b
  github.com/grafana/loki/pkg/chunkenc.(*Facade).Marshal()
      <autogenerated>:1 +0xd5
  github.com/cortexproject/cortex/pkg/chunk.(*Chunk).Encode()
      /loki/pkg/mod/github.com/cortexproject/[email protected]/pkg/chunk/chunk.go:220 +0x4dd
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushChunks()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:331 +0x76d
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushUserSeries()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:224 +0x2c8
  github.com/grafana/loki/pkg/ingester.(*Ingester).flushLoop()
      /loki/src/github.com/grafana/loki/pkg/ingester/flush.go:196 +0x5cc

Previous read at 0x00c000742e00 by goroutine 8:
  github.com/grafana/loki/pkg/chunkenc.(*headBlock).iterator()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:540 +0x63d
  github.com/grafana/loki/pkg/chunkenc.(*MemChunk).Iterator()
      /loki/src/github.com/grafana/loki/pkg/chunkenc/memchunk.go:489 +0x88b
  github.com/grafana/loki/pkg/ingester.(*stream).Iterator()
      /loki/src/github.com/grafana/loki/pkg/ingester/stream.go:262 +0x610
  github.com/grafana/loki/pkg/ingester.(*instance).Query.func1()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:211 +0x258
  github.com/grafana/loki/pkg/ingester.(*instance).forMatchingStreams()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:340 +0x380
  github.com/grafana/loki/pkg/ingester.(*instance).Query()
      /loki/src/github.com/grafana/loki/pkg/ingester/instance.go:207 +0x228
  github.com/grafana/loki/pkg/ingester.(*Ingester).Query()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:288 +0x219
  github.com/grafana/loki/pkg/logproto._Querier_Query_Handler()
      /loki/src/github.com/grafana/loki/pkg/logproto/logproto.pb.go:2222 +0x135
  github.com/grafana/loki/pkg/loki.glob..func3()
      /loki/src/github.com/grafana/loki/pkg/loki/fake_auth.go:29 +0x154
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/grpc-ecosystem/go-grpc-middleware/recovery.StreamServerInterceptor.func1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:44 +0xf0
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/opentracing-contrib/go-grpc.OpenTracingStreamServerInterceptor.func1()
      /loki/pkg/mod/github.com/opentracing-contrib/[email protected]/server.go:114 +0x427
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/weaveworks/common/middleware.StreamServerInstrumentInterceptor.func1()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_instrumentation.go:42 +0xa6
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/weaveworks/common/middleware.GRPCServerLog.StreamServerInterceptor()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_logging.go:49 +0xa8
  github.com/weaveworks/common/middleware.GRPCServerLog.StreamServerInterceptor-fm()
      /loki/pkg/mod/github.com/weaveworks/[email protected]/middleware/grpc_logging.go:47 +0xa1
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 +0x7a
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1()
      /loki/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 +0x135
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:1336 +0xa95
  google.golang.org/grpc.(*Server).handleStream()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:1409 +0x1271
  google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:746 +0xc8

Goroutine 154 (running) created at:
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:171 +0xb3
  github.com/grafana/loki/pkg/ingester.(*Ingester).starting-fm()
      /loki/src/github.com/grafana/loki/pkg/ingester/ingester.go:167 +0x55
  github.com/cortexproject/cortex/pkg/util/services.(*BasicService).main()
      /loki/pkg/mod/github.com/cortexproject/[email protected]/pkg/util/services/basic_service.go:134 +0x701

Goroutine 8 (finished) created at:
  google.golang.org/grpc.(*Server).serveStreams.func1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:744 +0xb8
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /loki/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:442 +0x16ab
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /loki/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_server.go:483 +0x459
  google.golang.org/grpc.(*Server).serveStreams()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:742 +0x19a
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /loki/pkg/mod/google.golang.org/[email protected]/server.go:703 +0x4c

This was referenced Jul 3, 2020
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/loki help wanted We would love help on these issues. Please come help us!
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix data race in ingester adityacs/loki
2 participants
@cyriltovena @adityacs