Skip to content

Commit

Permalink
compact: return metas copy from syncer (thanos-io#6801)
Browse files Browse the repository at this point in the history
Return copy of the map because the compactor runs garbage collector
concurrently that deletes entries from the original map. Fixes race:

```
10:55:35 compact-working-dedup: ==================
10:55:35 compact-working-dedup: WARNING: DATA RACE
10:55:35 compact-working-dedup: Write at 0x00c001822150 by goroutine 220:
10:55:35 compact-working-dedup: runtime.mapdelete()
10:55:35 compact-working-dedup: /usr/local/go/src/runtime/map.go:696 +0x0
10:55:35 compact-working-dedup: github.com/thanos-io/thanos/pkg/compact.(*Syncer).GarbageCollect()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:201 +0x324
10:55:35 compact-working-dedup: github.com/thanos-io/thanos/pkg/compact.(*BucketCompactor).Compact()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:1422 +0x60f
10:55:35 compact-working-dedup: main.runCompact.func7()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:426 +0xfa
10:55:35 compact-working-dedup: main.runCompact.func8.1()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:481 +0x69
10:55:35 compact-working-dedup: github.com/thanos-io/thanos/pkg/runutil.Repeat()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:74 +0xc3
10:55:35 compact-working-dedup: main.runCompact.func8()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:480 +0x224
10:55:35 compact-working-dedup: github.com/oklog/run.(*Group).Run.func1()
10:55:35 compact-working-dedup: /go/pkg/mod/github.com/oklog/[email protected]/group.go:38 +0x39
10:55:35 compact-working-dedup: github.com/oklog/run.(*Group).Run.func2()
10:55:35 compact-working-dedup: /go/pkg/mod/github.com/oklog/[email protected]/group.go:39 +0x4f
10:55:35 compact-working-dedup: Previous read at 0x00c001822150 by goroutine 223:
10:55:35 compact-working-dedup: runtime.mapiternext()
10:55:35 compact-working-dedup: /usr/local/go/src/runtime/map.go:867 +0x0
10:55:35 compact-working-dedup: github.com/thanos-io/thanos/pkg/compact.(*DefaultGrouper).Groups()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/pkg/compact/compact.go:289 +0xfd
10:55:35 compact-working-dedup: main.runCompact.func16.1()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:626 +0x4ae
10:55:35 compact-working-dedup: github.com/thanos-io/thanos/pkg/runutil.Repeat()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/pkg/runutil/runutil.go:74 +0xc3
10:55:35 compact-working-dedup: main.runCompact.func16()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/cmd/thanos/compact.go:591 +0x3f9
10:55:35 compact-working-dedup: github.com/oklog/run.(*Group).Run.func1()
10:55:35 compact-working-dedup: /go/pkg/mod/github.com/oklog/[email protected]/group.go:38 +0x39
10:55:35 compact-working-dedup: github.com/oklog/run.(*Group).Run.func2()
10:55:35 compact-working-dedup: /go/pkg/mod/github.com/oklog/[email protected]/group.go:39 +0x4f
10:55:35 compact-working-dedup: Goroutine 220 (running) created at:
10:55:35 compact-working-dedup: github.com/oklog/run.(*Group).Run()
10:55:35 compact-working-dedup: /go/pkg/mod/github.com/oklog/[email protected]/group.go:37 +0xad
10:55:35 compact-working-dedup: main.main()
10:55:35 compact-working-dedup: /go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:159 +0x2964
```

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: bazooka3000 <[email protected]>
  • Loading branch information
GiedriusS authored and bazooka3000 committed Nov 19, 2023
1 parent b91b8ec commit 337ab34
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion pkg/compact/compact.go
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,12 @@ func (s *Syncer) Metas() map[ulid.ULID]*metadata.Meta {
s.mtx.Lock()
defer s.mtx.Unlock()

return s.blocks
metas := make(map[ulid.ULID]*metadata.Meta, len(s.blocks))
for k, v := range s.blocks {
metas[k] = v
}

return metas
}

// GarbageCollect marks blocks for deletion from bucket if their data is available as part of a
Expand Down

0 comments on commit 337ab34

Please sign in to comment.