Code

Adds WAL support (experimental)#2981

Code

Merged

owen-d merged 58 commits intografana:mastergrafana/loki:masterfrom owen-d:wal/headblockowen-d/loki:wal/headblockCopy head branch name to clipboardNov 27, 2020

+2,387-308

Commits

Commits on Oct 27, 2020

• marshalable chunks

  owen-d

  committed
  1b2b931

  Copy full SHA for 1b2b931

• wal record types custom serialization

  owen-d

  committed
  ddecf02

  Copy full SHA for ddecf02

• proto types for wal checkpoints

  owen-d

  committed
  48fa77f

  Copy full SHA for 48fa77f

• byteswith output unaffected by buffer

  owen-d

  committed
  7ec0fb0

  Copy full SHA for 7ec0fb0

Commits on Oct 28, 2020

• wal & record pool ifcs

  owen-d

  committed
  d6ccccf

  Copy full SHA for d6ccccf

• wal record can hold entries from multiple series

  owen-d

  committed
  eb6a519

  Copy full SHA for eb6a519

• entry pool

  owen-d

  committed
  f089f09

  Copy full SHA for f089f09

Commits on Oct 29, 2020

• ingester uses noopWal

  owen-d

  committed
  211356d

  Copy full SHA for 211356d

• removes duplicate argument passing in ingester code. adds ingester config validation & derives chunk encoding.

  owen-d

  committed
  7adf966

  Copy full SHA for 7adf966

Commits on Nov 4, 2020

• Merge remote-tracking branch 'upstream/master' into wal/writes

  owen-d

  committed
  29fafd6

  Copy full SHA for 29fafd6

• segment writing

  owen-d

  committed
  5ed8ffb

  Copy full SHA for 5ed8ffb

Commits on Nov 5, 2020

• [WIP] wal recovery from segments

  owen-d

  committed
  107b03b

  Copy full SHA for 107b03b

• replay uses sync.Maps & preserves WAL fingerprints

  owen-d

  committed
  fb194f5

  Copy full SHA for fb194f5

• in memory wal recovery

  owen-d

  committed
  3d36b2d

  Copy full SHA for 3d36b2d

Commits on Nov 6, 2020

• wal segment recovery

  owen-d

  committed
  9a7eaa5

  Copy full SHA for 9a7eaa5

• ingester metrics struct

  owen-d

  committed
  30cdbf2

  Copy full SHA for 30cdbf2

Commits on Nov 9, 2020

• wal replay locks streamsMtx in instances, adds checkpoint codec

  owen-d

  committed
  d723c0b

  Copy full SHA for d723c0b

• ingester metrics

  owen-d

  committed
  897196e

  Copy full SHA for 897196e

Commits on Nov 10, 2020

• checkpointer

  owen-d

  committed
  8df2598

  Copy full SHA for 8df2598

• WAL checkpoint writer

  owen-d

  committed
  b5c2ef0

  Copy full SHA for b5c2ef0

• checkpointwriter can write multiple checkpoints

  owen-d

  committed
  60ebe5e

  Copy full SHA for 60ebe5e

• reorgs checkpointing

  owen-d

  committed
  ed14e0c

  Copy full SHA for ed14e0c

• wires up checkpointwriter to wal

  owen-d

  committed
  5acf745

  Copy full SHA for 5acf745

• ingester SeriesIter impl

  owen-d

  committed
  6dd28b1

  Copy full SHA for 6dd28b1

• wires up ingesterRecoverer to consume checkpoints

  owen-d

  committed
  a7e40fd

  Copy full SHA for a7e40fd

• generic recovery fn

  owen-d

  committed
  957717b

  Copy full SHA for 957717b

• generic recovery fn

  owen-d

  committed
  9384e98

  Copy full SHA for 9384e98

• recover from both wal types

  owen-d

  committed
  a1beb69

  Copy full SHA for a1beb69

Commits on Nov 12, 2020

• cleans up old tmp checkpoints & allows aborting in flight checkpoints

  owen-d

  committed
  4e27f7c

  Copy full SHA for 4e27f7c

• wires up wal checkpointing

  owen-d

  committed
  45840bc

  Copy full SHA for 45840bc

• more granular wal logging

  owen-d

  committed
  d1b7f6c

  Copy full SHA for d1b7f6c

• fixes off by 1 wal truncation & removes double logging

  owen-d

  committed
  2ca359d

  Copy full SHA for 2ca359d

• adds userID to wal records correctly

  owen-d

  committed
  ff6b35b

  Copy full SHA for ff6b35b

• wire chunk encoding tests

  owen-d

  committed
  d811aef

  Copy full SHA for d811aef

• more granular wal metrics

  owen-d

  committed
  6eb4ec0

  Copy full SHA for 6eb4ec0

• checkpoint encoding test

  owen-d

  committed
  7962164

  Copy full SHA for 7962164

Commits on Nov 13, 2020

• Merge remote-tracking branch 'upstream/master' into wal/writes

  owen-d

  committed
  1961726

  Copy full SHA for 1961726

• ignores debug bins

  owen-d

  committed
  67f298d

  Copy full SHA for 67f298d

• Merge remote-tracking branch 'upstream/master' into wal/writes

  owen-d

  committed
  d4318c7

  Copy full SHA for d4318c7

• segment replay ignores out of orders

  owen-d

  committed
  d43ecde

  Copy full SHA for d43ecde

• fixes bug between WAL reading []byte validity and proto unmarshalling refs

  owen-d

  committed
  ce72f33

  Copy full SHA for ce72f33

• conf validations, removes comments

  owen-d

  committed
  c9614aa

  Copy full SHA for c9614aa

Commits on Nov 18, 2020

• flush on shutdown config

  owen-d

  committed
  571f8fc

  Copy full SHA for 571f8fc

• POST /ingester/shutdown

  owen-d

  committed
  2c965f3

  Copy full SHA for 2c965f3

• renames flush on shutdown

  owen-d

  committed
  1c4d526

  Copy full SHA for 1c4d526

• wal & checkpoint use same segment size

  owen-d

  committed
  d9139de

  Copy full SHA for d9139de

Commits on Nov 19, 2020

• Merge remote-tracking branch 'upstream/master' into wal/writes

  owen-d

  committed
  9a4c810

  Copy full SHA for 9a4c810

• writes entries to wal regardless of tailers

  owen-d

  committed
  bd9cee0

  Copy full SHA for bd9cee0

Commits on Nov 20, 2020

• makes wal checkpoing duration default to 5m

  owen-d

  committed
  e042812

  Copy full SHA for e042812

• recovery metrics

  owen-d

  committed
  f632022

  Copy full SHA for f632022

• encodes headchunks separately for wal purposes

  owen-d

  committed
  a948785

  Copy full SHA for a948785

Commits on Nov 23, 2020

• merge upstream

  owen-d

  committed
  2e972c7

  Copy full SHA for 2e972c7

• typo

  owen-d

  committed
  43d8a78

  Copy full SHA for 43d8a78

• linting

  owen-d

  committed
  3166183

  Copy full SHA for 3166183

Commits on Nov 25, 2020

• Merge remote-tracking branch 'upstream/master' into wal/headblock

  owen-d

  committed
  6059256

  Copy full SHA for 6059256

• addresses pr feedback

  Show description for 51ff8d7
  owen-d

  committed
  51ff8d7

  Copy full SHA for 51ff8d7

• prevent shared access bug with tailers and entry pool

  owen-d

  committed
  ff211ce

  Copy full SHA for ff211ce

Commits on Nov 27, 2020

• removes stream push entry pool optimization

  owen-d

  committed
  cb9bc46

  Copy full SHA for cb9bc46