Code

Add index correction internal logic#2194

Code

Merged

vankichi merged 103 commits intomainvdaas/vald:mainfrom feature/index/correction-jobvdaas/vald:feature/index/correction-jobCopy head branch name to clipboardOct 3, 2023

+3,103-10

Commits

Commits on Sep 15, 2023

• implement the initail framework

  ykadowak

  committed
  a1ee9d8

• add corrector configuration

  ykadowak

  committed
  8a1221d

• add corrector logic

  ykadowak

  committed
  fa1eb75

• add build make command for index correction binary

  ykadowak

  committed
  39e7d71

• add Dockerfile for index correction

  ykadowak

  committed
  2bd41e9

• add Docker image for index job correction

  ykadowak

  committed
  6ef29e3

• add timer

  ykadowak

  committed
  c7e1ff2

• fix tag align

  ykadowak

  committed
  22dcaaa

• tmp

  ykadowak

  committed
  d51e26b

• fix log

  ykadowak

  committed
  6aa8fde

• temporally implement two versions of correct function

  ykadowak

  committed
  41faffd

• set eg limit from config

  ykadowak

  committed
  8f5bcf0

• add stream list concurrency config

  ykadowak

  committed
  065e9e2

• implement index id caching

  ykadowak

  committed
  8562a99

• add config to use cache or not

  ykadowak

  committed
  81345f3

• style: Format code with prettier and gofumpt

  

  deepsource-autofix[bot]

  authored and

  ykadowak

  committed
  77c33f0

• refactor availableAddrs

  ykadowak

  committed
  72c3956

• add kvs range duration

  ykadowak

  committed
  5ffd9b3

• add leftAgentAddrs for performance

  ykadowak

  committed
  5d3c6c7

• Revert "add kvs range duration"

  Show description for 3b3710b
  ykadowak

  committed
  3b3710b

• refactor

  ykadowak

  committed
  6b3934e

• fix without cache bug

  ykadowak

  committed
  bc591ae

• enable observability

  ykadowak

  committed
  5209a72

• refactor

  ykadowak

  committed
  a0cb7aa

• SIGTERM after complete

  ykadowak

  committed
  b453e61

• add metrics server

  ykadowak

  committed
  db6868d

• add pcache

  ykadowak

  committed
  7a630b3

• remove comment

  ykadowak

  committed
  ca84f56

• [TEMP] use pcache

  ykadowak

  committed
  feda428

• [TMP] use pcache

  ykadowak

  committed
  c229a87

• fix empty shard returns error

  ykadowak

  committed
  fa0b68b

• fix to use local map

  ykadowak

  committed
  06b312b

• [TMP] add prestop for pcache

  ykadowak

  committed
  490345f

• [TEMP] add pcache config

  ykadowak

  committed
  05838c3

• style: Format code with prettier and gofumpt

  

  deepsource-autofix[bot]

  authored and

  ykadowak

  committed
  9805e7e

• [TEMP] add pcache log

  ykadowak

  committed
  a0d698e

• fix map alloc size

  ykadowak

  committed
  9e5f2ae

• [TMP] Add bbolt cache

  ykadowak

  committed
  3728276

• update bbolt

  ykadowak

  committed
  1678ff7

• fix bbolt bug

  ykadowak

  committed
  4f87601

• add bbolt test

  ykadowak

  committed
  01225aa

• [TEMP] use bbolt as persistent cache

  ykadowak

  committed
  428ee35

• style: Format code with prettier and gofumpt

  

  deepsource-autofix[bot]

  authored and

  ykadowak

  committed
  81b800e

• add SetBatch to bbolt

  ykadowak

  committed
  bfc73c4

• use batch to write map to disk

  ykadowak

  committed
  d4f3695

• style: Format code with prettier and gofumpt

  

  deepsource-autofix[bot]

  authored and

  ykadowak

  committed
  9975237

• delete the map elements on finalize

  ykadowak

  committed
  2b00e05

• manually call GC after the map shrink

  ykadowak

  committed
  a611e88

• add limit to SetBatch goroutine number

  ykadowak

  committed
  e21441c

• stop unnecesarry GC

  ykadowak

  committed
  66bc79c

• increase eg limit to the MaxBatchSize

  ykadowak

  committed
  3a85405

• use ch to set batch bbolt

  ykadowak

  committed
  16d2986

• fix servers shutdown properly

  ykadowak

  committed
  919c9f9

• use internal/kvs/bbolt

  ykadowak

  committed
  9bd57c0

• refactor

  ykadowak

  committed
  55c6586

Commits on Sep 19, 2023

• always use bbolt cache for correction

  ykadowak

  committed
  bc92686

• update sample.yaml for correction

  ykadowak

  committed
  319ec8b

• style: format code with Prettier and Gofumpt

  Show description for eb5cdb0
  deepsource-autofix[bot]

  authored
  eb5cdb0

• use go std slices pkg

  ykadowak

  committed
  d3616e5

• refactor

  ykadowak

  committed
  89e2339

• add comment

  ykadowak

  committed
  e2eeabc

• remove valdsync

  ykadowak

  committed
  9a9e622

• use vald errgroup

  ykadowak

  committed
  4c22bcc

• Merge branch 'main' into feature/index/correction-job

  ykadowak

  authored
  e0ef7a8

• refactor

  ykadowak

  committed
  4dc4801

Commits on Sep 20, 2023

• Define ErrNoAvailableAgentToInsert

  ykadowak

  committed
  705dc29

• update comment in English

  ykadowak

  committed
  952a6e7

• Apply new actions yaml format

  ykadowak

  committed
  5f93398

• Disable godox

  ykadowak

  committed
  c860ddc

• style: format code with Prettier and Gofumpt

  Show description for 2c2bb09
  deepsource-autofix[bot]

  authored
  2c2bb09

• remove comment

  ykadowak

  committed
  2ecf8cf

• Apply format

  ykadowak

  committed
  8bb271d

• Add type check for type assertion

  ykadowak

  committed
  e5f73a3

• use const to specify filemode

  ykadowak

  committed
  d3e4d78

• Add bbolt concurrency as config

  ykadowak

  committed
  7e5cd71

• fix var style

  ykadowak

  committed
  a184b22

• Suppress linter

  ykadowak

  committed
  4972294

• fix comment

  ykadowak

  committed
  ef93123

• add test template

  ykadowak

  committed
  375c04e

Commits on Sep 25, 2023

• Refactor parameters for index correction

  ykadowak

  committed
  33ad603

• Refactor config

  ykadowak

  committed
  55bc821

• Add corrector test

  ykadowak

  committed
  004bf81

• style: format code with Prettier and Gofumpt

  Show description for 97ff3c0
  deepsource-autofix[bot]

  authored
  97ff3c0

• Merge branch 'main' into feature/index/correction-job

  ykadowak

  authored
  df88705

Commits on Sep 26, 2023

• Add timestamp check

  ykadowak

  committed
  8b0540d

• Apply format

  ykadowak

  committed
  5430f67

• fix schema type

  ykadowak

  committed
  bdc9c70

• Fix DeepSource errors

  ykadowak

  committed
  f7ffd6a

• Fix misspell

  ykadowak

  committed
  5e73905

• Add type check

  ykadowak

  committed
  ce78ccf

• Remove unused config

  ykadowak

  committed
  ef2e20d

• Fix DeepSource error

  ykadowak

  committed
  93d6ab7

• Add required go:build e2e tag

  ykadowak

  committed
  31058f5

• Remove memo

  ykadowak

  committed
  6f79736

• Refactor comment

  ykadowak

  committed
  0d4d9d2

Commits on Sep 27, 2023

• Remove TODO comment that is already done

  ykadowak

  committed
  0eebe24

Commits on Oct 2, 2023

• Remove unused config

  ykadowak

  committed
  0eba31e

• Add comment to errors

  ykadowak

  committed
  5a73044

Commits on Oct 3, 2023

• change app name

  ykadowak

  committed
  5117332

• replace filepath pkg with internal file

  Show description for 74bd1c6
  ykadowak

  committed
  74bd1c6

• Refactor

  Show description for d687013
  ykadowak

  committed
  d687013

• Fix gRPC spelling

  ykadowak

  committed
  ba241cb

• Remove memo

  ykadowak

  committed
  436da6e