Code

Concurrent file directories index#15

Code

Draft

jacobmou wants to merge 36 commits intomasterjacobmou/bazel-buildfarm:masterfrom concurrent_FileDirectoriesIndexjacobmou/bazel-buildfarm:concurrent_FileDirectoriesIndexCopy head branch name to clipboard

+261-90

Commits

Commits on Jun 30, 2020

• split FileDirectoriesIndex single db to 100 to improve concurrency

  

  Jianqiu

  committed
  2907e70

  Copy full SHA for 2907e70

• fix compiling error

  

  Jianqiu

  committed
  64c8850

  Copy full SHA for 64c8850

• remove logging

  

  Jianqiu

  committed
  c17cdd3

  Copy full SHA for c17cdd3

• add queue for each sqlite db as buffer zone

  

  Jianqiu

  committed
  502cc2d

  Copy full SHA for 502cc2d

• add logging for debug

  

  Jianqiu

  committed
  5417f09

  Copy full SHA for 5417f09

Commits on Jul 1, 2020

• adjust cas-test cas size and HashFunction and add logging

  

  Jianqiu

  committed
  ede212e

  Copy full SHA for ede212e

• drain all queues each time any queue is full

  

  Jianqiu

  committed
  1cc0f23

  Copy full SHA for 1cc0f23

• avoid npe

  

  Jianqiu

  committed
  91890be

  Copy full SHA for 91890be

• synchronize queues draining

  

  Jianqiu

  committed
  2648403

  Copy full SHA for 2648403

• increae max total queue size

  

  Jianqiu

  committed
  57859ef

  Copy full SHA for 57859ef

Commits on Jul 2, 2020

• adjust synchronization point and number of database

  

  Jianqiu

  committed
  48c1d54

  Copy full SHA for 48c1d54

• adjust synchronization point

  

  Jianqiu

  committed
  2070ff1

  Copy full SHA for 2070ff1

• adjust synchronization

  

  Jianqiu

  committed
  d452210

  Copy full SHA for d452210

• maintain all in memory and then flush to disk

  

  Jianqiu

  committed
  8a574af

  Copy full SHA for 8a574af

• avoid logging per entry

  

  Jianqiu

  committed
  88a4a07

  Copy full SHA for 88a4a07

• drain each queue independently

  

  Jianqiu

  committed
  0c7023b

  Copy full SHA for 0c7023b

• add logging for debugging

  

  Jianqiu

  committed
  6580d55

  Copy full SHA for 6580d55

• adjust queue size

  

  Jianqiu

  committed
  385af36

  Copy full SHA for 385af36

• adjust queue size and queue number

  

  Jianqiu

  committed
  bff954d

  Copy full SHA for bff954d

Commits on Jul 5, 2020

• drain queue concurrently

  

  Jianqiu

  committed
  cec07a6

  Copy full SHA for cec07a6

• increase total queue size

  

  Jianqiu

  committed
  16e19f6

  Copy full SHA for 16e19f6

• increase total number of database to reduce lock retension in adding

  

  Jianqiu

  committed
  75bdbe3

  Copy full SHA for 75bdbe3

• drain queue independently

  

  Jianqiu

  committed
  61cf41b

  Copy full SHA for 61cf41b

• incresease separate queue size

  

  Jianqiu

  committed
  bccffb2

  Copy full SHA for bccffb2

Commits on Jul 6, 2020

• using string array to replace MapEntry

  

  Jianqiu

  committed
  2b97d0e

  Copy full SHA for 2b97d0e

• adjust logging

  

  Jianqiu

  committed
  589fcba

  Copy full SHA for 589fcba

Commits on Jul 8, 2020

• use 40 times number of queue

  

  Jianqiu

  committed
  783ea45

  Copy full SHA for 783ea45

• logging out of synchronization block

  

  Jianqiu

  committed
  2e3fe95

  Copy full SHA for 2e3fe95

• drain queue concurrently

  

  Jianqiu

  committed
  5b03122

  Copy full SHA for 5b03122

• adjust number of thread

  

  Jianqiu

  committed
  68e9d80

  Copy full SHA for 68e9d80

• add synchronization

  

  Jianqiu

  committed
  ab29fa7

  Copy full SHA for ab29fa7

Commits on Jul 9, 2020

• double producer count

  

  Jianqiu

  committed
  4848ce1

  Copy full SHA for 4848ce1

• use ConcurrentHashMap

  

  Jianqiu

  committed
  8432eba

  Copy full SHA for 8432eba

• unlimited queue

  

  Jianqiu

  committed
  8cc33ce

  Copy full SHA for 8cc33ce

• add logging

  

  Jianqiu

  committed
  042c9e0

  Copy full SHA for 042c9e0

Commits on Jul 10, 2020

• drain queue independently

  

  Jianqiu

  committed
  b258c24

  Copy full SHA for b258c24