Code

delete_timeline: request coalescing#4159

Code

Closed

koivunej wants to merge 66 commits intomainneondatabase/neon:mainfrom joonas/dkr/deleted-flag-in-remote-index-contneondatabase/neon:joonas/dkr/deleted-flag-in-remote-index-contCopy head branch name to clipboard

+1,059-119

Commits

Commits on Apr 21, 2023

• Use is_deleted flag in IndexPart to prevent timeline resurrection

  Show description for 9fda377
  LizardWizzard

  committed
  9fda377

  Copy full SHA for 9fda377

• do better at preventing racing index part uploads during timeline delete

  LizardWizzard

  committed
  f0ab08c

  Copy full SHA for f0ab08c

• tests: improve assert, change wording

  LizardWizzard

  committed
  e855479

  Copy full SHA for e855479

• use separate enum variant to represent index part for deleted timeline

  LizardWizzard

  committed
  23b3c66

  Copy full SHA for 23b3c66

• combine UploadedQueueStopped fields

  LizardWizzard

  committed
  3ffdb19

  Copy full SHA for 3ffdb19

• some more review suggestions

  LizardWizzard

  committed
  d826cac

  Copy full SHA for d826cac

• Update libs/remote_storage/src/local_fs.rs

  Show description for dbdec75
  

  LizardWizzard

  and

  koivunej

  committed
  dbdec75

  Copy full SHA for dbdec75

• replace bool is_deleted with deleted_at containing a timestamp

  LizardWizzard

  committed
  3b818d2

  Copy full SHA for 3b818d2

• delete local data when facing timeline that is marked as deleted in s3

  LizardWizzard

  committed
  efe914f

  Copy full SHA for efe914f

• add test

  LizardWizzard

  committed
  aba2d60

  Copy full SHA for aba2d60

Commits on Apr 27, 2023

• improve the test added in previous commit

  Show description for 33b5248
  problame

  committed
  33b5248

  Copy full SHA for 33b5248

• clean up test_timeline_resurrection_on_attach

  Show description for dd36f54
  problame

  committed
  dd36f54

  Copy full SHA for dd36f54

• test_timeline_resurrection_on_attach: improve variable naming & simplify assertions

  Show description for dead779
  problame

  committed
  dead779

  Copy full SHA for dead779

• test_timeline_resurrection_on_attach: have variant that doesn't fill the branch

  problame

  committed
  b69d12d

  Copy full SHA for b69d12d

• cleanup: persist_index_part_with_deleted_flag doesn't need self: &Arc<Self>

  problame

  committed
  95b560d

  Copy full SHA for 95b560d

• utils::failpoint_sleep_millis_async: ability to run an async block before sleeping

  problame

  committed
  bcf49d3

  Copy full SHA for bcf49d3

• add (failing) test case for index upload causing delete timeline request timeout

  Show description for 958fd57
  problame

  committed
  958fd57

  Copy full SHA for 958fd57

• fix the problem exposed by the previously added test case

  problame

  committed
  7f3ee0d

  Copy full SHA for 7f3ee0d

• add test case that demonstrates delete_timeline is not cancel safe

  Show description for 0fcc803
  problame

  committed
  0fcc803

  Copy full SHA for 0fcc803

• timeline_delete => persist_index_part_with_deleted_flag: make cancel safe

  Show description for d5280bf
  problame

  committed
  d5280bf

  Copy full SHA for d5280bf

• distinguished error types

  problame

  committed
  e41b2ed

  Copy full SHA for e41b2ed

• fixup typo in 'fix the problem exposed by the previously added test case'

  problame

  committed
  48112bd

  Copy full SHA for 48112bd

• push metadata.to_bytes() out of stop() into persist_index_part_with_deleted_flag()

  Show description for 11c18b0
  problame

  committed
  11c18b0

  Copy full SHA for 11c18b0

• Merge remote-tracking branch 'origin/main' into dkr/deleted-flag-in-remote-index

  problame

  committed
  3720e90

  Copy full SHA for 3720e90

• add TODO comment regarding de-configuration of remote storage

  problame

  committed
  db9d781

  Copy full SHA for db9d781

Commits on May 3, 2023

• clarify / improve comments in Tenant::delete_timeline

  problame

  committed
  16486ed

  Copy full SHA for 16486ed

• fixup comment from 'add TODO comment regarding de-configuration of remote storage'

  problame

  committed
  b58bf56

  Copy full SHA for b58bf56

• Merge remote-tracking branch 'origin/main' into dkr/deleted-flag-in-remote-index

  problame

  committed
  de3f233

  Copy full SHA for de3f233

• at some point we committed changes submodule refs in vendor/ , revert that

  problame

  committed
  2785b6a

  Copy full SHA for 2785b6a

• clippy: tracing::Span is only used in cfg(feature = "testing")

  problame

  committed
  4903953

  Copy full SHA for 4903953

• test_delete_timeline_client_hangup: account for logging of dropped requests

  Show description for 30ea7b1
  problame

  committed
  30ea7b1

  Copy full SHA for 30ea7b1

• improve XXX above remove_dir_all regarding atomicity

  Show description for 08558b8
  problame

  committed
  08558b8

  Copy full SHA for 08558b8

• revert some intermediate changes to failpoint_sleep_millis_async

  problame

  committed
  dd73c1d

  Copy full SHA for dd73c1d

• improve docstring on persist_index_part_with_deleted_flag

  problame

  committed
  b8eba89

  Copy full SHA for b8eba89

Commits on May 4, 2023

• actually give it 5 seconds, not 2.5 (and drive-by fix the error message)

  Show description for c025617
  problame

  committed
  c025617

  Copy full SHA for c025617

• clarifications around the QueueUninitialized error

  problame

  committed
  8425b6a

  Copy full SHA for 8425b6a

• wip: revert: 11c18b0

  Show description for fd6d81b
  koivunej

  committed
  fd6d81b

  Copy full SHA for fd6d81b

• complete the revert with the datastructure change

  koivunej

  committed
  ad70535

  Copy full SHA for ad70535

• wip: try with request coalescing

  koivunej

  committed
  5a924f1

  Copy full SHA for 5a924f1

• wip: try: request coalescing (now building)

  koivunej

  committed
  e00da98

  Copy full SHA for e00da98

• refactor: InnerDeleteTimelineError does not need to be Error

  koivunej

  committed
  a543e70

  Copy full SHA for a543e70

• doc: note unhandled case

  koivunej

  committed
  1c20cea

  Copy full SHA for 1c20cea

• chore: forgotten cfg feature testing

  koivunej

  committed
  fffb23e

  Copy full SHA for fffb23e

• request coalescing with MaybeDone

  koivunej

  committed
  88d2551

  Copy full SHA for 88d2551

• restore anyhow::anyhow!(s) error transformations

  koivunej

  committed
  a77bde5

  Copy full SHA for a77bde5

• chore: yet another cfg feature testing

  koivunej

  committed
  022ff36

  Copy full SHA for 022ff36

• fix: resubscribe while holding the lock

  koivunej

  committed
  fa947f0

  Copy full SHA for fa947f0

• doc: fix typo and assert message

  koivunej

  committed
  505febf

  Copy full SHA for 505febf

• doc: comment pass

  koivunej

  committed
  4ef9f38

  Copy full SHA for 4ef9f38

Commits on May 5, 2023

• chore: last clippy

  koivunej

  committed
  04e83ed

  Copy full SHA for 04e83ed

• test: fix test_concurrent_timeline_delete_if_first_stuck_at_index_upload

  koivunej

  committed
  245a2a0

  Copy full SHA for 245a2a0

• test: less racy test_delete_timeline_client_hangup

  Show description for 3fe371e
  koivunej

  committed
  3fe371e

  Copy full SHA for 3fe371e

• chore: python refmt

  koivunej

  committed
  e6c285a

  Copy full SHA for e6c285a

• chore: use older union syntax

  koivunej

  committed
  81d3eb6

  Copy full SHA for 81d3eb6

• test: check that new req is attempted after failure

  koivunej

  committed
  5409851

  Copy full SHA for 5409851

• doc: replace stray comment

  koivunej

  authored
  4f7c3ff

  Copy full SHA for 4f7c3ff

• chore: restore using error for TimelineMetadata serialization

  koivunej

  committed
  8bd900d

  Copy full SHA for 8bd900d

• chore: remove raised exception to investigate log

  koivunej

  committed
  ae48bdd

  Copy full SHA for ae48bdd

• undo take_mut usage introduced in 11c18b0 (#4156)

  Show description for 7aba9eb
  koivunej

  authored
  7aba9eb

  Copy full SHA for 7aba9eb

• comment regarding spawn_blocking + failpoint block

  problame

  committed
  02bf4f0

  Copy full SHA for 02bf4f0

• test_timeline_resurrection_on_attach: only start pg in and during fill_branch=True

  problame

  committed
  512a799

  Copy full SHA for 512a799

• Merge remote-tracking branch 'origin/dkr/deleted-flag-in-remote-index' into joonas/dkr/deleted-flag-in-remote-index-cont

  Show description for 2e63755
  problame

  committed
  2e63755

  Copy full SHA for 2e63755

• add back the timeline_id instrument field for load_local_timeline

  Show description for 67eafaa
  problame

  committed
  67eafaa

  Copy full SHA for 67eafaa

• make timeline deletion retryable if it fails uploading index_part or deleting the files (#4147)

  Show description for 1312cbc
  problame

  authored
  1312cbc

  Copy full SHA for 1312cbc

• Merge remote-tracking branch 'origin/dkr/deleted-flag-in-remote-index' into joonas/dkr/deleted-flag-in-remote-index-cont

  Show description for f86bca3
  problame

  committed
  f86bca3

  Copy full SHA for f86bca3

• re-fix retryability of timeline_delete

  Show description for 6b28e1c
  problame

  committed
  6b28e1c

  Copy full SHA for 6b28e1c