fix(pageserver): ensure upload happens after delete

[skyzh]

Problem

Follow up of #9682, that patch didn't fully address the problem: what if shutdown fails due to whatever reason and then we reattach the tenant? Then we will still remove the future layer. The underlying problem is that the fix for #5878 gets voided because of the generation optimizations.

Of course, we also need to ensure that delete happens after uploads, but note that we only schedule deletes when there are no ongoing upload tasks, so that's fine.

Summary of changes

• Add a test case to reproduce the behavior (by changing the original test case to attach the same generation).
• If layer upload happens after the deletion, drain the deletion queue before uploading.
  • If blocked_deletion is enabled, directly remove it from the blocked_deletion queue.
• Local fs backend fix to avoid race between deletion and preload.
• test_emergency_mode does not need to wait for uploads (and it's generally not possible to wait for uploads).
• Optimize deletion executor to skip validation if there are no files to delete. this doesn't work

[github-actions]

5544 tests run: 5327 passed, 0 failed, 217 skipped (full report)

---

Flaky tests (2)

Postgres 17

• test_cli_start_stop: release-arm64

Postgres 15

• test_pull_timeline[True]: release-arm64

Code coverage* (full report)

• functions: 31.4% (7962 of 25349 functions)
• lines: 49.3% (63161 of 128148 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
2a7d002 at 2024-11-22T18:38:22.198Z :recycle:}

[skyzh]

I have no idea why 9e7fa98 will cause a lot of test failures but after reverting everything seems so good.

[jcsp]

I see how this PR addresses the case of deleting+recreating the same image layer within the lifetime of a Timeline, but what about the case where some earlier shutdown went wrong and we start a new Timeline while some deletions are still in the global deletion queue? Don't we still need a change similar to #9796 to make it flush the deletion queue on start?

[skyzh]

I see how this PR addresses the case of deleting+recreating the same image layer within the lifetime of a Timeline, but what about the case where some earlier shutdown went wrong and we start a new Timeline while some deletions are still in the global deletion queue? Don't we still need a change similar to #9796 to make it flush the deletion queue on start?

Yes; in theory, this could happen, but it has not been observed yet, and I assume it is rarer to happen. wWe can fix it in another patch. The problem I would like to address is the race between future layer deletion and the upload of the same layer. Both operations happen after the tenant reattach.

[skyzh]

... and if I'm going to fix it, I will populate the recently_deleted hashset with the initial deletion list.

[jcsp]

problem I would like to address is the race between future layer deletion and the upload of the same layer. Both operations happen after the tenant reattach.

Makes sense