Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(pageserver): ensure all layers are flushed before measuring RSS #9861

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(pageserver): ensure all layers are flushed before measuring RSS #9861

wants to merge 1 commit into from
+7 −2

Conversation

skyzh
Copy link
Member

@skyzh skyzh commented Nov 22, 2024

Problem

close #9761

The test assumed that no new L0 layers are flushed throughout the process, which is not true.

Summary of changes

Fix the test case test_compaction_l0_memory by flushing in-memory layers before compaction.

@skyzh skyzh requested review from a team and yliang412 and removed request for a team November 22, 2024 19:16
@skyzh skyzh added the run-benchmarks Indicates to the CI that benchmarks should be run for PR marked with this label label Nov 22, 2024
@github-actions GitHub Actions
Copy link

5644 tests run: 5416 passed, 0 failed, 228 skipped (full report)

Flaky tests (3)

Postgres 16

Postgres 15

Postgres 14

Code coverage* (full report)

  • functions: 31.4% (7963 of 25346 functions)
  • lines: 49.3% (63173 of 128158 lines)

* collected from Rust tests only

The comment gets automatically updated with the latest test results
c17165f at 2024-11-22T21:08:10.735Z :recycle:

@skyzh
Copy link
Member Author

skyzh commented Nov 22, 2024

...finally, we are measuring this in a right way

2024-11-22 20:30:28.433 INFO [test_compaction.py:115] l0s before compaction 21 (6057607168)
2024-11-22 20:30:28.463 INFO [http.py:678] Requesting compact: tenant 95b63686f477e0070470955b9216ed91, timeline caafd52dffdbb92e4464381023666905
2024-11-22 20:30:59.54  INFO [http.py:684] Got compact request response code: 200
2024-11-22 20:30:59.113 INFO [test_compaction.py:129] RSS across compaction: 833417216.0 -> 3129913344.0 (grew 2296496128.0)
2024-11-22 20:30:59.115 INFO [test_compaction.py:133] l0s after compaction 11 (3271204864)

no extra layers flushed, 10 L0 is the maximum L0 per job imposed by default checkpoint threshold

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yliang412 yliang412 yliang412 approved these changes

Assignees
No one assigned
Labels
run-benchmarks Indicates to the CI that benchmarks should be run for PR marked with this label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

test_compaction_l0_memory failure
2 participants
@skyzh @yliang412