Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to OMB validation tests for T1 and T2 #1

Merged

Conversation

travisdowns
Copy link

Changes to omb_validation tests based on T1 and T2 runs.

Set the throughput per producer and consumer to 5 from 4, and remove
the requirement to have at least 8 of each (require a minimum of 1),
which brings us into line with prior cloud testing for T1 which used
4 producers and 4 consumers per subscription.

Producer and consumer count will continue to scale in the same way
for higher tiers.
This is the timeout from when the benchmark has been started to
when the warmup traffic starts. Since this also involves deleting the
topics from the previous run (because reset: true), this may take a
long time as topic creation and user group formation can be greatly
slowed down by the work associated with the deletion.

This causes tests which run after max_partitions_test to often fail to
start. Later we may get a more sophisticated cleanup strategy, but for
now just increase the start timeout to 5 minutes.
To detect test start we look for a "starting warm-up traffic" message
which only appears if the test has > 0 warmup time, otherwise we will
just hang for 5 minutes and then time out the startup (even though
the test is likely still running).

Instead, assert that warmup time is at least 1.
Currently the OMB verification tests often fail because work
from one test bleeds into the next (especially, the max partition count
one which takes a while to clean up).

The impact from topic deletions is generally felt right at the start of
the next test when OMB tries to delete all topics, and this can cause
latency threshold failures when this work extends beyond the warmup
time into the measured portion of the test.

This change moves to a 5/5 minute test duration (warmup/runtime) from
1/5 to mitigate this.
@ballard26 ballard26 merged commit 3691aa6 into ballard26:perf-val-dt-tests Nov 14, 2023
1 check passed
ballard26 pushed a commit that referenced this pull request Oct 2, 2024
=================================================================
==524==ERROR: AddressSanitizer: container-overflow on address 0x52100028a100 at pc 0x7f29c0bf51fd bp 0x7ffda75397d0 sp 0x7ffda75397c8
READ of size 4 at 0x52100028a100 thread T0
    #0 0x7f29c0bf51fc in util::mem_tracker::pretty_print_ascii() const /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-024f8676eb7b139e9-1/redpanda/vtools/src/v/utils/tracking_allocator.cc:75:31
    #1 0x55be1ac7eb7b in mem_tracker_pretty_printing::test_method() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-024f8676eb7b139e9-1/redpanda/vtools/src/v/utils/tests/tracking_allocator_tests.cc:82:5
    redpanda-data#2 0x55be1ac7d651 in mem_tracker_pretty_printing_invoker() /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-024f8676eb7b139e9-1/redpanda/vtools/src/v/utils/tests/tracking_allocator_tests.cc:50:1

Signed-off-by: Noah Watkins <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants