Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

streamingest: TestTenantStreamingMultipleNodes fails/flaky under remote execution #109185

Closed
rickystewart opened this issue Aug 21, 2023 · 2 comments · Fixed by #112827
Closed

streamingest: TestTenantStreamingMultipleNodes fails/flaky under remote execution #109185

rickystewart opened this issue Aug 21, 2023 · 2 comments · Fixed by #112827
Assignees
@msbutler
Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). O-remote-execution An issue found only using Bazel Remote Execution function. T-disaster-recovery

Comments

@rickystewart
Copy link
Collaborator

rickystewart commented Aug 21, 2023 •
edited
Loading

EngFlow link: https://tanzanite.cluster.engflow.com/invocation/c2ee5641-cdda-4fc2-8184-1c4b58533f62
Reproduction command: bazel test //pkg/ccl/streamingccl/streamingest:streamingest_test --config engflow --config crosslinux --jobs 400 --config cibase --test_filter=TestTenantStreamingMultipleNodes --test_arg -show-logs --test_sharding_strategy=disabled --nocache_test_results --runs_per_test=10

The test is flaky. I see more successes than failures but failures are still fairly common (3/10 failures in the run I did).

https://cockroachlabs.atlassian.net/wiki/spaces/devinf/pages/3141107902/How+to+reproduce+a+test+failure+on+EngFlow

Epic: CRDB-8308

Jira issue: CRDB-30815
@rickystewart rickystewart added C-test-failure Broken test (automatically or manually discovered). A-disaster-recovery T-disaster-recovery labels Aug 21, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 21, 2023

cc @cockroachdb/disaster-recovery

@dt dt removed the C-test-failure Broken test (automatically or manually discovered). label Aug 22, 2023
@lunevalex lunevalex added C-test-failure Broken test (automatically or manually discovered). O-remote-execution An issue found only using Bazel Remote Execution function. labels Aug 22, 2023
@lidorcarmel lidorcarmel self-assigned this Aug 22, 2023
@adityamaru
Copy link
Contributor

@lidorcarmel do you plan to address this or should we re-triage?

@msbutler msbutler self-assigned this Oct 22, 2023
@msbutler msbutler mentioned this issue Oct 22, 2023
Merged
msbutler added a commit to msbutler/cockroach that referenced this issue Oct 22, 2023
@msbutler
c2c: deflake a few e2e tests
2e5fe5c 
This patch changes the TestTenantStreamingMultipleNodes and
TestTenantStreamingUnavailable tests to run on a single single host cluster,
instead of two, which should reduce cpu contention and flakes.

Informs cockroachdb#112748
Fixes cockroachdb#112783
Fixes cockroachdb#109185

Release note: none
craig bot pushed a commit that referenced this issue Oct 23, 2023
@msbutler
Merge #112827
f94626d 
112827: c2c: deflake a few e2e tests r=stevendanna a=msbutler

This patch changes the TestTenantStreamingMultipleNodes and
TestTenantStreamingUnavailable tests to run on a single single host cluster,
instead of two, which should reduce cpu contention and flakes.

Informs #112748
Fixes #112783
Fixes #109185

Release note: none

Co-authored-by: Michael Butler <[email protected]>
@craig craig bot closed this as completed in 4e8d00c Oct 23, 2023
msbutler added a commit to msbutler/cockroach that referenced this issue Oct 24, 2023
@msbutler
release-23.2: c2c: deflake a few e2e tests
81d080c 
This patch changes the TestTenantStreamingMultipleNodes and
TestTenantStreamingUnavailable tests to run on a single single host cluster,
instead of two, which should reduce cpu contention and flakes.

Informs cockroachdb#112748
Fixes cockroachdb#112783
Fixes cockroachdb#109185

Release note: none
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@msbutler msbutler

Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). O-remote-execution An issue found only using Bazel Remote Execution function. T-disaster-recovery
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

c2c: deflake a few e2e tests msbutler/cockroach
6 participants
@dt @rickystewart @adityamaru @msbutler @lidorcarmel @lunevalex