Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #108695

Closed
cockroach-teamcity opened this issue Aug 14, 2023 · 4 comments
Closed

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed #108695

cockroach-teamcity opened this issue Aug 14, 2023 · 4 comments
Assignees
@annrpom
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 14, 2023 •
edited by cockroach-jira-scripts
Loading

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed with artifacts on release-23.1.9-rc @ d417968cd3eb53414a97b29692ba61cf1466276d:

1, 29, role_id_seq, 48
1, 29, role_members, 23
1, 29, role_options, 33
1, 29, scheduled_jobs, 37
1, 29, settings, 6
1, 29, span_configurations, 47
1, 29, span_stats_buckets, 55
1, 29, span_stats_samples, 56
1, 29, span_stats_tenant_boundaries, 57
1, 29, span_stats_unique_keys, 54
1, 29, sql_instances, 46
1, 29, sqlliveness, 39
1, 29, statement_activity, 60
1, 29, statement_bundle_chunks, 34
1, 29, statement_diagnostics, 36
1, 29, statement_diagnostics_requests, 35
1, 29, statement_statistics, 42
1, 29, table_statistics, 20
1, 29, task_payloads, 58
1, 29, tenant_id_seq, 62
1, 29, tenant_settings, 50
1, 29, tenant_tasks, 59
1, 29, tenant_usage, 45
1, 29, tenants, 8
1, 29, transaction_activity, 61
1, 29, transaction_statistics, 43
1, 29, ui, 14
1, 29, users, 4
1, 29, web_sessions, 19
1, 29, zones, 5
100, 0, public, 101
102, 0, public, 103
104, 0, public, 105
104, 0, schema406, 202
104, 105, _enum144, 140
104, 105, _enum188, 148
104, 105, _enum24, 114
104, 105, _enum251, 167
104, 105, _enum391, 201
104, 105, _enum59, 120
104, 105, enum144, 139
104, 105, enum188, 147
104, 105, enum24, 113
104, 105, enum251, 166
104, 105, enum391, 200
104, 105, enum59, 119
    schema_change_external_test.go:80: backup in /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2000366506
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/fcba020ca7cb44bf8385bd0e13bcbe2d/logTestWorkload2000366506
--- FAIL: TestWorkload (115.00s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-30597
@cockroach-teamcity cockroach-teamcity added branch-release-23.1.9-rc C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Aug 14, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Aug 14, 2023
@annrpom annrpom self-assigned this Aug 14, 2023
@annrpom
Copy link
Contributor

annrpom commented Aug 15, 2023 •
edited
Loading

trying to repro on my gce-worker with

./dev test --stress pkg/ccl/testccl/workload/schemachange/ --filter=TestWorkload -- --test_env=COCKROACH_RANDOM_SEED=2535861318732466597 --define gotags=bazel,gss,deadlock

@annrpom
Copy link
Contributor

annrpom commented Aug 15, 2023 •
edited
Loading

stressing here

@annrpom annrpom mentioned this issue Aug 15, 2023
Closed
@annrpom
Copy link
Contributor

annrpom commented Aug 15, 2023

stressing properly here

@annrpom
Copy link
Contributor

annrpom commented Aug 17, 2023

Closing due to updates from #105517.

@annrpom annrpom closed this as completed Aug 17, 2023
@annrpom annrpom mentioned this issue Aug 18, 2023
Merged
craig bot pushed a commit that referenced this issue Aug 22, 2023
@annrpom @aliher1911 @knz
Merge #108922 #109223 #109232 #109238
a03be32 
108922: sql: use seeded random for schemachange opsgen r=annrpom a=annrpom

A schemachange TestWorkload failure was difficult to reproduce; to address this, this patch uses randutil's `NewTestRand()` to allow for a global seed to be set when stressing this test.

Epic: none
Informs: #108695
Informs: #105517
Informs: #109218
Release note: none

109223: loqrecovery: add logging of for network errors r=pavelkalinnikov a=aliher1911

Previously unrecoverable errors that caused nodes to be skipped were treated opaquely and provided no visibility of what actual error is.
This PR adds extra logging for errors at info level. Since number of dead nodes is not expected to be high (otherwise LOQ tools won't give much benefit), extra logging is justified as it adds more visibility to help discover more retryable cases.

Epic: none

Release note: None

Touches: #108429

109232: server: properly connect the scheduler latency sampler to its listener r=irfansharif,stevendanna a=knz

Prior to this patch, if we had N servers running (e.g. in tests) we would have N^2 callbacks called every sample round (one sampler per server, N calls per sampler) because it used a global callback registry.

Also it wasn't particularly elegant design, as we should eschew global registries altogether.

This patch enhances the situation by only having one listener per sampler.

Release note: None

Needed to solve #109225.
Epic: CRDB-26691

109238: cmd: error if try to use --stream-output with --stress r=liamgillies a=liamgillies

Previously running these two flags together would force serialization on `--stress`, which doesn't make any sense. This PR adds an error when the two flags are attempted to be used together.

Fixes: #108150
Release note: None

Co-authored-by: Annie Pompa <[email protected]>
Co-authored-by: Oleg Afanasyev <[email protected]>
Co-authored-by: Raphael 'kena' Poss <[email protected]>
Co-authored-by: Liam Gillies <[email protected]>
This was referenced Aug 22, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@annrpom annrpom

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
2 participants
@cockroach-teamcity @annrpom