Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: TestAsyncSchemaChanger failed #55930

Closed
cockroach-teamcity opened this issue Oct 24, 2020 · 1 comment · Fixed by #56775
Closed

sql: TestAsyncSchemaChanger failed #55930

cockroach-teamcity opened this issue Oct 24, 2020 · 1 comment · Fixed by #56775
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(sql).TestAsyncSchemaChanger failed on master@75234294d22cc9a6e12724b043f49e78f0ac92d2:

=== RUN   TestAsyncSchemaChanger
*
* WARNING: disk slowness detected: unable to sync log files within 10s
*


ERROR: exit status 255

1 runs completed, 1 failures, over 24m20s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestAsyncSchemaChanger PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Oct 24, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 24, 2020
@nvanbenschoten
Copy link
Member

F201024 06:21:06.742872 375366 sqlmigrations/migrations.go:667  [n1] not enough time left on migration lease, terminating for safety
goroutine 375366 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xbb6da01, 0x0, 0x0, 0x6350473)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xc6
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xb1b15c0, 0x4, 0x1640d95e624708ce, 0x5ba46, 0xabe801b, 0x1b, 0x29b, 0xc0b1d0e340, 0x3f, 0xc0368b2de0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:243 +0xb31
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x8341e40, 0xc033e984e0, 0x4, 0x2, 0x643fb46, 0x3f, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:52 +0x19f
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x8341e40, 0xc033e984e0, 0x1, 0xc000000004, 0x643fb46, 0x3f, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:173
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations.func2(0x8341e40, 0xc033e984e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:667 +0x347
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc030fccdc0, 0x8341e40, 0xc033e984e0, 0xc0b14afa40, 0x29, 0x0, 0x0, 0xc02b130e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:347 +0x163
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:342 +0x14b

andreimatei added a commit to andreimatei/cockroach that referenced this issue Nov 16, 2020
We're currently running with -p 2. We're seeing the TeamCity machines
struggle with this - we're getting mysterious failures with logs spammed
with slow requests and stalled disks.

Fixes cockroachdb#56693
Fixes cockroachdb#56484
Fixes cockroachdb#56355

Tests found by searching for "disk slowness detected" and applying a
modicum of common sense.

Fixes cockroachdb#55404
Fixes cockroachdb#55418
Fixes cockroachdb#55420
Fixes cockroachdb#55421
Fixes cockroachdb#55423
Fixes cockroachdb#55574
Fixes cockroachdb#55627
Fixes cockroachdb#55799
Fixes cockroachdb#55884
Fixes cockroachdb#55930
Fixes cockroachdb#56037
Fixes cockroachdb#56106
Fixes cockroachdb#56172
Fixes cockroachdb#56274
Fixes cockroachdb#56402
Fixes cockroachdb#56702
Fixes cockroachdb#56704

Release note: None
craig bot pushed a commit that referenced this issue Nov 24, 2020
56775: teamcity-trigger: run nightly stressrace with -p 1 r=andreimatei a=andreimatei

We're currently running with -p 2. We're seeing the TeamCity machines
struggle with this - we're getting mysterious failures with logs spammed
with slow requests and stalled disks.

Fixes #56693
Fixes #56484
Fixes #56355

Tests found by searching for "disk slowness detected" and applying a
modicum of common sense.

Fixes #55404
Fixes #55418
Fixes #55420
Fixes #55421
Fixes #55423
Fixes #55574
Fixes #55627
Fixes #55799
Fixes #55884
Fixes #55930
Fixes #56037
Fixes #56106
Fixes #56172
Fixes #56274
Fixes #56402
Fixes #56702
Fixes #56704

Release note: None

Co-authored-by: Andrei Matei <[email protected]>
@craig craig bot closed this as completed in #56775 Nov 24, 2020
@craig craig bot closed this as completed in 40ab4eb Nov 24, 2020
TheSamHuang pushed a commit to TheSamHuang/cockroach that referenced this issue Nov 25, 2020
We're currently running with -p 2. We're seeing the TeamCity machines
struggle with this - we're getting mysterious failures with logs spammed
with slow requests and stalled disks.

Fixes cockroachdb#56693
Fixes cockroachdb#56484
Fixes cockroachdb#56355

Tests found by searching for "disk slowness detected" and applying a
modicum of common sense.

Fixes cockroachdb#55404
Fixes cockroachdb#55418
Fixes cockroachdb#55420
Fixes cockroachdb#55421
Fixes cockroachdb#55423
Fixes cockroachdb#55574
Fixes cockroachdb#55627
Fixes cockroachdb#55799
Fixes cockroachdb#55884
Fixes cockroachdb#55930
Fixes cockroachdb#56037
Fixes cockroachdb#56106
Fixes cockroachdb#56172
Fixes cockroachdb#56274
Fixes cockroachdb#56402
Fixes cockroachdb#56702
Fixes cockroachdb#56704
Fixes cockroachdb#56717
Fixes cockroachdb#56794
Fixes cockroachdb#56889
Fixes cockroachdb#56891
Fixes cockroachdb#56946
Fixes cockroachdb#56990

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants