Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvserver: TestReplicateQueueDownReplicate failed #55421

Closed
cockroach-teamcity opened this issue Oct 11, 2020 · 6 comments · Fixed by #56775
Closed

kv/kvserver: TestReplicateQueueDownReplicate failed #55421

cockroach-teamcity opened this issue Oct 11, 2020 · 6 comments · Fixed by #56775
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
20.2

Comments

@cockroach-teamcity
Copy link
Member

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@e824da5a2b33168fa4ef93c83295c8205acfadb7:

=== RUN   TestReplicateQueueDownReplicate
--- FAIL: TestReplicateQueueDownReplicate (0.04s)
    test_log_scope.go:107: can't use TestLogScope with secondary loggers active


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Oct 11, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Oct 11, 2020
@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@b4bf42c4fe10155717ac48f2e68ad8b501ca0b1a:

=== RUN   TestReplicateQueueDownReplicate
    test_log_scope.go:155: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueDownReplicate544040055
    test_log_scope.go:64: use -show-logs to present logs inline
*
* WARNING: disk slowness detected: unable to sync log files within 10s
*


ERROR: exit status 1

1 runs completed, 1 failures, over 19m0s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

andreimatei added a commit to andreimatei/cockroach that referenced this issue Nov 16, 2020
@andreimatei
teamcity-trigger: run nightly stressrace with -p 1
5821969 
We're currently running with -p 2. We're seeing the TeamCity machines
struggle with this - we're getting mysterious failures with logs spammed
with slow requests and stalled disks.

Fixes cockroachdb#56693
Fixes cockroachdb#56484
Fixes cockroachdb#56355

Tests found by searching for "disk slowness detected" and applying a
modicum of common sense.

Fixes cockroachdb#55404
Fixes cockroachdb#55418
Fixes cockroachdb#55420
Fixes cockroachdb#55421
Fixes cockroachdb#55423
Fixes cockroachdb#55574
Fixes cockroachdb#55627
Fixes cockroachdb#55799
Fixes cockroachdb#55884
Fixes cockroachdb#55930
Fixes cockroachdb#56037
Fixes cockroachdb#56106
Fixes cockroachdb#56172
Fixes cockroachdb#56274
Fixes cockroachdb#56402
Fixes cockroachdb#56702
Fixes cockroachdb#56704

Release note: None
@andreimatei andreimatei mentioned this issue Nov 16, 2020
Merged
@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@e2e2cecbaec8d93a70f62951b05f5c2f30cc36de:

=== RUN   TestReplicateQueueDownReplicate
    test_log_scope.go:155: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueDownReplicate249922470
    test_log_scope.go:64: use -show-logs to present logs inline
*
* FATAL: disk stall detected: unable to sync log files within 1m0s
*


ERROR: exit status 8

1 runs completed, 1 failures, over 41m27s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@06944cb1996ed9ccfa036118b810e46673fe0ff4:

=== RUN   TestReplicateQueueDownReplicate
    test_log_scope.go:155: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueDownReplicate720403334
    test_log_scope.go:64: use -show-logs to present logs inline
*
* FATAL: disk stall detected: unable to sync log files within 1m0s
*


ERROR: exit status 8

1 runs completed, 1 failures, over 18m33s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@8ff64f5d897a9c7b39ca437193a11a7300cecbc1:

=== RUN   TestReplicateQueueDownReplicate
    test_log_scope.go:155: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueDownReplicate646740810
    test_log_scope.go:64: use -show-logs to present logs inline


ERROR: exit status 8

1 runs completed, 1 failures, over 24m26s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@609cad52785f49338f7ceff289486b87db4cecb6:

=== RUN   TestReplicateQueueDownReplicate
    test_log_scope.go:150: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueDownReplicate843955505
    test_log_scope.go:61: use -show-logs to present logs inline


ERROR: exit status 8

1 runs completed, 1 failures, over 22m10s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(kv/kvserver).TestReplicateQueueDownReplicate failed on master@f5f0f11d6e31a6667fe1cafa46cd1a472df38f76:

=== RUN   TestReplicateQueueDownReplicate
    test_log_scope.go:150: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestReplicateQueueDownReplicate070298087
    test_log_scope.go:61: use -show-logs to present logs inline


ERROR: exit status 8

1 runs completed, 1 failures, over 18m6s
context canceled


Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters:

  • TAGS=
  • GOFLAGS=-race -parallel=2
make stressrace TESTS=TestReplicateQueueDownReplicate PKG=./pkg/kv/kvserver TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

craig bot pushed a commit that referenced this issue Nov 24, 2020
@andreimatei
Merge #56775
48f860b 
56775: teamcity-trigger: run nightly stressrace with -p 1 r=andreimatei a=andreimatei

We're currently running with -p 2. We're seeing the TeamCity machines
struggle with this - we're getting mysterious failures with logs spammed
with slow requests and stalled disks.

Fixes #56693
Fixes #56484
Fixes #56355

Tests found by searching for "disk slowness detected" and applying a
modicum of common sense.

Fixes #55404
Fixes #55418
Fixes #55420
Fixes #55421
Fixes #55423
Fixes #55574
Fixes #55627
Fixes #55799
Fixes #55884
Fixes #55930
Fixes #56037
Fixes #56106
Fixes #56172
Fixes #56274
Fixes #56402
Fixes #56702
Fixes #56704

Release note: None

Co-authored-by: Andrei Matei <[email protected]>
@craig craig bot closed this as completed in 40ab4eb Nov 24, 2020
@craig craig bot closed this as completed in #56775 Nov 24, 2020
TheSamHuang pushed a commit to TheSamHuang/cockroach that referenced this issue Nov 25, 2020
@andreimatei @TheSamHuang
teamcity-trigger: run nightly stressrace with -p 1
28554a3 
We're currently running with -p 2. We're seeing the TeamCity machines
struggle with this - we're getting mysterious failures with logs spammed
with slow requests and stalled disks.

Fixes cockroachdb#56693
Fixes cockroachdb#56484
Fixes cockroachdb#56355

Tests found by searching for "disk slowness detected" and applying a
modicum of common sense.

Fixes cockroachdb#55404
Fixes cockroachdb#55418
Fixes cockroachdb#55420
Fixes cockroachdb#55421
Fixes cockroachdb#55423
Fixes cockroachdb#55574
Fixes cockroachdb#55627
Fixes cockroachdb#55799
Fixes cockroachdb#55884
Fixes cockroachdb#55930
Fixes cockroachdb#56037
Fixes cockroachdb#56106
Fixes cockroachdb#56172
Fixes cockroachdb#56274
Fixes cockroachdb#56402
Fixes cockroachdb#56702
Fixes cockroachdb#56704
Fixes cockroachdb#56717
Fixes cockroachdb#56794
Fixes cockroachdb#56889
Fixes cockroachdb#56891
Fixes cockroachdb#56946
Fixes cockroachdb#56990

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
20.2
Development

Successfully merging a pull request may close this issue.

teamcity-trigger: run nightly stressrace with -p 1 andreimatei/cockroach
1 participant
@cockroach-teamcity