Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

acceptance: TestDocker failing under race #87700

Closed
knz opened this issue Sep 9, 2022 · 1 comment · Fixed by #87713
Closed

acceptance: TestDocker failing under race #87700

knz opened this issue Sep 9, 2022 · 1 comment · Fixed by #87713
Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@knz
Copy link
Contributor

knz commented Sep 9, 2022 •
edited by cockroach-jira-scripts
Loading

Describe the problem

In an unrelated PR I reliably see the "testrace" CI target failing in TestDockerC (the first of the docker tests: the specific test doesn't matter, the issue repros with all of them) with the stack trace below.

This corresponds to the following go code failing:

  if *cockroachImage == defaultImage && !exists(*CockroachBinary) {
    log.Fatalf(ctx, "\"%s\": does not exist", *CockroachBinary)
  }

The fatal message suggests that the CockroachBinary string is empty.

Example here https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_Tests_Testrace/6409520?showRootCauses=false&expandBuildChangesSection=true&expandBuildTestsSection=true

[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1  "": does not exist
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !goroutine 43 [running]:
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x0)
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x9c
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc000c3b0e0, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}}, 0x171335b442fe7329, ...})
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:260 +0xb7
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x985c508, 0xc000080098}, 0x2, 0x4, 0x0, 0x0, {0x68641ad, 0x14}, {0xc001bda800, 0x1, ...})
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x6e6
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild-ST-1665e0aa65f7/bin/pkg/util/log/log_channels_generated.go:848
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/acceptance/cluster.CreateDocker({0x985c508, 0xc000080098}, {{0x682f22f, 0x1}, {0xc000c3b410, 0x1, 0x1}, 0x12a05f200, 0x0, 0x0}, ...)
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/acceptance/cluster/pkg/acceptance/cluster/dockercluster.go:155 +0x1d6
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/acceptance.StartCluster({0x985c508, 0xc000080098}, 0xc000701a00, {{0x682f22f, 0x1}, {0xc000c3b410, 0x1, 0x1}, 0x12a05f200, 0x0, ...})
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/acceptance/util_cluster.go:79 +0x3fd
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !github.com/cockroachdb/cockroach/pkg/acceptance.testDocker.func1(0xc000701a00)
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	github.com/cockroachdb/cockroach/pkg/acceptance/util_docker.go:124 +0xae7
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !testing.tRunner(0xc000701a00, 0xc000c30440)
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	GOROOT/src/testing/testing.go:1446 +0x217
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !created by testing.(*T).Run
[15:55:35] :		 [Run race detector tests] F220909 13:55:35.764202 43 acceptance/cluster/dockercluster.go:155  [-] 1 !	GOROOT/src/testing/testing.go:1493 +0x75e

Jira issue: CRDB-19489
@knz knz added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. A-testing Testing tools and infrastructure labels Sep 9, 2022
@knz knz mentioned this issue Sep 9, 2022
Merged
@rickystewart
Copy link
Collaborator

Looks like we should skip testing acceptance in testrace, which makes sense (it's for unit tests, acceptance is not a unit test).

@rickystewart rickystewart mentioned this issue Sep 9, 2022
Merged
rickystewart added a commit to rickystewart/cockroach that referenced this issue Sep 9, 2022
@rickystewart
ci: skip "integration"-style tests in testrace
e572dce 
Closes cockroachdb#87700.
Release note: None
craig bot pushed a commit that referenced this issue Sep 9, 2022
@ajwerner @adityamaru
@tbg @rickystewart
Merge #87614 #87646 #87667 #87702 #87713
be989ac 
87614: sql/opt/norm: propagate kv errors from cast folding r=ajwerner a=ajwerner

Swallowing KV errors here leads to incorrect results. Writes can be missed and serializability can be silently violated. This comes up in the context of the randomized schema change testing.

May deal with #85677
relates to #80764

Release note: None

87646: backupccl: deflake backup tests r=adityamaru a=adityamaru

See individual commits.

Release note: None

87667: server: clean up some logging r=ajwerner a=ajwerner

For one, this fixes a format directive issue in the first line. It also stops rendering some arguments to strings directly so that redaction works correctly.

Release note: None

87702: kvserver: explain our use of (*raftGroup).ReportSnapshot r=andrewbaptist a=tbg

Closes #87581.

Release note: None


87713: ci: skip "integration"-style tests in testrace r=knz a=rickystewart

Closes #87700.
Release note: None

Co-authored-by: Andrew Werner <[email protected]>
Co-authored-by: adityamaru <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
@craig craig bot closed this as completed in #87713 Sep 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ci: skip "integration"-style tests in testrace rickystewart/cockroach
2 participants
@knz @rickystewart