roachtest: import/tpch/nodes=8 failed #68037

cockroach-teamcity · 2021-07-25T14:59:13Z

roachtest.import/tpch/nodes=8 failed with artifacts on master @ 9baaa282b3a09977b96bd3e5ae6e2346adfa2c16:

		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2087
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:225
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) 4: dead (exit status 134)
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (4) secondary error attachment
		  | 6: dead (exit status 134)
		  | (1) attached stack trace
		  |   -- stack trace:
		  |   | main.glob..func14
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1168
		  |   | main.wrap.func1
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:276
		  |   | github.com/spf13/cobra.(*Command).execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  |   | github.com/spf13/cobra.(*Command).ExecuteC
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  |   | github.com/spf13/cobra.(*Command).Execute
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  |   | main.main
		  |   | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2087
		  |   | runtime.main
		  |   | 	/usr/local/go/src/runtime/proc.go:225
		  |   | runtime.goexit
		  |   | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		  | Wraps: (2) 6: dead (exit status 134)
		  | Error types: (1) *withstack.withStack (2) *errutil.leafError
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1168
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:276
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2087
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) 5: dead (exit status 134)
		Error types: (1) errors.Unclassified (2) *secondary.withSecondaryError (3) *secondary.withSecondaryError (4) *secondary.withSecondaryError (5) *withstack.withStack (6) *errutil.leafError

Reproduce

To reproduce, try:

## Simple repro (linux-only):
  $ make cockroachshort bin/worklaod bin/roachprod bin/roachtest
  $ PATH=$PWD/bin:$PATH roachtest run import/tpch/nodes=8 --local

## Proper repro probably needs more roachtest flags, or running
## the programs remotely on GCE. For more details, refer to
## pkg/cmd/roachtest/README.md.

Same failure on other branches

roachtest: import/tpch/nodes=8 failed #66263 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

The text was updated successfully, but these errors were encountered:

pbardea · 2021-07-27T14:24:58Z

The test timedout again, this isn't new. Perhaps it's worth bumping the timeout of this test to 10h until we get time to look into improving the performance here.

test timed out (8h0m0s)

adityamaru · 2021-07-27T17:20:27Z

Created #68118 and filed #68117.

cockroach-teamcity · 2021-08-07T15:12:34Z

roachtest.import/tpch/nodes=8 failed with artifacts on master @ cab185ff71f0924953d987fe6ffd14efdd32a3a0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/import/tpch/nodes=8/run_1
	monitor.go:128,import.go:278,test_runner.go:777: monitor failure: monitor task failed: pgurl for nodes :4 got urls [problem loading clusters: could not read /root/.roachprod/hosts/teamcity-3273485-1628317117-50-n4cpu8: open /root/.roachprod/hosts/teamcity-3273485-1628317117-50-n4cpu8: no such file or directory
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:278
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*clusterImpl).pgURLErr
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2033
		  | main.(*clusterImpl).ExternalPGUrl
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2063
		  | main.(*clusterImpl).ConnE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2243
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*HealthChecker).Runner
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:113
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) pgurl for nodes :4 got urls [problem loading clusters: could not read /root/.roachprod/hosts/teamcity-3273485-1628317117-50-n4cpu8: open /root/.roachprod/hosts/teamcity-3273485-1628317117-50-n4cpu8: no such file or directory
		  | 'postgres://[email protected]:26257?sslmode=disable'] from stdout:
		  | problem loading clusters: could not read /root/.roachprod/hosts/teamcity-3273485-1628317117-50-n4cpu8: open /root/.roachprod/hosts/teamcity-3273485-1628317117-50-n4cpu8: no such file or directory
		  | 'postgres://[email protected]:26257?sslmode=disable'
		  | stderr:
		  | 
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

See: CI job to stress roachtests

For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^import/tpch/nodes=8$` * Parameters / `env.COUNT`: <number of runs>

Same failure on other branches

roachtest: import/tpch/nodes=8 failed #66263 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot branch-release-20.1 release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-10-29T06:28:20Z

roachtest.import/tpch/nodes=8 failed with artifacts on master @ 40bd241158c8dd4a68d277fa732b91f56e411b38:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/import/tpch/nodes=8/run_1
	monitor.go:128,import.go:308,test_runner.go:777: monitor failure: monitor task failed: import failed: pq: relation "csv.public.lineitem" does not exist
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:308
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1.2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:291
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) import failed
		Wraps: (7) pq: relation "csv.public.lineitem" does not exist
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pq.Error

Help

See: roachtest README

|

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpch/nodes=8 failed #72034 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1 release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-10-30T06:16:06Z

roachtest.import/tpch/nodes=8 failed with artifacts on master @ db229cad354c2776ea35c9dc4f78e6ce66b9437c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/import/tpch/nodes=8/run_1
	monitor.go:128,import.go:308,test_runner.go:777: monitor failure: monitor task failed: import failed: pq: relation "csv.public.lineitem" does not exist
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:308
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1.2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:291
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) import failed
		Wraps: (7) pq: relation "csv.public.lineitem" does not exist
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pq.Error

Help

See: roachtest README

|

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpch/nodes=8 failed #72034 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1 release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

cockroach-teamcity · 2021-10-31T06:20:02Z

roachtest.import/tpch/nodes=8 failed with artifacts on master @ a15b840db34de504bf060b2599276e049cc64799:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/import/tpch/nodes=8/run_1
	monitor.go:128,import.go:308,test_runner.go:777: monitor failure: monitor task failed: import failed: pq: relation "csv.public.lineitem" does not exist
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:308
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerImportTPCH.func1.2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/import.go:291
		  | main.(*monitorImpl).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:106
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) import failed
		Wraps: (7) pq: relation "csv.public.lineitem" does not exist
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pq.Error

Help

See: roachtest README

|

See: How To Investigate (internal)

Same failure on other branches

roachtest: import/tpch/nodes=8 failed #72034 roachtest: import/tpch/nodes=8 failed [C-test-failure O-roachtest O-robot T-bulkio branch-release-21.1 release-blocker]

/cc @cockroachdb/bulk-io _{This test on roachdash | Improve this report!}

72002: pgwire: add a test for server aborting on repeated errors r=otan a=rafiss This adds a test for a recent commit that changed the pgwire loop to give up if it saw an error repeatedly. I added a testing knob to inject an error, and did a small refactor so that errors are handled consistently. Previously, some error handling would only happen for "MessageTooBig" errors. Release note: None 72279: roachtest: fix import roachtest r=stevendanna a=adityamaru This change fixes the import roachtest to switch to the `csv` database before creating the table and importing into it. This broke when we switched the roachtest to use IMPORT INTO from IMPORT CREATE USING in #72097. Fixes: #68037 Release note: None Co-authored-by: Rafi Shamim <[email protected]> Co-authored-by: Aditya Maru <[email protected]>

blathers-crl bot added the T-disaster-recovery label Jul 25, 2021

livlobo assigned adityamaru Jul 26, 2021

pbardea removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jul 27, 2021

cockroach-teamcity mentioned this issue Aug 5, 2021

roachtest: import/tpch/nodes=8 failed #68471

Closed

This was referenced Aug 12, 2021

roachtest: import/tpch/nodes=8 failed #68786

Closed

roachtest: import/tpch/nodes=8 failed #69033

Closed

cockroach-teamcity mentioned this issue Aug 26, 2021

roachtest: import/tpch/nodes=8 failed #69417

Closed

cockroach-teamcity mentioned this issue Sep 14, 2021

roachtest: import/tpch/nodes=8 failed #70199

Closed

This was referenced Oct 6, 2021

roachtest: import/tpch/nodes=8 failed #71203

Closed

roachtest: import/tpch/nodes=8 failed #71208

Closed

cockroach-teamcity mentioned this issue Oct 27, 2021

roachtest: import/tpch/nodes=8 failed #72034

Closed

adityamaru mentioned this issue Nov 1, 2021

roachtest: fix import roachtest #72279

Merged

craig bot closed this as completed in 1e7cd0b Nov 1, 2021

github-project-automation bot added this to Disaster Recovery Backlog Aug 28, 2024

github-project-automation bot moved this to Done in Disaster Recovery Backlog Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roachtest: import/tpch/nodes=8 failed #68037

roachtest: import/tpch/nodes=8 failed #68037

cockroach-teamcity commented Jul 25, 2021

pbardea commented Jul 27, 2021 •

edited

Loading

adityamaru commented Jul 27, 2021

cockroach-teamcity commented Aug 7, 2021

cockroach-teamcity commented Oct 29, 2021

cockroach-teamcity commented Oct 30, 2021

cockroach-teamcity commented Oct 31, 2021

roachtest: import/tpch/nodes=8 failed #68037

roachtest: import/tpch/nodes=8 failed #68037

Comments

cockroach-teamcity commented Jul 25, 2021

pbardea commented Jul 27, 2021 • edited Loading

adityamaru commented Jul 27, 2021

cockroach-teamcity commented Aug 7, 2021

cockroach-teamcity commented Oct 29, 2021

cockroach-teamcity commented Oct 30, 2021

cockroach-teamcity commented Oct 31, 2021

pbardea commented Jul 27, 2021 •

edited

Loading