roachtest: tpccbench/nodes=3/cpu=4 failed

[cockroach-teamcity]

(roachtest).tpccbench/nodes=3/cpu=4 failed on release-19.1@3dd3057c8c9fa9476c47765612a35fd7b4e9698c:

		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitor).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2585
		  | main.(*monitor).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2593
		  | main.runTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:782
		  | main.registerTPCCBenchSpec.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:617
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:754
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitor).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2641
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.(*cluster).RunE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2245
		  | main.loadTPCCBench
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:684
		  | main.runTPCCBench.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:780
		  | main.(*monitor).Go.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2575
		  | golang.org/x/sync/errgroup.(*Group).Go.func1
		  | 	/home/agent/work/.go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1357
		Wraps: (6) output in run_054637.819_n4_workload_fixtures_load_tpcc
		Wraps: (7) /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-2219111-1598334129-12-n4cpu4:4 -- ./workload fixtures load tpcc --warehouses=1000 --scatter --checks=false {pgurl:1} returned
		  | stderr:
		  |  73 ccl/workloadccl/fixture.go:586  loaded 17 GiB table customer in 23m32.041061143s (30000000 rows, 30000000 index entries, 12 MiB)
		  | W200825 06:22:10.876758 1 ccl/workloadccl/fixture.go:497  error enabling automatic stats: pq: update-setting: remote wall time is too far ahead (978.18791ms) to be trustworthy
		  | Error: restoring fixture: pq: could not mark job 584135475024396289 as failed: importing 794 ranges: importing span /Table/58/2/43{4/82253/434/3/1563/11-9/85310/439/4/2866/14}: addsstable [/Table/57/2/434/82253/434/4/2283/2/0,/Table/57/2/439/85310/439/4/2866/13/0/NULL): remote wall time is too far ahead (533.986686ms) to be trustworthy: log-job: remote wall time is too far ahead (530.649214ms) to be trustworthy
		  | Error: COMMAND_PROBLEM: exit status 1
		  | (1) COMMAND_PROBLEM
		  | Wraps: (2) Node 4. Command with error:
		  |   | ```
		  |   | ./workload fixtures load tpcc --warehouses=1000 --scatter --checks=false {pgurl:1}
		  |   | ```
		  | Wraps: (3) exit status 1
		  | Error types: (1) errors.Cmd (2) *hintdetail.withDetail (3) *exec.ExitError
		  |
		  | stdout:
		Wraps: (8) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *main.withCommandDetails (8) *exec.ExitError

More

Artifacts: /tpccbench/nodes=3/cpu=4
Related:

• roachtest: tpccbench/nodes=3/cpu=4 failed #53282 roachtest: tpccbench/nodes=3/cpu=4 failed C-test-failure O-roachtest O-robot branch-master release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[tbg]

remote wall time is too far ahead (533.986686ms) to be trustworthy: log-job: remote wall time is too fa