Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #40360

Closed
cockroach-teamcity opened this issue Aug 30, 2019 · 4 comments · Fixed by #40431
Closed

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed #40360

cockroach-teamcity opened this issue Aug 30, 2019 · 4 comments · Fixed by #40431
Assignees
@solongordon
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/e8faca611a902766154ed82581d6d3a7483ad231

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/multi-region PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1460982&tab=buildLog

The test failed on branch=provisional_201908291837_v19.2.0-beta.20190903, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190829-1460982/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2114,tpcc.go:868,tpcc.go:579,test_runner.go:673: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567109858-76-n12cpu4-geo:4 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 --zones="us-east1-b,us-west1-b,europe-west2-b" {pgurl:1} returned:
		stderr:
		
		stdout:
		o:544  loaded 63 GiB table stock in 43m46.367051398s (200000000 rows, 200000000 index entries, 24 MiB)
		Error: restoring fixture: pq: importing 306 ranges: split at key /Table/54/1/441/"8\x81\xab\x1e\xb5\x86H\x00\x80\x00\x00\x00\x00\xca\x15C"/PrefixEnd failed: descriptor changed: [expected] r148:/Table/5{4/1/225/"\x1c\xe2>N\x19hG\x00\x80\x00\x00\x00\x00gK\xe0"/PrefixEnd-5} [(n6,s6):1, (n1,s1):7, (n9,s9):6, (n2,s2):8LEARNER, next=9, gen=57] != [actual] r148:/Table/5{4/1/225/"\x1c\xe2>N\x19hG\x00\x80\x00\x00\x00\x00gK\xe0"/PrefixEnd-5} [(n6,s6):1, (n1,s1):7, (n9,s9):6, (n2,s2):8, next=9, gen=58]: unexpected value: raw_bytes:"@s\026*\003\010\224\001\022\034\276\211\366\341\022\034\342>N\031hG\000\377\200\000\377\000\377\000\377\000\377gK\340\000\002\032\001\277\"\010\010\006\020\006\030\001 \000\"\010\010\001\020\001\030\007 \000\"\010\010\t\020\t\030\006 \000\"\010\010\002\020\002\030\010 \000(\t0::\n\010\267\236\341\373\212\233\344\337\025@\001" timestamp:<wall_time:1567126902000117790 > 
		Error:  exit status 1
		: exit status 1
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Aug 30, 2019
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Aug 30, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/66bd279c9aa682c2b7adcec87ec0c639b8039a33

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/multi-region PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1461635&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190830-1461635/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2114,tpcc.go:868,tpcc.go:579,test_runner.go:673: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567146353-65-n12cpu4-geo:4 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 --zones="us-east1-b,us-west1-b,europe-west2-b" {pgurl:1} returned:
		stderr:
		
		stdout:
		2 MiB)
		I190830 10:57:54.532440 72 ccl/workloadccl/fixture.go:544  loaded 8.5 GiB table history in 15m57.043065729s (60000000 rows, 120000000 index entries, 9.1 MiB)
		I190830 11:33:11.002815 77 ccl/workloadccl/fixture.go:544  loaded 46 GiB table order_line in 51m13.513826719s (600008449 rows, 600008449 index entries, 15 MiB)
		I190830 11:34:46.686256 71 ccl/workloadccl/fixture.go:544  loaded 34 GiB table customer in 52m49.197090071s (60000000 rows, 60000000 index entries, 11 MiB)
		I190830 12:02:18.181287 76 ccl/workloadccl/fixture.go:544  loaded 63 GiB table stock in 1h20m20.691860106s (200000000 rows, 200000000 index entries, 13 MiB)
		I190830 12:02:44.473786 1 ccl/workloadccl/cliccl/fixtures.go:290  restored 155 GiB bytes in 9 tables (took 1h20m47.192351043s, 32.67 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "ALTER PARTITION p1_0 OF TABLE order_line CONFIGURE ZONE USING constraints = '[+zone=us-east1-b]'": pq: partition "p1_0" does not exist on index "primary"
		Error:  exit status 1
		: exit status 1

@solongordon
Copy link
Contributor

Looks like this test needs an update due to recent partitioning changes.

@solongordon solongordon mentioned this issue Aug 30, 2019
Closed
18 tasks
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e8faca611a902766154ed82581d6d3a7483ad231

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/multi-region PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1462518&tab=buildLog

The test failed on branch=provisional_201908291837_v19.2.0-beta.20190903, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190830-1462518/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2114,tpcc.go:868,tpcc.go:579,test_runner.go:673: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567188851-75-n12cpu4-geo:4 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 --zones="us-east1-b,us-west1-b,europe-west2-b" {pgurl:1} returned:
		stderr:
		
		stdout:
		 entries, 6.9 MiB)
		I190830 23:12:19.675431 33 ccl/workloadccl/fixture.go:544  loaded 254 MiB table new_order in 16m38.225661328s (18000000 rows, 0 index entries, 261 KiB)
		I190830 23:27:22.191869 84 ccl/workloadccl/fixture.go:544  loaded 46 GiB table order_line in 31m40.74216353s (600008449 rows, 600008449 index entries, 25 MiB)
		I190830 23:27:55.760112 30 ccl/workloadccl/fixture.go:544  loaded 34 GiB table customer in 32m14.310136729s (60000000 rows, 60000000 index entries, 18 MiB)
		I190830 23:39:22.194614 83 ccl/workloadccl/fixture.go:544  loaded 63 GiB table stock in 43m40.744631516s (200000000 rows, 200000000 index entries, 24 MiB)
		I190830 23:39:48.365563 1 ccl/workloadccl/cliccl/fixtures.go:290  restored 155 GiB bytes in 9 tables (took 44m8.099666809s, 59.80 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "ALTER PARTITION p1_0 OF TABLE order_line CONFIGURE ZONE USING constraints = '[+zone=us-east1-b]'": pq: partition "p1_0" does not exist on index "primary"
		Error:  exit status 1
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/d51fa78ff90a113c9009d263dfaf58d3672670a6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=9/cpu=4/multi-region PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1463583&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190902-1463583/tpccbench/nodes=9/cpu=4/multi-region/run_1
	cluster.go:2114,tpcc.go:868,tpcc.go:579,test_runner.go:673: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1567405952-72-n12cpu4-geo:4 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false --partitions=3 --zones="us-east1-b,us-west1-b,europe-west2-b" {pgurl:1} returned:
		stderr:
		
		stdout:
		es, 7.1 MiB)
		I190902 11:11:37.980497 72 ccl/workloadccl/fixture.go:544  loaded 2.7 GiB table order in 22m7.414024198s (60000000 rows, 60000000 index entries, 2.1 MiB)
		I190902 11:39:42.580746 76 ccl/workloadccl/fixture.go:544  loaded 46 GiB table order_line in 50m12.014439932s (600008449 rows, 600008449 index entries, 16 MiB)
		I190902 11:47:37.495398 70 ccl/workloadccl/fixture.go:544  loaded 34 GiB table customer in 58m6.928858225s (60000000 rows, 60000000 index entries, 10 MiB)
		I190902 12:07:06.316148 75 ccl/workloadccl/fixture.go:544  loaded 63 GiB table stock in 1h17m35.749540034s (200000000 rows, 200000000 index entries, 14 MiB)
		I190902 12:07:31.312535 1 ccl/workloadccl/cliccl/fixtures.go:290  restored 155 GiB bytes in 9 tables (took 1h18m1.707791723s, 33.83 MiB/s)
		Error: Could not postload: could not partition tables: Couldn't exec "ALTER PARTITION p1_0 OF TABLE order_line CONFIGURE ZONE USING constraints = '[+zone=us-east1-b]'": pq: partition "p1_0" does not exist on index "primary"
		Error:  exit status 1
		: exit status 1

@solongordon solongordon mentioned this issue Sep 3, 2019
Closed
@solongordon solongordon mentioned this issue Sep 3, 2019
Merged
craig bot pushed a commit that referenced this issue Sep 3, 2019
@rytaft @solongordon
Merge #40248 #40431
48ed5f0 
40248: opt: calculate number of rows processed when costing joins r=rytaft a=rytaft

This PR updates the costing of joins to take into account the number of
rows processed by the operator. This number may be larger than the
number of output rows if an additional filter is applied as part of the
ON condition that is not used to determine equality
columns for the join.

For example, consider the query
  `SELECT * FROM abc JOIN def ON a = e AND b = 3;`

Assuming there is no index on b, if a lookup join is used to execute this
query, the number of rows processed is actually the same as the query
  `SELECT * FROM abc JOIN def ON a = e;`

The difference is that the filter b=3 must also be applied to every row in
the first query. The coster now takes this into account when determining
the cost of joins.

Fixes #34810

Release note: None

40431: workload: fix partition commands in tpcc import r=solongordon a=solongordon

The commands for partitioning indexes in the TPCC import were erroring
out due to a syntax change introduced in #39332. I updated them to use
`ALTER PARTITION ... OF INDEX` rather than `ALTER PARTITION ... OF
TABLE`.

Fixes #39005
Fixes #40360
Fixes #40416

Release note: None

Co-authored-by: Rebecca Taft <[email protected]>
Co-authored-by: Solon Gordon <[email protected]>
@craig craig bot closed this as completed in f56a83d Sep 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@solongordon solongordon

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

workload: fix partition commands in tpcc import solongordon/cockroach
3 participants
@andreimatei @solongordon @cockroach-teamcity