Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: cdc/bank failed #41177

Closed
cockroach-teamcity opened this issue Sep 28, 2019 · 26 comments · Fixed by #42746
Closed

roachtest: cdc/bank failed #41177

cockroach-teamcity opened this issue Sep 28, 2019 · 26 comments · Fixed by #42746
Assignees
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/250f4c36de2b88eff443cf9be9cd5d2759312c88

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1512280&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190928-1512280/cdc/bank/run_1
	cluster.go:1764,cdc.go:639,cdc.go:248,cdc.go:607,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569647447-15-n4cpu4:4 -- yes | sudo apt-get -q install default-jre 2>&1 > logs/apt-get-install.log returned:
		stderr:
		
		stdout:
		debconf: unable to initialize frontend: Dialog
		debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
		debconf: falling back to frontend: Readline
		debconf: unable to initialize frontend: Readline
		debconf: (This frontend requires a controlling tty.)
		debconf: falling back to frontend: Teletype
		dpkg-preconfigure: unable to re-open stdin: 
		Error:  exit status 255
		: exit status 1

@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Sep 28, 2019
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Sep 28, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4dcfd7d8899d90dd1816153bf59b5df647c9bbd3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1516560&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191002-1516560/cdc/bank/run_1
	cluster.go:1764,cdc.go:639,cdc.go:248,cdc.go:607,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1569993086-05-n4cpu4:4 -- yes | sudo apt-get -q install default-jre 2>&1 > logs/apt-get-install.log returned:
		stderr:
		
		stdout:
		debconf: unable to initialize frontend: Dialog
		debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
		debconf: falling back to frontend: Readline
		debconf: unable to initialize frontend: Readline
		debconf: (This frontend requires a controlling tty.)
		debconf: falling back to frontend: Teletype
		dpkg-preconfigure: unable to re-open stdin: 
		Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1ed03c9811f01fef31950a5cb75a7b591af6fc26

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1518416&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191003-1518416/cdc/bank/run_1
	cluster.go:1764,cdc.go:638,cdc.go:248,cdc.go:607,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570079462-09-n4cpu4:4 -- sudo apt-get -q update 2>&1 > logs/apt-get-update.log returned:
		stderr:
		
		stdout:
		Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/06026bab5cdb3695ddeba209c94175cd9460b170

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1528278&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191009-1528278/cdc/bank/run_1
	cluster.go:1764,cdc.go:653,cdc.go:648,cdc.go:256,cdc.go:608,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570597867-04-n4cpu4:4 -- CONFLUENT_CURRENT=/mnt/data1/confluent /mnt/data1/confluent/confluent-4.0.0/bin/confluent start schema-registry returned:
		stderr:
		
		stdout:
		Starting zookeeper
		
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/�zookeeper is [�[0;32mUP�[0m]
		Starting kafka
		
-
\
|
/
-
\
|
/
-
\
|
/
-Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/41bac2fd99a91d92cb3ff6426789f7e64dd6b14a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1534498&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191011-1534498/cdc/bank/run_1
	cluster.go:2146,cdc.go:356,cdc.go:608,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570770659-15-n4cpu4:4 -- ./workload run bank {pgurl:1} --max-rate=10 returned:
		stderr:
		
		stdout:
		     10.0           10.0      4.1      5.5      5.5      5.5 transfer
		   50.0s        0           10.0           10.0      4.2      7.1      7.1      7.1 transfer
		   51.0s        0           10.0           10.0      4.1      7.6      7.6      7.6 transfer
		   52.0s        0           10.0           10.0      4.1      5.0      5.0      5.0 transfer
		   53.0s        0           10.0           10.0      4.5      6.8      6.8      6.8 transfer
		   54.0s        0           10.0           10.0      4.5     10.0     10.0     10.0 transfer
		   55.0s        0           10.0           10.0      4.5      7.3      7.3      7.3 transfer
		   56.0s        0           10.0           10.0      4.5      5.2      5.2      5.2 transfer
		   57.0s        0           10.0           10.0      4.5      5.5      5.5      5.5 transfer
		   58.0s        0           10.0           10.0      4.2      6.8      6.8      6.8 transfer
		   59.0s        0           10.0           10.0      4.5      5.2      5.2      5.2 transfer
		Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/702408aa7276c315ca561ddc9ef0ecf503363d33

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1537203&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191014-1537203/cdc/bank/run_1
	cluster.go:1766,cdc.go:636,cdc.go:248,cdc.go:608,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1571029844-10-n4cpu4:4 -- curl -s https://packages.confluent.io/archive/4.0/confluent-oss-4.0.0-2.11.tar.gz | tar -xz -C /mnt/data1/confluent returned:
		stderr:
		
		stdout:
		Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8dd98ccc71626582d087852998e15620b81c7cd7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1540575&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191016-1540575/cdc/bank/run_1
	cluster.go:1766,cdc.go:653,cdc.go:648,cdc.go:256,cdc.go:608,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1571202659-10-n4cpu4:4 -- CONFLUENT_CURRENT=/mnt/data1/confluent /mnt/data1/confluent/confluent-4.0.0/bin/confluent start schema-registry returned:
		stderr:
		
		stdout:
		Starting zookeeper
		
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/�zookeeper is [�[0;32mUP�[0m]
		Starting kafka
		
-
\
|
/
-
\
|
/
-
\
|Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/86d6a7e0f571e2e2c637fc3d73318f2e2336451b

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1547645&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191020-1547645/cdc/bank/run_1
	cluster.go:1779,cdc.go:653,cdc.go:648,cdc.go:256,cdc.go:608,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1547645-1571548338-04-n4cpu4:4 -- CONFLUENT_CURRENT=/mnt/data1/confluent /mnt/data1/confluent/confluent-4.0.0/bin/confluent start schema-registry returned:
		stderr:
		
		stdout:
		Starting zookeeper
		
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-�zookeeper is [�[0;32mUP�[0m]
		Starting kafka
		
\
|
/
-
\Error:  exit status 255
		: exit status 1

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b9dc16b9d6cbd4627bac4b6329e87e29e52401b9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1571295&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191103-1571295/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=dfa4aef7 key=/Table/54/1/884/0 rw=true pri=0.10022116 stat=PENDING epo=0 ts=1572765938.760300782,15 orig=1572765938.715149650,0 min=1572765938.715149650,0 max=1572765938.760300782,14 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/239513342a2d23f683bbc1d386f87ff59cc78d10

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1575479&tab=artifacts#/cdc/bank

The test failed on branch=provisional_201910141814_v19.2.0-rc.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1575479/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=b23d21a4 key=/Table/54/1/461/0 rw=true pri=0.00955216 stat=PENDING epo=0 ts=1573004546.274592142,5 orig=1573004546.264586825,0 min=1573004546.264586825,0 max=1573004546.331066349,0 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/2535f0363b5d9b1e466ca3144fe5bcf7ff17e63c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1575491&tab=artifacts#/cdc/bank

The test failed on branch=provisional_201911052027_v19.2.0-rc.4, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1575491/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=9c537de3 key=/Table/54/1/723/0 rw=true pri=0.00255411 stat=PENDING epo=0 ts=1573005783.369381572,0 orig=1573005783.369381572,0 min=1573005783.369381572,0 max=1573005783.465432920,3 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/3b0662a83e1226dd4df2e82badacfcfd7440179f

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1575698&tab=artifacts#/cdc/bank

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1575698/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=b14aedd8 key=/Table/54/1/923/0 rw=true pri=0.02362421 stat=PENDING epo=0 ts=1573024157.315773190,1 orig=1573024157.303444679,0 max=1573024157.371373341,0 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/dcddfec777c9c799ed5717e8f14899cfdb03b6e6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1578929&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191107-1578929/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=3ee13f19 key=/Table/54/1/760/0 rw=true pri=0.00006473 stat=PENDING epo=0 ts=1573112052.376451435,0 orig=1573112052.376451435,0 min=1573112052.376451435,0 max=1573112052.482407078,0 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/e206eec2aaac3620d1d27873b830bc5f781c94aa

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1580926&tab=artifacts#/cdc/bank

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191108-1580926/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=782e8f4c key=/Table/54/1/900/0 rw=true pri=0.01940206 stat=PENDING epo=0 ts=1573197384.060418511,1 orig=1573197384.040869464,0 max=1573197384.112158952,2 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ae07b57a07428f3fb9b1d26d98b8204766091dad

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1580908&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191108-1580908/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=77a7fdb6 key=/Table/54/1/62/0 pri=0.00545215 epo=0 ts=1573199046.278745973,9 min=1573199046.267370979,0 seq=2} rw=true stat=PENDING orig=1573199046.267370979,0 max=1573199046.278745973,8 rts=0.000000000,0 wto=false

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/338deb20a6e34750635d6be5385498d5871ff68c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1580944&tab=artifacts#/cdc/bank

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191108-1580944/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=d3cdd894 key=/Table/54/1/318/0 rw=true pri=0.02602413 stat=PENDING epo=0 ts=1573199253.077809326,26 orig=1573199253.077809326,26 min=1573199253.077809326,26 max=1573199253.163803386,0 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/33b96613ae532b25a1b6b716453bece9b60ba2d6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1583742&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191109-1583742/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=e7b3b996 key=/Table/54/1/478/0 pri=0.03402326 epo=0 ts=1573284959.974446782,0 min=1573284959.974446782,0 seq=2} rw=true stat=PENDING orig=1573284959.974446782,0 max=1573284960.076375159,0 rts=0.000000000,0 wto=false

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/35e138aa3c2be545fb4e17a85ea6f1b8d6525e53

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1587967&tab=artifacts#/cdc/bank

The test failed on branch=provisional_201911111508_v20.1.0-alpha.20191118, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191112-1587967/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=c830c548 key=/Table/54/1/322/0 pri=0.00339416 epo=0 ts=1573572539.904052817,0 min=1573572539.904052817,0 seq=2} rw=true stat=PENDING orig=1573572539.904052817,0 max=1573572540.006945855,0 rts=0.000000000,0 wto=false

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8622fad01478fb4a4f05b5579eb0b8561c02e491

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1591079&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191114-1591079/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=a4339a33 key=/Table/54/1/431/0 pri=0.00675558 epo=0 ts=1573718041.760525559,1 min=1573718041.749185353,0 seq=2} rw=true stat=PENDING orig=1573718041.749185353,0 max=1573718041.760525559,0 rts=0.000000000,0 wto=false

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/53eef0857d14cc3af720e136ddaff4eeab026fd0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1599080&tab=artifacts#/cdc/bank

The test failed on branch=provisional_201911182308_v19.2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191119-1599080/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=4b4979c1 key=/Table/54/1/548/0 rw=true pri=0.02034451 stat=PENDING epo=0 ts=1574188042.437241570,1 orig=1574188042.408028297,0 min=1574188042.408028297,0 max=1574188042.475330472,2 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1c3fce73cd25fa69e5a8c05ce8e215d66b4f49e4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1599833&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191120-1599833/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=9e31c397 key=/Table/54/1/316/0 pri=0.04406557 epo=0 ts=1574235231.765756067,0 min=1574235231.765756067,0 seq=2} rw=true stat=PENDING orig=1574235231.765756067,0 max=1574235231.853160562,0 rts=0.000000000,0 wto=false

@thoszhang thoszhang mentioned this issue Nov 20, 2019
18 tasks
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/66290e232d6af34c4d09b2f90f71d520ed47dbe8

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1602090&tab=artifacts#/cdc/bank

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191121-1602090/cdc/bank/run_1
	cluster.go:2163,cdc.go:356,cdc.go:608,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" id=152d9ddf key=/Table/55/1/780/0 rw=true pri=0.01590438 stat=PENDING epo=0 ts=1574320105.003771302,1 orig=1574320104.989225384,0 max=1574320105.056587593,0 wto=false seq=2

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c128b5b9a0b1d98181bf124f775154df97948fd2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1604949&tab=artifacts#/cdc/bank

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191122-1604949/cdc/bank/run_1
	cdc.go:282,cdc.go:615,test_runner.go:697: pq: invalid option "diff"

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/668b66da0b4c7789fb13d77c43334c99a7e65793

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1604967&tab=artifacts#/cdc/bank

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191122-1604967/cdc/bank/run_1
	cdc.go:282,cdc.go:615,test_runner.go:697: pq: invalid option "diff"

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/1179c76e34d057259854d88c7948830d2b4b05cd

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1606953&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191123-1606953/cdc/bank/run_1
	cluster.go:2163,cdc.go:363,cdc.go:615,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=cc82a554 key=/Table/54/1/639/0 pri=0.03103637 epo=0 ts=1574493981.847114151,0 min=1574493981.847114151,0 seq=2} rw=true stat=PENDING rts=1574493981.847114151,0 wto=false max=1574493981.949383029,0

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/2e1807ad6ac9656d2afa40cda6533f6a29e80383

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1606971&tab=artifacts#/cdc/bank

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191123-1606971/cdc/bank/run_1
	cdc.go:282,cdc.go:615,test_runner.go:697: pq: invalid option "diff"

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/60b7b3b2ca511cf9adb30799fcf2b8cffd6223d9

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=cdc/bank PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1608628&tab=artifacts#/cdc/bank

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191125-1608628/cdc/bank/run_1
	cluster.go:2163,cdc.go:363,cdc.go:615,test_runner.go:697: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED_POSSIBLE_REPLAY): "sql txn" meta={id=d67458f7 key=/Table/54/1/882/0 pri=0.00418224 epo=0 ts=1574667276.863235538,1 min=1574667276.818228575,0 seq=2} rw=true stat=PENDING rts=1574667276.818228575,0 wto=false max=1574667276.886386582,0

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Nov 25, 2019
craig bot pushed a commit that referenced this issue Nov 25, 2019
42650: sql: stop observing the CommitTimestamp in TRUNCATE r=ajwerner a=ajwerner

In #40581 we stopped observing the commit timestamp to write it into table
descriptors. In this change I overlooked (rather forgot) about this additional
place in the code where we observed the commit timestamp. As far as I can tell
we don't read this field anywhere ever. Furthermore we know that the the table
descriptor in question to which we are referring must be alive and equal to
the provided value at the timestamp at which it was read due to serializability.
In short, this minor change continues to populate the field with a sensible
value and will permit TRUNCATE to be pushed.

Fixes #41566.

Release note (bug fix): Long running transactions which attempt to TRUNCATE
can now be pushed and will commit in cases where they previously could fail
or retry forever.

42746: roachtest/cdc: fix cdc/bank and cdc/schemareg r=nvanbenschoten a=nvanbenschoten

Fixes #41177.
Fixes #42690.

These were both broken by #41793 because prior versions of crdb didn't support the `WITH diff` option.

Co-authored-by: Andrew Werner <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in 7044348 Nov 25, 2019
@andreimatei andreimatei mentioned this issue Dec 16, 2019
19 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants