Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cdc/w=100/nodes=3/init=false/chaos=true failed on release-2.1 #29196

Closed
cockroach-teamcity opened this issue Aug 28, 2018 · 7 comments
Closed

roachtest: cdc/w=100/nodes=3/init=false/chaos=true failed on release-2.1 #29196

cockroach-teamcity opened this issue Aug 28, 2018 · 7 comments
Assignees
@mrtracy
Labels
A-cdc Change Data Capture C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/11b99034b7a7313355228c6b6851752cb329dbe7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=cdc/w=100/nodes=3/init=false/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=867071&tab=buildLog

	test.go:494,cluster.go:822,cdc.go:63,cdc.go:108,cdc.go:290: /home/agent/work/.go/bin/roachprod run teamcity-867071-cdc-w-100-nodes-3-init-false-chaos-true:4 -- ./workload fixtures load tpcc --warehouses=100 --checks=false {pgurl:3}: exit status 1
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone Aug 28, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 28, 2018
@mrtracy
Copy link
Contributor

mrtracy commented Aug 28, 2018

This is an unstable test which is designed to be failing at this point. Completion of #28636 should allow this test to pass.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/edef8e7ba4d8adb522fb1c4fde3415bc4973d327

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=cdc/w=100/nodes=3/init=false/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=869190&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/f9c094897e9574685b091e7e0969d333d2991290

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=cdc/w=100/nodes=3/init=false/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=869242&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/def23eda0df4250b0113997bc223103a43508179

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=cdc/w=100/nodes=3/init=false/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=882125&tab=buildLog

tbg added a commit to tbg/cockroach that referenced this issue Sep 4, 2018
@tbg
roachtest: skip (intentionally) failing Kafka chaos test
fc4a473 
This test has no chance of passing until Kafka chaos is actually
supported (see cockroachdb#28636).

Touches cockroachdb#29196.

Release note: None
@tbg tbg mentioned this issue Sep 4, 2018
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/def23eda0df4250b0113997bc223103a43508179

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=cdc/w=100/nodes=3/init=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=882125&tab=buildLog

craig bot pushed a commit that referenced this issue Sep 4, 2018
@danhhz @jseldess
@petermattis @RaduBerinde @tbg @benesch
Merge #29367 #29427 #29488 #29538 #29546 #29550 #29552
5692414 
29367: changefeedccl: error when a watched table backfills r=mrtracy,vivekmenezes a=danhhz

When a table is currently being backfilled for a schema change (e.g.
adding a column with a default value), it's unclear what the expectation
is for any rows that are changed during the backfill. Our current
invariant is that rows are emitted with an updated timestamp and a later
SELECT ... AS OF SYSTEM TIME for that row would exactly match the
emitted data. During the backfill, there is nothing we can emit that
would definitely meet that invariant (because the backfill can be
aborted and rolled back).

In the meantime, this commit makes sure that we error whenever a
backfill happens, even if it's fast enough that we never get it from
leasing.

This also paves the way for switching to RangeFeed, which doesn't have
the convenient `fetchSpansForTargets` hook that the ExportRequest based
poller was (ab)using.

Closes #28643

Release note (bug fix): CHANGEFEEDs now error when a watched table
backfills (instead of undefined behavior)

29427: docs: Fix replace and link in table_ref diagram r=jseldess a=jseldess

Needed for cockroachdb/docs#3682.

Release note: None

29488: roachtest: add acceptance/decommission r=benesch,tschottdorf a=petermattis

Move the decommission acceptance test to a new acceptance/decommission
roachtest.

Fixes #29151

Release note: None

29538: stats: document stats-related commands as experimental r=RaduBerinde a=RaduBerinde

Update the documentation inside `sql.y` to designate the stats-related
statements as experimental.

Release note: None

29546: roachtest: skip (intentionally) failing Kafka chaos test r=petermattis a=tschottdorf

This test has no chance of passing until Kafka chaos is actually
supported (see #28636).

Touches #29196.

Release note: None

29550: testcluster: make manual replication mode disable the merge queue r=petermattis a=benesch

TestClusters have a manual replication mode for use in tests that need
to precisely control replication on a cluster. Teach that mode to
disable the merge queue in addition to the split and replicate queues.
This decreases the number of tests that need to directly disable the
merge queue.

Release note: None

29552: ui: add attributes to login form so LastPass will autofill it r=vilterp a=vilterp

LastPass wasn't confident enough to autofill and autologin without these
attributes.

Fixes #29529 (fixes for LastPass, but maybe not other PW managers)

Release note (admin ui change): Add attributes to the login form to allow LastPass to properly recognize it.

Co-authored-by: Daniel Harrison <[email protected]>
Co-authored-by: Jesse Seldess <[email protected]>
Co-authored-by: Peter Mattis <[email protected]>
Co-authored-by: Radu Berinde <[email protected]>
Co-authored-by: Tobias Schottdorf <[email protected]>
Co-authored-by: Nikhil Benesch <[email protected]>
Co-authored-by: Pete Vilter <[email protected]>
Amruta-Ranade pushed a commit that referenced this issue Sep 4, 2018
@danhhz @jseldess
@petermattis @RaduBerinde @tbg @benesch @Amruta-Ranade
Merge #29367 #29427 #29488 #29538 #29546 #29550 #29552
4f06619 
29367: changefeedccl: error when a watched table backfills r=mrtracy,vivekmenezes a=danhhz

When a table is currently being backfilled for a schema change (e.g.
adding a column with a default value), it's unclear what the expectation
is for any rows that are changed during the backfill. Our current
invariant is that rows are emitted with an updated timestamp and a later
SELECT ... AS OF SYSTEM TIME for that row would exactly match the
emitted data. During the backfill, there is nothing we can emit that
would definitely meet that invariant (because the backfill can be
aborted and rolled back).

In the meantime, this commit makes sure that we error whenever a
backfill happens, even if it's fast enough that we never get it from
leasing.

This also paves the way for switching to RangeFeed, which doesn't have
the convenient `fetchSpansForTargets` hook that the ExportRequest based
poller was (ab)using.

Closes #28643

Release note (bug fix): CHANGEFEEDs now error when a watched table
backfills (instead of undefined behavior)

29427: docs: Fix replace and link in table_ref diagram r=jseldess a=jseldess

Needed for cockroachdb/docs#3682.

Release note: None

29488: roachtest: add acceptance/decommission r=benesch,tschottdorf a=petermattis

Move the decommission acceptance test to a new acceptance/decommission
roachtest.

Fixes #29151

Release note: None

29538: stats: document stats-related commands as experimental r=RaduBerinde a=RaduBerinde

Update the documentation inside `sql.y` to designate the stats-related
statements as experimental.

Release note: None

29546: roachtest: skip (intentionally) failing Kafka chaos test r=petermattis a=tschottdorf

This test has no chance of passing until Kafka chaos is actually
supported (see #28636).

Touches #29196.

Release note: None

29550: testcluster: make manual replication mode disable the merge queue r=petermattis a=benesch

TestClusters have a manual replication mode for use in tests that need
to precisely control replication on a cluster. Teach that mode to
disable the merge queue in addition to the split and replicate queues.
This decreases the number of tests that need to directly disable the
merge queue.

Release note: None

29552: ui: add attributes to login form so LastPass will autofill it r=vilterp a=vilterp

LastPass wasn't confident enough to autofill and autologin without these
attributes.

Fixes #29529 (fixes for LastPass, but maybe not other PW managers)

Release note (admin ui change): Add attributes to the login form to allow LastPass to properly recognize it.

Co-authored-by: Daniel Harrison <[email protected]>
Co-authored-by: Jesse Seldess <[email protected]>
Co-authored-by: Peter Mattis <[email protected]>
Co-authored-by: Radu Berinde <[email protected]>
Co-authored-by: Tobias Schottdorf <[email protected]>
Co-authored-by: Nikhil Benesch <[email protected]>
Co-authored-by: Pete Vilter <[email protected]>
tbg added a commit to tbg/cockroach that referenced this issue Sep 6, 2018
@tbg
roachtest: skip (intentionally) failing Kafka chaos test
f433b66 
This test has no chance of passing until Kafka chaos is actually
supported (see cockroachdb#28636).

Touches cockroachdb#29196.

Release note: None
@tbg tbg added the A-cdc Change Data Capture label Sep 18, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c25ba3c680aa1e1d4fd6c669e40177ed6d55c5e6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stressrace instead of stress and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stress TESTS=cdc/w=100/nodes=3/init=false/chaos=true PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-stderr=false -maxtime 20m -timeout 10m'

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=952739&tab=buildLog

The test failed on master:
	test.go:570,cluster.go:975,cdc.go:35,cdc.go:114,cdc.go:307: /home/agent/work/.go/bin/roachprod run teamcity-952739-cdc-w-100-nodes-3-init-false-chaos-true:4 -- yes | sudo apt-get -q install default-jre returned:
		stderr:
		
		stdout:
		27 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/p/pulseaudio/libpulse0_8.0-0ubuntu3.10_amd64.deb  503  Service Unavailable [IP: 35.192.35.27 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/j/java-common/default-jre_1.8-56ubuntu2_amd64.deb  503  Service Unavailable [IP: 35.192.35.27 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/f/fonts-dejavu/fonts-dejavu-extra_2.35-1_all.deb  503  Service Unavailable [IP: 35.192.35.27 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/h/hicolor-icon-theme/hicolor-icon-theme_0.15-0ubuntu1.1_all.deb  503  Service Unavailable [IP: 35.192.35.27 80]
		
		E: Failed to fetch http://us-central1.gce.archive.ubuntu.com/ubuntu/pool/main/g/gtk+2.0/libgtk2.0-bin_2.24.30-1ubuntu1.16.04.2_amd64.deb  503  Service Unavailable [IP: 35.192.35.27 80]
		
		E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
		Error:  exit status 100
		: exit status 1

@tbg
Copy link
Member

tbg commented Oct 9, 2018

network fluke. The test seems to have passed for a while so I'm going to close the whole thing.

@tbg tbg closed this as completed Oct 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrtracy mrtracy

Labels
A-cdc Change Data Capture C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
3 participants
@tbg @mrtracy @cockroach-teamcity