Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/changefeedccl: package timed out under stress #32444

Closed
cockroach-teamcity opened this issue Nov 17, 2018 · 5 comments · Fixed by #32465
Closed

ccl/changefeedccl: package timed out under stress #32444

cockroach-teamcity opened this issue Nov 17, 2018 · 5 comments · Fixed by #32465
Assignees
@danhhz
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
19.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/ecbef846abb2855afca8532097c42864d688f51d

Parameters:

TAGS=
GOFLAGS=

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=(unknown) PKG=github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1016922&tab=buildLog

Slow failing tests:
TestChangefeedTruncateRenameDrop/sinkless - 0.17s

Slow passing tests:
TestChangefeedSchemaChangeNoBackfill - 4.94s
TestChangefeedSchemaChangeAllowBackfill - 3.45s
TestChangefeedCursor - 2.21s
TestChangefeedInterleaved - 2.00s
TestChangefeedColumnFamily - 1.40s
TestChangefeedEnvelope - 1.12s
TestChangefeedBasics - 0.95s
TestChangefeedResolvedFrequency - 0.90s
TestChangefeedMultiTable - 0.88s
TestChangefeedTimestamps - 0.87s
TestChangefeedUpdatePrimaryKey - 0.86s
TestChangefeedComputedColumn - 0.80s
@cockroach-teamcity cockroach-teamcity added this to the 2.2 milestone Nov 17, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Nov 17, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ecbef846abb2855afca8532097c42864d688f51d

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=(unknown) PKG=github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1017185&tab=buildLog

Slow failing tests:
TestChangefeedSchemaChangeNoBackfill/sinkless/add_column - 0.09s

Slow passing tests:
TestChangefeedCursor - 13.00s
TestChangefeedEnvelope - 5.80s
TestChangefeedBasics - 5.43s
TestChangefeedTimestamps - 5.13s
TestChangefeedResolvedFrequency - 5.01s
TestChangefeedMultiTable - 4.75s

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b49107e6da02b19c1ad99bed1acfb4ff8a262936

Parameters:

TAGS=
GOFLAGS=-race

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
make stressrace TESTS=(unknown) PKG=github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1017935&tab=buildLog

Slow failing tests:
TestChangefeedSchemaChangeNoBackfill/sinkless/add_column - 0.10s

Slow passing tests:
TestChangefeedCursor - 13.38s
TestChangefeedMultiTable - 5.61s
TestChangefeedEnvelope - 5.56s
TestChangefeedResolvedFrequency - 5.19s
TestChangefeedTimestamps - 5.13s
TestChangefeedBasics - 4.38s

@tbg tbg assigned danhhz and vivekmenezes and unassigned andreimatei Nov 19, 2018
@tbg
Copy link
Member

tbg commented Nov 19, 2018

@vivekmenezes I expect this to be similar to #32433, though I haven't taken a look.

@tbg
Copy link
Member

tbg commented Nov 19, 2018 •
edited
Loading

Similar failure in TestChangefeedTruncateRenameDrop/sinkless in CI.

@tbg tbg mentioned this issue Nov 19, 2018
Merged
tbg added a commit to tbg/cockroach that referenced this issue Nov 19, 2018
@tbg
Revert "engineccl: ignore intents beneath start in MVCCIncrementalIte…
ceb4072 
…rator"

This reverts commit 765df40.

Unfortunately, cockroachdb#32421 has turned

> make test TESTS=TestChangefeedSchemaChangeNoBackfill PKG=github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl TESTTIMEOUT=30s TESTFLAGS=-v

and various other tests into failing propositions (they take forever).
This slows down the CDC pipeline and the symptoms affect various other
CDC tests so that skipping the test isn't an attractive option.

Besides, this was clearly not intended, so there's probably something
wrong with the code being reverted here.

We should investigate and submit a fix.

Closes cockroachdb#32461.
Fixes cockroachdb#32433.
Fixes cockroachdb#32444.

Release note: None
@tbg tbg mentioned this issue Nov 19, 2018
Merged
craig bot pushed a commit that referenced this issue Nov 19, 2018
@tbg
Merge #32465
5377795 
32465: Revert "engineccl: ignore intents beneath start in MVCCIncrementalIterator" r=benesch a=tbg

This reverts commit 765df40.

Unfortunately, #32421 has turned

> make test TESTS=TestChangefeedSchemaChangeNoBackfill PKG=github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl TESTTIMEOUT=30s TESTFLAGS=-v

and various other tests into failing propositions (they take forever).
This slows down the CDC pipeline and the symptoms affect various other
CDC tests so that skipping the test isn't an attractive option.

Besides, this was clearly not intended, so there's probably something
wrong with the code being reverted here.

We should investigate and submit a fix.

Closes #32461.
Fixes #32433.
Fixes #32444.

Release note: None

Co-authored-by: Tobias Schottdorf <[email protected]>
@vivekmenezes
Copy link
Contributor

@mrtracy so he's aware

@vivekmenezes vivekmenezes removed their assignment Nov 19, 2018
@craig craig bot closed this as completed in #32465 Nov 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danhhz danhhz

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
19.1
Development

Successfully merging a pull request may close this issue.

Revert "engineccl: ignore intents beneath start in MVCCIncrementalIterator" tbg/cockroach
5 participants
@danhhz @andreimatei @tbg @vivekmenezes @cockroach-teamcity