Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: TestConditionalPutUpdatesTSCacheOnError failed under stress #38256

Closed
cockroach-teamcity opened this issue Jun 18, 2019 · 2 comments · Fixed by #38295
Closed

storage: TestConditionalPutUpdatesTSCacheOnError failed under stress #38256

cockroach-teamcity opened this issue Jun 18, 2019 · 2 comments · Fixed by #38295
Assignees
@nvanbenschoten
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
19.2

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/5f1d58bc79224fa7b83645fc8822d137a83e6fd2

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestConditionalPutUpdatesTSCacheOnError PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1345149&tab=buildLog

=== RUN   TestConditionalPutUpdatesTSCacheOnError
I190618 06:12:31.241327 16741 util/stop/stopper.go:544  quiescing; tasks left:
1      [async] intent_resolver_ir_batcher
1      [async] intent_resolver_gc_batcher
I190618 06:12:31.241351 16741 util/stop/stopper.go:544  quiescing; tasks left:
1      [async] intent_resolver_gc_batcher
--- FAIL: TestConditionalPutUpdatesTSCacheOnError (0.01s)
    replica_test.go:2826: expected write timestamp to upgrade to 2.000000000,1; got 2.000000000,2
    replica_test.go:2870: expected write timestamp to upgrade to 2.000000000,1; got 2.000000000,2
@cockroach-teamcity cockroach-teamcity added this to the 19.2 milestone Jun 18, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jun 18, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/d2818382eea5612d3d3e3263941456e0734776e3

Parameters:

TAGS=
GOFLAGS=-parallel=4

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=TestConditionalPutUpdatesTSCacheOnError PKG=github.com/cockroachdb/cockroach/pkg/storage TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1348344&tab=buildLog

=== RUN   TestConditionalPutUpdatesTSCacheOnError
I190619 06:34:36.435891 16743 util/stop/stopper.go:544  quiescing; tasks left:
1      [async] intent_resolver_ir_batcher
1      [async] intent_resolver_gc_batcher
I190619 06:34:36.435922 16743 util/stop/stopper.go:544  quiescing; tasks left:
1      [async] intent_resolver_gc_batcher
--- FAIL: TestConditionalPutUpdatesTSCacheOnError (0.01s)
    replica_test.go:2826: expected write timestamp to upgrade to 2.000000000,1; got 2.000000000,2
    replica_test.go:2870: expected write timestamp to upgrade to 2.000000000,1; got 2.000000000,2

@tbg tbg assigned nvanbenschoten and unassigned spencerkimball Jun 19, 2019
@tbg
Copy link
Member

tbg commented Jun 19, 2019

@nvanbenschoten this must be fallout from #38122

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Jun 19, 2019
@nvanbenschoten
storage: fix flake in Test{Conditional,Init}PutUpdatesTSCacheOnError
e1f60cd 
Fixes cockroachdb#38256.

A request was slipping in between the manual clock update and the first
request the test sent with an unspecified timestamp. This commit fixes
the issue by explicitly specifying the timestamp.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Jun 19, 2019
Merged
craig bot pushed a commit that referenced this issue Jun 19, 2019
@jeffrey-xiao @nvanbenschoten
Merge #38079 #38295
a9802cd 
38079: bulk: use expiration time to disable automatic merging in import, backfill, restore r=jeffrey-xiao a=jeffrey-xiao

The existing mechanism to prevent the merge queue from automatically merging splits created in import, backfill, and restore was to gossip the table keys that the merge queue should ignore when scanning
replicas. Now that there is support for specifying an expiration time at a range level, we can use that instead of the gossip mechanism.

All splits created during backfill, restore, and import use an expiration time of an hour. The rationale behind using an expiration time rather than unsplitting the ranges at the end of the operation is because adding an additional O(n) cost to bulk operations is not ideal when n can be large.

Fixes #37697.

@dt I don't have context on how long import jobs take, so an hour might be too conservative.

38295: storage: fix flake in Test{Conditional,Init}PutUpdatesTSCacheOnError r=nvanbenschoten a=nvanbenschoten

Fixes #38256.

A request was slipping in between the manual clock update and the first
request the test sent with an unspecified timestamp. This commit fixes
the issue by explicitly specifying the timestamp.

Release note: None

Co-authored-by: Jeffrey Xiao <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@craig craig bot closed this as completed in #38295 Jun 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
19.2
Development

Successfully merging a pull request may close this issue.

storage: fix flake in Test{Conditional,Init}PutUpdatesTSCacheOnError nvanbenschoten/cockroach
4 participants
@tbg @nvanbenschoten @spencerkimball @cockroach-teamcity