beta: 20160414

[mberhault]

@bdarnell: let's track this in an issue instead. More visible than arbitrary emails.

sha 08d6640
Running on cluster:

• beta since 2016/04/13 morning. block_writer (concurrency=3) and photos against each node
• gamma since 2016/04/13 morning. block_writer (concurrency=5) and photos against each node

Notes:
Turning up concurrency on the block_writer seems tied to problems. Often getting deadline exceeded errors and more crashes.

Previously known issues:

• stability: index out of range on (*TxnMeta).MarshalTo #5998: index out of range on (*TxnMeta).MarshalTo

Seen for the first time, but may be artifacts of restarts/crashes:

• stability: panic on RangeLookup dispatched to correct range #6000: panic on RangeLookup dispatched to correct range
• stability: unexpected panic: llrb: inverted range #6027: unexpected panic: llrb: inverted range

Perhaps new?

• stability: crash on (*Timestamp).MarshalTo #6020: crash on (*Timestamp).MarshalTo

Fixed in this sha:

• stability: crash on nil pointer in markDead #6001: crash on nil pointer in markDead

[bdarnell]

What about #5970? Isn't the beta cluster basically down?

[mberhault]

That one is most likely not new.
The beta cluster is still mostly hosed: manual sql statements on system objects work, and the UI is responsive, but everything else times out.

[nvanbenschoten]

#5994 is going to be a necessary breaking change, so ideally we would be able to get this into this weeks release instead of waiting for next week. It should be merged within the next hour. Is it possible for it to be included here?

[bdarnell]

It's not going to get as much burn-in time, but I guess that doesn't matter since nothing on the beta cluster is using decimals anyway. I'd be OK with putting #5994 in this release although I'd want it to be cherry-picked rather than picking up everything else that's been merged since 08d6640.

[nvanbenschoten]

I just merged the branch so we should be good to cherry pick for the release. Note that the PR included 3 commits which should all be included.

[mberhault]

another new one for this sha: #6052 concurrent map read/write in Store

[bdarnell]

The concurrency-related issues (#6052, #6020, #5998) seem pretty common, and we weren't seeing those before. Is that a blocker for this release? Have we made any headway in understanding those? I haven't found any likely suspects in the git logs. Is this a regression worth blocking this week's beta for, or is this likely to be a preexisting issue that may have been exposed by other brokenness on the beta cluster (and more frequent retries, etc)?

I've prepared a cherry-picked version of #5994 in https://github.com/bdarnell/cockroach/tree/beta-branch-20160414. I haven't started running it on any of the test clusters.

[mberhault]

I don't think any of the concurrency crashes are new. Some were definitely seen before, and some seemed to only have happened on the gamma cluster. So I don't think we need to hold up the beta for those.

[bdarnell]

OK, I'll tag it and start the build.