scripts: enhance the release notes #32416

knz · 2018-11-16T14:07:17Z

With this the amount of release notes for the first 2.2 alpha in cockroachdb/docs#4051 is reduced to just under two pages.

Also this PR makes it easier to monitor progress during the execution of the script.

Release note: None

With this patch, the release notes script learns how to exclude listing release notes that are also present on a different range (e.g branch) of commits. This can be used to e.g. exclude all commits already on the first version of a release branch (eg v2.1.0) when releasing an alpha (eg v2.2-alpha). Usage: ``` python3.6 scripts/release-notes.py \ --from v2.2.0-alpha.00000000 --until <Candidate 2.2 SHA> \ --exclude-from v2.2.0-alpha.00000000 --exclude-until v2.1.0 ``` Release note: None

cockroach-teamcity · 2018-11-16T14:07:24Z

This change is

knz · 2018-11-16T14:08:11Z

The relevant command:


python3.6 scripts/release-notes.py \
    --from v2.2.0-alpha.00000000 --until <Candidate 2.2 SHA> \
    --exclude-from v2.2.0-alpha.00000000 --exclude-until v2.1.0

jseldess

Just tested and it seems to work perfectly. Thank you, @knz, for saving me an immense amount of time.

LGREATTM, but you might want to have another dev review the code.

jseldess

Just have two small requests.

One is inline (adding copy to clipboard for the docker image code block).

The second I don't know exactly where to pinpoint in the code. It'd be great if you could remove the line breaks from individual bullets. I know the line width you use is friendlier on the eyes, but I end up removing them anyway in the release notes file because we don't use that convention and because the script I use to auto-generate docs issues from release notes currently expects a single line per bullet.

Reviewable status: complete! 0 of 0 LGTMs obtained

scripts/release-notes.py, line 754 at r1 (raw file):

    print("""### Docker image

~~~shell

It'd be wonderful if you'd put this on the line preceding ~~~shell:

{% include copy-clipboard.html %}

knz · 2018-11-16T14:36:18Z

yes this seems reasonable. let me have a look

Release note: None

This new option unwraps every release note in the output document. Release note: None

knz · 2018-11-16T14:41:58Z

I have added a new option --one-line which does what you request. Please try it.

jseldess · 2018-11-16T14:47:03Z

Works great! Thanks again, @knz!

knz · 2018-11-16T14:49:29Z

I will take payment in the form of you spending some time with me to resume the conversation around #29810!

jseldess · 2018-11-16T14:52:39Z

Haha. Yes, sorry for dropping the ball on that. I need to edit the release notes today, but I'll reserve time early next week to get back to that with you.

knz · 2018-11-16T14:57:31Z

I'll even happily wait until next week, or the week after so we can avoid the stress of Thanksgiving.

jseldess

Reviewable status: complete! 0 of 0 LGTMs obtained

scripts/release-notes.py, line 132 at r2 (raw file):

# Section titles for release notes.
relnotetitles = {
    'cli change': "Command line changes",

Please change Command line changes to Command-line changes.

scripts/release-notes.py, line 725 at r2 (raw file):

    print("---")
    print("title: What&#39;s New in", current_version)
    print("toc: false")

Since this is still open, would you please change toc: false to toc: true.

scripts/release-notes.py, line 764 at r2 (raw file):

{% include copy-clipboard.html %}
~~~shell
docker pull cockroachdb/cockroach:""" + current_version + """

Since this is still open, would you mind adding $ to the start of the command as well?

$ docker pull cockroachdb/cockroach:""" + current_version + """

@jseldess

... as requested by @jseldess. Release note: None

knz

Reviewable status: complete! 0 of 0 LGTMs obtained

scripts/release-notes.py, line 754 at r1 (raw file):

Previously, jseldess (Jesse Seldess) wrote…

It'd be wonderful if you'd put this on the line preceding ~~~shell:
{% include copy-clipboard.html %}

Done.

scripts/release-notes.py, line 132 at r2 (raw file):

Previously, jseldess (Jesse Seldess) wrote…

Please change Command line changes to Command-line changes.

Done.

scripts/release-notes.py, line 725 at r2 (raw file):

Previously, jseldess (Jesse Seldess) wrote…

Since this is still open, would you please change toc: false to toc: true.

Done.

scripts/release-notes.py, line 764 at r2 (raw file):

Previously, jseldess (Jesse Seldess) wrote…

Since this is still open, would you mind adding $ to the start of the command as well?
$ docker pull cockroachdb/cockroach:""" + current_version + """

Done.

knz · 2018-11-29T17:13:17Z

bors r+

31997: storage/spanlatch: create spanlatch.Manager using immutable btrees r=nvanbenschoten a=nvanbenschoten Informs #4768. Informs #31904. This change was inspired by #31904 and is a progression of the thinking started in #4768 (comment). The change introduces `spanlatch.Manager`, which will replace the `CommandQueue` **in a future PR**. The new type isn't hooked up yet because doing so will require a lot of plumbing changes in that storage package that are best kept in a separate PR. The structure uses a new strategy that reduces lock contention, simplifies the code, avoids allocations, and makes #31904 easier to implement. The primary objective, reducing lock contention, is addressed by minimizing the amount of work we perform under the exclusive "sequencing" mutex while locking the structure. This is made possible by employing a copy-on-write strategy. Before this change, commands would lock the queue, create a large slice of prerequisites, insert into the queue and unlock. After the change, commands lock the manager, grab an immutable snapshot of the manager's trees in O(1) time, insert into the manager, and unlock. They can then iterate over the immutable tree snapshot outside of the lock. Effectively, this means that the work performed under lock is linear with respect to the number of spans that a command declares but NO LONGER linear with respect to the number of other commands that it will wait on. This is important because `Replica.beginCmds` repeatedly comes up as the largest source of mutex contention in our system, especially on hot ranges. The use of immutable snapshots also simplifies the code significantly. We're no longer copying our prereqs into a slice so we no longer need to carefully determine which transitive dependencies we do or don't need to wait on explicitly. This also makes lock cancellation trivial because we no longer explicitly hold on to our prereqs at all. Instead, we simply iterate through the snapshot outside of the lock. While rewriting the structure, I also spent some time optimizing its allocations. Under normal operation, acquiring a latch now incurs only a single allocation - that being for the `spanlatch.Guard`. All other allocations are avoided through object pooling where appropriate. The overhead of using a copy-on-write technique is almost entirely avoided by atomically reference counting immutable btree nodes, which allows us to release them back into the btree node pools when they're no longer needed. This means that we don't expect any allocations when inserting into the internal trees, even with the copy-on-write policy. Finally, this will make the approach taken in #31904 much more natural. Instead of tracking dependents and prerequisites for speculative reads and then iterating through them to find overlaps after, we can use the immutable snapshots directly! We can grab a snapshot and sequence ourselves as usual, but avoid waiting for prereqs. We then execute optimistically before finally checking whether we overlapped any of our prereqs. The great thing about this is that we already have the prereqs in an interval tree structure, so we get an efficient validation check for free. ### Naming changes | Before | After | |----------------------------|-----------------------------------| | `CommandQueue` | `spanlatch.Manager` | | "enter the command queue" | "acquire span latches" | | "exit the command queue" | "release span latches" | | "wait for prereq commands" | "wait for latches to be released" | The use of the word "latch" is based on the definition of latches presented by Goetz Graefe in https://15721.courses.cs.cmu.edu/spring2016/papers/a16-graefe.pdf (see https://i.stack.imgur.com/fSRzd.png). An important reason for avoiding the word "lock" here is that it is critical for understanding that we don't confuse the operational locking performed by the CommandQueue/spanlatch.Manager with the transaction-scoped locking enforced by intents and our transactional concurrency control model. ### Microbenchmarks NOTE: these are single-threaded benchmarks that don't benefit at all from the concurrency improvements enabled by this new structure. ``` name old time/op new time/op delta ReadOnlyMix/size=1-4 706ns ±20% 404ns ±10% -42.81% (p=0.008 n=5+5) ReadOnlyMix/size=4-4 649ns ±23% 382ns ± 5% -41.13% (p=0.008 n=5+5) ReadOnlyMix/size=16-4 611ns ±16% 367ns ± 5% -39.83% (p=0.008 n=5+5) ReadOnlyMix/size=64-4 692ns ±14% 370ns ± 1% -46.49% (p=0.016 n=5+4) ReadOnlyMix/size=128-4 637ns ±22% 398ns ±14% -37.48% (p=0.008 n=5+5) ReadOnlyMix/size=256-4 676ns ±15% 385ns ± 4% -43.01% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=0-4 12.2µs ± 4% 0.6µs ±17% -94.85% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=1-4 7.88µs ± 2% 0.55µs ± 7% -92.99% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=4-4 4.19µs ± 3% 0.58µs ± 5% -86.26% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=16-4 2.09µs ± 6% 0.54µs ±13% -74.13% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=64-4 875ns ±17% 423ns ±29% -51.64% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=128-4 655ns ± 6% 362ns ±16% -44.71% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=256-4 549ns ±16% 314ns ±13% -42.73% (p=0.008 n=5+5) name old alloc/op new alloc/op delta ReadOnlyMix/size=1-4 223B ± 0% 160B ± 0% -28.25% (p=0.079 n=4+5) ReadOnlyMix/size=4-4 223B ± 0% 160B ± 0% -28.25% (p=0.008 n=5+5) ReadOnlyMix/size=16-4 223B ± 0% 160B ± 0% -28.25% (p=0.008 n=5+5) ReadOnlyMix/size=64-4 223B ± 0% 160B ± 0% -28.25% (p=0.008 n=5+5) ReadOnlyMix/size=128-4 217B ± 4% 160B ± 0% -26.27% (p=0.008 n=5+5) ReadOnlyMix/size=256-4 223B ± 0% 160B ± 0% -28.25% (p=0.079 n=4+5) ReadWriteMix/readsPerWrite=0-4 1.25kB ± 0% 0.16kB ± 0% -87.15% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=1-4 1.00kB ± 0% 0.16kB ± 0% -84.00% (p=0.079 n=4+5) ReadWriteMix/readsPerWrite=4-4 708B ± 0% 160B ± 0% -77.40% (p=0.079 n=4+5) ReadWriteMix/readsPerWrite=16-4 513B ± 0% 160B ± 0% -68.81% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=64-4 264B ± 0% 160B ± 0% -39.39% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=128-4 221B ± 0% 160B ± 0% -27.60% (p=0.079 n=4+5) ReadWriteMix/readsPerWrite=256-4 198B ± 0% 160B ± 0% -19.35% (p=0.008 n=5+5) name old allocs/op new allocs/op delta ReadOnlyMix/size=1-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadOnlyMix/size=4-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadOnlyMix/size=16-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadOnlyMix/size=64-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadOnlyMix/size=128-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadOnlyMix/size=256-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadWriteMix/readsPerWrite=0-4 38.0 ± 0% 1.0 ± 0% -97.37% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=1-4 24.0 ± 0% 1.0 ± 0% -95.83% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=4-4 12.0 ± 0% 1.0 ± 0% -91.67% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=16-4 5.00 ± 0% 1.00 ± 0% -80.00% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=64-4 2.00 ± 0% 1.00 ± 0% -50.00% (p=0.008 n=5+5) ReadWriteMix/readsPerWrite=128-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ReadWriteMix/readsPerWrite=256-4 1.00 ± 0% 1.00 ± 0% ~ (all equal) ``` There are a few interesting things to point about about these benchmark results: - The `ReadOnlyMix` results demonstrate a fixed improvement, regardless of size. This is due to the replacement of the hash-map with a linked-list for the readSet structure. - The `ReadWriteMix` is more interesting. We see that the spanlatch implementation is faster across the board. This is especially true with a high write/read ratio. - We see that the allocated memory stays constant regardless of the write/read ratio in the spanlatch implementation. This is due to the memory recylcing that it performs on btree nodes. It is not the case for the CommandQueue implementation. Release note: None 32416: scripts: enhance the release notes r=knz a=knz Fixes #25180. With this the amount of release notes for the first 2.2 alpha in cockroachdb/docs#4051 is reduced to just under two pages. Also this PR makes it easier to monitor progress during the execution of the script. Co-authored-by: Nathan VanBenschoten <[email protected]> Co-authored-by: Raphael 'kena' Poss <[email protected]>

craig · 2018-11-29T17:43:56Z

Build succeeded

GitHub CI (Cockroach)

knz added 3 commits November 16, 2018 14:50

scripts: add missing names to the release notes script

431b2d8

Release note: None

scripts: update title capitalization in rn script tests

942b0f5

Release note: None

knz requested review from couchand, jseldess and a team November 16, 2018 14:07

jseldess approved these changes Nov 16, 2018

View reviewed changes

knz added 2 commits November 16, 2018 15:39

scripts: add copy/paste include in docker image section

cf73c7e

Release note: None

scripts: add an option --one-line to the release note script

26d1d9d

This new option unwraps every release note in the output document. Release note: None

jseldess reviewed Nov 19, 2018

View reviewed changes

scripts: additional improvements to the release notes script

8cb12f3

... as requested by @jseldess. Release note: None

knz commented Nov 20, 2018

View reviewed changes

craig bot merged commit 8cb12f3 into cockroachdb:master Nov 29, 2018

knz deleted the 20181116-release-notes branch February 14, 2019 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scripts: enhance the release notes #32416

scripts: enhance the release notes #32416

knz commented Nov 16, 2018 •

edited

Loading

cockroach-teamcity commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess left a comment

jseldess left a comment

knz commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess left a comment

knz left a comment

knz commented Nov 29, 2018

craig bot commented Nov 29, 2018

scripts: enhance the release notes #32416

scripts: enhance the release notes #32416

Conversation

knz commented Nov 16, 2018 • edited Loading

cockroach-teamcity commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess left a comment

Choose a reason for hiding this comment

jseldess left a comment

Choose a reason for hiding this comment

knz commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess commented Nov 16, 2018

knz commented Nov 16, 2018

jseldess left a comment

Choose a reason for hiding this comment

knz left a comment

Choose a reason for hiding this comment

knz commented Nov 29, 2018

craig bot commented Nov 29, 2018

Build succeeded

knz commented Nov 16, 2018 •

edited

Loading