RLS 0.21.1

[TomAugspurger]

closes #18530

---

Release Checklist

This template can be copied into the RLS issue for a release and checked off as you go.

Pre-Release

• Open a pre-release issue and mention related parties. (A few weeks before release) with this template.
• Clear the bug queue
• Run an ASV: `cd asv_bench; asv continuous -f 1.1 $(git describe --abbrev=0)..HEAD``
• Review deprecations from previous releases here
• Review experimental features from previous releases here
• Clean up release.rst (contributors and stats will be done later)
• Clean up whatsnew.rst
• Run a full doc build and make sure everything is clean

  ./make.py clean; ./make.py; ./make.py zip_html;  ./make.py latex_forced
  

  • Check the output, check whatsnew and release.rst for formatting errors.
• run build_dist.sh, uninstall cython and make a test install of the tarball to ensure
  cython is not required for installation from tarball/pypi (now also part of travis scripts, but check).
• Ensure you have SSH access to pandas.pydata.org (@TomAugspurger or Andy Terrel should be able to get you an account).
• Ensure you have upload rights to PyPI (Wes or Jeff)
• Ensure you have commit rights on github.com/macPython/pandas-wheels/ and github.com/conda-forge/pandas-feedstock

The Release

Final Pre-Release

To be done just before tagging the commit. This is optional for release-candidates.

• Call a hold on master in the pre-release issue before you start.
• Finalize the documentation by running the
  • Run scripts/announce.py and add to release.rst
  • Update the release date in release.rst and whatsnew/<version>.rst
• Commit the doc changes

Tag the Release

• Create an empty commit: git commit --allow-empty -m 'RLS: v0.20.0'
  • For Release-Candidates, it's -m 'RLS: v0.20.0.rc2'
• Create the tag: git tag -a v0.20.0 -m "Version 0.20.0"

Test the Release

Some local sanity tests before pushing, making the release official

• Build the release
  • Conda:

# build a conda package
conda build ./conda.recipe --numpy 1.11 --python 2.7 -q --output

# fresh conda env
conda create -n pandas_0.18.2_test python=2.7 pandas nose

# build and install into that env
conda install -n pandas_0.18.2_test pandas --use-local

# switch to that env & test
source activate pandas_0.18.2_test
import pandas as pd
pd.test()

• Pip

# build the wheel
python setup.py bdist wheel

# activate another environment
workon pandas-test
pip uninstall -y pandas
pip install <path/to/wheel>
python -c "import pandas; pandas.test()"

• Build the docs. Check the version number and release notes

Push the Release

This is where things become final. No going back once the tagged commit is pushed

• Push the release
  • Major release: git push upstream master --follow-tags
  • Maintenance: git push upstream 0.20.x --follow-tags

Build Source Distribution

• python setup.py sdist --formats=gztar (TODO: update build_dist.sh to handle maintenance branches)

Release on Github

• Draft a new release here
• If this is an RC, check the box saying it's a pre-release
• Upload the pandas-<version>.tar.gz from the previous step as a "binary"
• Generate the SHA256 for the release with openssl dgst -sha256 dist/pandas-<version>.tar.gz
  We use the SHA from our .tar.gz since the one auto-generated by Github isn't stable if the release page
  is later modified.

Build Binary Distributions

Windows wheels are built automatically by Christoph Gohlke.

• Open a PR for Mac and Linux wheels to update the BUILD_COMMIT here
• Final Release Only Open a PR for the pandas-feedstock
  • Update the version
  • Update the SHA256
  • Set the build number 5to 0

Download Binary Distributions

Conda-Forge is handled automatically. We handle wheels.

• Windows wheels from here
• Mac / Linux wheels with python scripts/fetch_wheels.py, from here

Upload to PyPI

Upload the source and wheels simultaneously. Uploading just the source may break some users'
workflows, if they trigger an update but don't have a C-complier

• Assuming you've downloaded everything to dist: twine upload dist/*
• You may want to hide the release until the docs are up and announcement is sent out

Build and Upload the Docs

• Build and upload

cd docs;
./make.py clean; ./make.py; ./make.py zip_html;  ./make.py latex_forced
python make.py upload 0.20.2

Update the Website

• clone the repo: https://github.com/pandas-dev/pandas-website.git
• major only: Move latest.rst to previous.rst
• Update latest.rst
• Update releases.json and pre_release.json (if RC, add to pre_release.json, else make it blank)
• Update _themes/pydata/layout.html to have the most recent minor release of each major version
• commit and push
• build: python make.py html
• upload: python upload.py # TODO: verify this works
• Verify it updated (maybe in an incognito browser to avoid caching)

Announce

• Make sure everything is in place
  • pypi binaries for windows, mac wheels, source
  • docs built and published
  • website updated
• Send an ANN to the mailing list. include links to both release notes and whatsnew.
• send announcements to [email protected], [email protected], [email protected], [email protected], [email protected]

** Final Version**

• 0.20.3
• 0.20.2
• 0.20.1
• 0.19.2
• 0.19.1
• 0.19.0
• 0.18.1
• 0.18.0
• 0.17.1
• 0.17.0
• 0.16.2
• 0.16.1
• 0.16.0
• 0.15.2
• 0.15.1
• 0.15.0
• 0.14.1
• 0.14.0
• 0.13.1
• 0.13.0
• 0.11.0

** Release Candidates **

• 0.20.0rc1
• 0.18rc2
• 0.18rc1
• 0.17rc2
• 0.17rc1
• 0.16rc1
• 0.15rc1
• 0.14rc1
• 0.13rc1

Start the next release cycle

Only after a major release

When finishing a major release we have a few extra steps to ensure that
the development version is always ahead of the backports version.

• Create a backports branch off the release commmit, e.g. git checkout -b 0.20.x master
• Push that to upstream: git push -u upstream 0.20.x
• Switch back to master and make a new commit to start the next cycle
  • git checkout master
  • git commit --allow-empty -m 'DEV: Start 0.21 cycle'
  • git tag -a v0.21.0.dev0 -m 'Version 0.21.0 start'
  • git push upstream master --follow-tags
• Ensure that a whatsnew file is available for the next major and minor releases

Finish

• Close the Milestone on github
• Close the RLS issue

Miscellany I haven't gotten to yet.

Update documentation

• Get commits since last release:
  git log v0.18.1.. --format='%an#%s' | grep -v Merge > commits
• Include a summary by contributor in release notes (release.rst):
  cat commits | gawk -F '#' '{ print "- " $1 }' | sort | uniq
• Run release_stats.sh to get number of contributors and commits.
• In release notes/whatsnew, update release date.
• Update release notes (release.rst with highlights of whatsnew).

[jorisvandenbossche]

Do we want to include any / all of the items currently in 0.21.1.txt?

I would already include those things that are labeled for 0.21.1. We already had some regression fixed, so IMO it would be a bit stupid not to include this.

[jreback]

yes this should include amy backports

[jreback]

let’s wait till the meeting to discuss

no really pressing PRs

[topper-123]

It there any chance someone could take a look at #18051 for 0.21.1?

The issue is that ATM, if you are not absolutely sure that all values in a categorical series are not nan, you cannot use .unique and .nunique with series of categoricals, (or, you can wrap them in a try/exceptstatement, though is bit ugly and easy to forget the need of).

[shoyer]

Let's not forget:

• Deprecate automatically registering matplotlib units (partial revert of 0.21.0) #18301: Deprecate automatically registering matplotlib units (partial revert of 0.21.0)

[jorisvandenbossche]

#18301: Deprecate automatically registering matplotlib units (partial revert of 0.21.0)

Yes, given the continuing stream of bug reports about that one, it would be nice to get a fix out as soon as possible. Although the sum needs some more time to discuss ...

[jorisvandenbossche]

So we should plan for the 0.21.1 release. @TomAugspurger has some time the coming days (and I am also available to help).

Regardless of the following questions, we can already start backporting and fixing the remaining issues https://github.com/pandas-dev/pandas/milestone/52.

But the questions we need to answer:

• Do we release "0.21.1" (the first thing we release) as it is now (once everything tagged 0.21.1 is merged / backported), or do we include the sum change? (assuming we are going to change it, although this is still subject to the result of the discussion)

and depending on that we need to discuss whether we call that 0.21.1 or 0.22.0, but let's first decide on the above question (again, which is tied to the result of the discussion, so the question is maybe more: do we wait on that, or we think we are ready to decide on it now).

[jreback]

yeah the remaining issues I have only tagged if they are basically ready to merge. So I would say we could release at any time for 0.21.1 (obivously need to actually backport things :>) and prob some small doc cleanups in whatsnew. This should be 0.21.1. If/when we change the whole .sum() this would need to be a major release, and sit in master for a bit.

[jorisvandenbossche]

If/when we change the whole .sum() this would need to be a major release, and sit in master for a bit.

IMO we shouldn't let that sit in master for a bit, but release as quick as possible (and quick as possible might still mean a rc etc).
And, I also think we should not include many other API changes. So even if we call it 0.22.0, I would do this release of the 0.21.x branch. Which means in practice re-purposing the existing 0.22 milestone as 0.23 (because the things we merged there can use some more time in master + because it will be easier to deal with downstream)

[jreback]

why?

we have a bug fix release already teed up. conflating a major change is just not a great idea. We have contentious small sample feedback, why would you exacerbate things?

[jorisvandenbossche]

Because, supposing we go for option 1, we are changing the behaviour again. The longer we wait with doing this, the more people start relying on the intermediate behaviour that will go away, the more packages will need to implement workarounds, etc.

[jreback]

and there is no consensus on option 1.

[jorisvandenbossche]

Then please take the time to answer on the mailing list

[TomAugspurger]

Since #18678 seems to be stuck at the moment, I propose that we clear the 0.21.1 issue list and do a release today.

#18629 should be mergable, #18715 may have to wait for a fastparquet release.

w.r.t doing a release with #18678 still unresolved, I care less about the number of release between now and the fix for it, and more about the number of calendar days. I want as few people as possible using this in-between behavior, and I think the number of days between 0.21.0 and a release with that fixed matters more than the number of releases. Given the matplotlib registry stuff, and the py27 wheels, I'd like to do a release soon, preferably today.

[TomAugspurger]

Alright, everything has been backported. The only remaining issue is the bad wheels on PyPI, which will be fixed by the release.

@jreback, @shoyer, @gfyoung, @chris-b1, any objections to releasing 0.21.1 as is on the 0.21.x branch?

[jreback]

lgtm

[TomAugspurger]

Everything should be in place. I'll tag it tomorrow morning unless something comes up.

[TomAugspurger]

It seems like Christoph is building wheels against CPython 3.7 too: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pandas Unless anyone objects, I'll upload those to PyPI as well.

• Mac and Linux wheels are building: https://travis-ci.org/MacPython/pandas-wheels/builds/315499578
• Conda-Forge is building
• Anaconda defaults has already been updated

I'll send the announce out when the wheels are done.

[jreback]

we have no testing at all against 3.7

[TomAugspurger]

Yep. I figure the two options are 1. We don't upload the wheels, and 3.7 users build from source. Stuff either works or it doesn't. 2. We do upload the wheels, and users get the binary. Stuff either works or it doesn't. We know 0.21.1 builds on 3.7 on Windows, since Christoph was able to build it. By uploading those wheels, we get a bit more exposure to see if there are any issues with his build system on 3.7, which would be good to sort out early. I don't know if uploading wheels for 3.7 is saying that "3.7 is officially supported for pandas 0.21.1". I'd say no.

…

On Tue, Dec 12, 2017 at 1:35 PM, Jeff Reback ***@***.***> wrote: we have no testing at all against 3.7 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18244 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIiq5qg-Q8k-SYwwoSZEJuPGsBlyLks5s_tVxgaJpZM4Qa8_Z> .

[cgohlke]

It seems like Christoph is building wheels against CPython 3.7 too: https://www.lfd.uci.edu/~gohlke/pythonlibs/#pandas Unless anyone objects, I'll upload those to PyPI as well.

Better not. These wheels are built with Python 3.7a3 and Visual Studio 2017. They might not work with later 3.7 releases. I'm just starting to test the scipy stack with Python 3.7 and VS 2017.

[TomAugspurger]

Thanks for clarifying @cgohlke.

[gfyoung]

we have no testing at all against 3.7

@jreback : I think Travis has an image for Python 3.7, so maybe in the future, we could consider?

[jreback]

we don’t use the travis images at all

only conda

[TomAugspurger]

Wheels are done, announcement is sent. Thanks everyone.

[jorisvandenbossche]

@TomAugspurger Thanks a lot for the release!

[jreback]

thanks @TomAugspurger it seems so easy now with your checklist!! hahaha

[TomAugspurger]

Only a few dozen steps :)

…

On Wed, Dec 13, 2017 at 8:09 AM, Jeff Reback ***@***.***> wrote: thanks @TomAugspurger <https://github.com/tomaugspurger> it seems so easy now with your checklist!! hahaha — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18244 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIqsqFrQD74BWy09HNvQH_Fmuh2W8ks5s_9qhgaJpZM4Qa8_Z> .

[wesm]

thanks @TomAugspurger!

[shoyer]

Thanks!

…

On Wed, Dec 13, 2017 at 8:07 AM Wes McKinney ***@***.***> wrote: thanks @TomAugspurger <https://github.com/tomaugspurger>! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18244 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKS1ron-7Vn4xRVMANfxoPZXqbhPfpJks5s__ZdgaJpZM4Qa8_Z> .