Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release manager needed for pandas 0.24.3 #166

Open
datapythonista opened this issue Sep 11, 2019 · 33 comments
Open

Release manager needed for pandas 0.24.3 #166

datapythonista opened this issue Sep 11, 2019 · 33 comments
Assignees
@Aya-S
Labels
pandas

Comments

@datapythonista
Copy link
Member

The next version of pandas will be 0.25.2, but we'll release another version of 0.24, the 0.24.3 since pip install pandas may fail in Python 2. The pandas 0.24 series is the last compatible with Python 2, but numpy released a new Python 2 compatible version, that can be automatically installed with pip install pandas, and that is not compatible with pandas 0.24.

The proposed solution is to release a new version 0.24.3, with the only change respect to 0.24.2 that requires numpy < 1.17.

See the original issue here: pandas-dev/pandas#27435

Making the change to setup.py is trivial, but releasing a new pandas version not as much (never did it myself). Most of the work will be in understanding the process, and also on building packages (for pip and for conda). Couple of the steps will require the right permissions (access to the server where we have the documentation, and commit rights on the pandas git repo, but someone with those permissions can take care of those steps.

Those are the notes from the meeting where this was discussed:

  • pip install pandas works when there’s a binary available.
  • pip install --no-binary=pandas pandas numpy==1.16 fails
  • pip install numpy && pip install --no-binary=pandas pandas works (if you have a compiler)
  • We need a PR to update numpy dependency environment marker for just py 27. Backport to 0.24.x
  • Then one of us has to tag 0.24.3 we do the sdsit to github
  • Someone builds binaries on CF(?) wheels.
  • One of use uploads binaries and sdist to pypi.
@TomAugspurger
Copy link

Most of this is semi-automated in https://github.com/pandas-dev/pandas-release (using docker for the local stuff, and making pull requests to GitHub).

@Aya-S
Copy link
Contributor

Aya-S commented Sep 17, 2019

How realistic do you think is it to have someone who is a noob attempt this?

@datapythonista
Copy link
Member Author

How realistic do you think is it to have someone who is a noob attempt this?

If you're motivated to do it, you should totally do it. It'll require a bit of time to understand things (like, building coda packages...), but there is nothing challenging that you can end up saying "I tried, but I wasn't good enough". :)

And of course you won't be alone, any question we're here to help.

@datapythonista
Copy link
Member Author

@Aya-S if you have some time in the next days, I'm happy to do the release together. I'll have to learn as much as you about it, but I'm sure we'll manage. :)

Let me know if you're interested.

@jorisvandenbossche
Copy link

The PR that is needed for the 0.24.3 release (the fix to enable pandas to be installed from source on python 2.7) is pandas-dev/pandas#28511

@Aya-S
Copy link
Contributor

Aya-S commented Sep 20, 2019

@Aya-S if you have some time in the next days, I'm happy to do the release together. I'll have to learn as much as you about it, but I'm sure we'll manage. :)

Let me know if you're interested.

@datapythonista this sounds like a very good deal! I couldn't bring myself to feel confident enough to attempt this at first.
how do we start :D?

@datapythonista
Copy link
Member Author

That's great!

First thing would be to get a branch with the version to release. That is basically branching from 0.24.2 and adding the commit from pandas-dev/pandas#28511 (which is not yet merged).

And after that, create a new release on GitHub, create packages for pip and conda, and update the website/documentation.

But we'll practice a bit before we do it for real. Tom sent us the instructions to do the release: https://github.com/pandas-dev/pandas-release

Do you want to have a look at the document, try to do what's in there, and discuss the questions you have? There are things like pushing to git, uploading the documentation... that you won't be able to do. That's good news, because you can test as much as you want, that you won't be able to break anything. :) Just skip these steps when you find them.

Does this make sense?

@Aya-S
Copy link
Contributor

Aya-S commented Sep 20, 2019

That's great!

First thing would be to get a branch with the version to release. That is basically branching from 0.24.2 and adding the commit from pandas-dev/pandas#28511 (which is not yet merged).

And after that, create a new release on GitHub, create packages for pip and conda, and update the website/documentation.

But we'll practice a bit before we do it for real. Tom sent us the instructions to do the release: https://github.com/pandas-dev/pandas-release

Do you want to have a look at the document, try to do what's in there, and discuss the questions you have? There are things like pushing to git, uploading the documentation... that you won't be able to do. That's good news, because you can test as much as you want, that you won't be able to break anything. :) Just skip these steps when you find them.

Does this make sense?

ok, great. Then I'll start with the document and let you know how things go!

@Aya-S
Copy link
Contributor

Aya-S commented Sep 21, 2019

@datapythonista @TomAugspurger
I checked out the tag for v0.24.2
`commit cb00deb94500205fcb27a33cc1d0df79a9727f8b (HEAD -> trial_0_24_2, tag: v0.24.2)
Author: Joris Van den Bossche [email protected]
Date: Tue Mar 12 22:12:11 2019 +0100

RLS: 0.24.2

`

and was starting to follow the instructions but the makefile I have now, which is the following, doesn't match the instructions, seems like the checkout is not right? any idea what might be causing the discrepancy?
Thanks in advance :D
`tseries: pandas/_libs/lib.pyx pandas/_libs/tslib.pyx pandas/_libs/hashtable.pyx
python setup.py build_ext --inplace

.PHONY : develop build clean clean_pyc tseries doc

clean:
-python setup.py clean

clean_pyc:
-find . -name '*.py[co]' -exec rm {} ;

build: clean_pyc
python setup.py build_ext --inplace

lint-diff:
git diff upstream/master --name-only -- "*.py" | xargs flake8

develop: build
-python setup.py develop

doc:
-rm -rf doc/build doc/source/generated
cd doc;
python make.py clean;
python make.py html
python make.py spellcheck`

@datapythonista
Copy link
Member Author

The Makefile you need to use is not the pandas one, but the one in the repo for the pandas release: https://github.com/pandas-dev/pandas-release/blob/master/Makefile

@Aya-S
Copy link
Contributor

Aya-S commented Sep 22, 2019 via email

@Aya-S
Copy link
Contributor

Aya-S commented Sep 24, 2019

Hello @datapythonista !
I'm at the stage of fetching the PR to merge to the v0.24.2 tag.
i used this command:
git fetch upstream pull/28511/head:test-branch
wondering about

  1. how to test if the fetch succeeded
  2. how to test the codebase for coherency and make sure the PR was properly fetched

@datapythonista
Copy link
Member Author

In git all repos are the same technically speaking, so you can do that with your fork to test is everything works.

But I'd move forward and check more the packaging part, I think that's the tricky part of the process, more than anything else. What do you think?

@Aya-S
Copy link
Contributor

Aya-S commented Sep 24, 2019

In git all repos are the same technically speaking, so you can do that with your fork to test is everything works.

But I'd move forward and check more the packaging part, I think that's the tricky part of the process, more than anything else. What do you think?

I'm currently at: make docker-image docker-doc
it has been on for a while and not returning, not sure if it is supposed to take so long or if there is a certain bottleneck on my side.
Hopefully, it returns by morning so I could proceed :D

@datapythonista
Copy link
Member Author

I don't know much about docker, but I don't think that should take more than some minutes. @TomAugspurger do you know if it's normal that it takes time that step, or what can be the problem?

Thanks @Aya-S for working on this, sounds really promising, I think we'll have a new pandas release soon. :)

@Aya-S
Copy link
Contributor

Aya-S commented Sep 24, 2019

I don't know much about docker, but I don't think that should take more than some minutes. @TomAugspurger do you know if it's normal that it takes time that step, or what can be the problem?

Thanks @Aya-S for working on this, sounds really promising, I think we'll have a new pandas release soon. :)

fingers crossed :D !

@TomAugspurger
Copy link

TomAugspurger commented Sep 24, 2019 via email

@Aya-S
Copy link
Contributor

Aya-S commented Sep 26, 2019

Hey guys!
I went through the steps till make conda test
the output files of both make pip-test make conda-test are attached
conda-test.txt
pip-test-out.txt

understandably, I can't execute make push-tag so skipped it.

created a dummy release on my fork at:https://github.com/Aya-S/pandas/releases/tag/v0.24.3

now on the next step: make conda-forge
I'm getting this error

File "/home/aya/anaconda2/lib/python2.7/site-packages/conda_smithy/configure_feedstock.py", line 13, in
import conda.config
ImportError: No module named config

I'm still to resolve it

@jorisvandenbossche
Copy link

What version of conda and conda-smithy do you have installed?

@Aya-S
Copy link
Contributor

Aya-S commented Sep 26, 2019 via email

@jorisvandenbossche
Copy link

That seems an old release of conda-smithy, they are currently at 3.4.8 (https://github.com/conda-forge/conda-smithy/releases), so I would try to upgrade that, and see if that solves the problem.

@Aya-S
Copy link
Contributor

Aya-S commented Sep 26, 2019

you're right, that's very weird indeed because my first instinct was to do a conda update and I thought i had the latest version. thanks, will make sure I have the latest version

@datapythonista
Copy link
Member Author

If conda env create and conda env update installs the wrong version when using the environment.yml, may be it make sense to force a minimum version in that file? (e.g. conda-smithy>=3.4)

@Aya-S
Copy link
Contributor

Aya-S commented Sep 26, 2019

I didn't create a conda env using the environment.yml file actually
guess it wasn't mentioned anywhere :D.
will try it out

@datapythonista
Copy link
Member Author

Feel free to add it to the instructions. conda env create from the root of the pandas-release repo should create the environment with the dependencies needed for the release.

@Aya-S
Copy link
Contributor

Aya-S commented Sep 26, 2019 via email

@Aya-S
Copy link
Contributor

Aya-S commented Sep 26, 2019 •
edited
Loading

So, after using the conda env based on environment.yml things worked fine (Y)

now i'm at the step that says

Open PRs for each of those

Should I actually do that or leave this for now until we do some further testing on the artefacts of the build?

@datapythonista
Copy link
Member Author

Do you have the packages built then? If that's the case, I think we're ready to start again for real when the PR we want to release is merged. Or am I missing something?

@Aya-S
Copy link
Contributor

Aya-S commented Sep 27, 2019

well, I am definitely missing something.
at which point are the packages built?
I assumed, make conda-forge and make wheels are the rules that build the binaries?
they succeeded yesterday but afterwardds I couldn't find any .whl packages under the "pandas-release" dir.

and surly enough, upon checking the scripts for both rules no building seems to be taking place, as far as I could tell. as it seems, I need to start PRs for the building to start on a remote server? would be great to get some clarification on this :) @TomAugspurger

@TomAugspurger
Copy link

The only package built locally is the sdist. The binaries (wheels and conda packages) are built on CI services. make conda-forge and make wheels create commits and push to a branch, but you would need to open the pull requests on GitHub through your browser once it's time to do this for real.

@Aya-S
Copy link
Contributor

Aya-S commented Sep 29, 2019 •
edited
Loading

OK great, this makes sense. Thanks @TomAugspurger.
I put the built sdist on a dummy release on my fork: https://github.com/Aya-S/pandas/releases/tag/v0.24.3.
Can you confirm that there is nothing more to do at this point until the branch is ready for the actual build?
what I did was that I executed the commands till make wheels
skipping make push-tag and this block:
git checkout -b <TAG>.x
git push upstream <TAG>.x
git checkout master
git commit --allow-empty -m "Start <NEXT_TAG>"
git tag -a v<NEXT_TAG>.dev0 -m 'DEV: Start <NEXT_TAG> cycle'
git push upstream master --follow-tags

@TomAugspurger
Copy link

TomAugspurger commented Oct 2, 2019 via email

@Aya-S
Copy link
Contributor

Aya-S commented Oct 7, 2019

when the branch is ready, I can build the sdist(tar) for someone with the needed permission to upload.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Aya-S Aya-S

Labels
pandas
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jorisvandenbossche @TomAugspurger @Aya-S @datapythonista