Skip to content
This repository has been archived by the owner on Oct 5, 2020. It is now read-only.

Only include taskcluster.async on Python 3 #82

Closed
wants to merge 1 commit into from
Closed

Only include taskcluster.async on Python 3 #82

wants to merge 1 commit into from

Conversation

edmorley
Copy link
Contributor

@edmorley edmorley commented Aug 7, 2017

Since it doesn't work on Python 2, so might as well be excluded there from the wheel and/or files installed from the sdist.

Copy link
Contributor

@jhford jhford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the objective here, but I don't think we can safely merge this commit. The problem is that the runtime which builds the source tarball has control over whether or not taskcluster.async is included in the package that needs to be runtime version agnostic.

This means that if you build sdist on 2.7 you don't get async but if you build on 3.5+ you do get it. This would result in a 3.5 built source distribution installed into a 2.7 runtime environment having a non-functioning async module installed and a 2.7 built source distribution installed into a 3.5 runtime environment would be missing the async module.

Because of the missing async module, we'd have to limit our building of the source distribution to python3.5+, which would result in continuing to package taskcluster.async for python2.7 anyway, defeating this commit's intention.

As well, the packaging of taskcluster.async doesn't cause python2.7 to generate an invalid source distribution. We also get an exception throw in 2.7 when trying to import taskcluster.async.

It always amazes me that python packaging doesn't have facilities for marking packages/files as being specific to python3 vs python2, since that was a breaking change and all.

Maybe there's a way to black list that module when building the py2 or py3 specific wheels?

Since it doesn't work on Python 2, so might as well be excluded there
from the wheel and/or files installed from the sdist.
@edmorley
Copy link
Contributor Author

edmorley commented Aug 8, 2017

This means that if you build sdist on 2.7 you don't get async but if you build on 3.5+ you do get it.

When generating an sdist, the build step that would invoke setup.py isn't executed at all, so the sdist isn't affected by Python version specific conditionals. (This is the reason why wheels are so good - they've already performed the steps that would otherwise have to be performed on the client at install time.)

To confirm this:

$ make devel
...
$ ./.tox/py36/bin/python setup.py sdist
running sdist
...
$ mv dist/taskcluster-1.3.4.tar.gz dist/py36.tar.gz
$ ./.tox/py27/bin/python setup.py sdist
running sdist
...
$ mv dist/taskcluster-1.3.4.tar.gz dist/py27.tar.gz
$ diff -U 0 <( tar -tf dist/py27.tar.gz | sort ) <( tar -tf dist/py36.tar.gz | sort )
$

Maybe there's a way to black list that module when building the py2 or py3 specific wheels?

There is - this PR :-)
(Compare the separate python 2 and 3 wheels generated after this PR when using the release script updated by #80)

@jhford
Copy link
Contributor

jhford commented Aug 8, 2017

It looks to me like setup.py is being executed as the entry point into the packaging framework when building the sdist package, and that it is giving differing results when executed by 2.7 and 3.6. Is there something I'm overlooking here?

Python 2.7 sdist creation

jhford-macbook:~/taskcluster/taskcluster-client.py $ python2.7 setup.py sdist
running sdist
running egg_info
creating taskcluster.egg-info
writing requirements to taskcluster.egg-info/requires.txt
writing taskcluster.egg-info/PKG-INFO
writing top-level names to taskcluster.egg-info/top_level.txt
writing dependency_links to taskcluster.egg-info/dependency_links.txt
writing manifest file 'taskcluster.egg-info/SOURCES.txt'
reading manifest file 'taskcluster.egg-info/SOURCES.txt'
writing manifest file 'taskcluster.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
creating taskcluster-1.3.4
creating taskcluster-1.3.4/taskcluster
creating taskcluster-1.3.4/taskcluster.egg-info
creating taskcluster-1.3.4/test
copying files to taskcluster-1.3.4...
copying setup.cfg -> taskcluster-1.3.4
copying setup.py -> taskcluster-1.3.4
copying taskcluster/__init__.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/_client_importer.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/auth.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/authevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/awsprovisioner.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/awsprovisionerevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/client.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/exceptions.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/github.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/githubevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/hooks.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/index.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/login.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/notify.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/pulse.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/purgecache.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/purgecacheevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/queue.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/queueevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/scheduler.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/schedulerevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/secrets.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/treeherderevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/utils.py -> taskcluster-1.3.4/taskcluster
copying taskcluster.egg-info/PKG-INFO -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/SOURCES.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/dependency_links.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/not-zip-safe -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/requires.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/top_level.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying test/test_async.py -> taskcluster-1.3.4/test
copying test/test_client.py -> taskcluster-1.3.4/test
copying test/test_utils.py -> taskcluster-1.3.4/test
Writing taskcluster-1.3.4/setup.cfg
creating dist
Creating tar archive
removing 'taskcluster-1.3.4' (and everything under it)

Python 2.7 sdist output listing

jhford-macbook:~/taskcluster/taskcluster-client.py $ tar tfz dist/taskcluster-1.3.4.tar.gz
taskcluster-1.3.4/
taskcluster-1.3.4/PKG-INFO
taskcluster-1.3.4/setup.cfg
taskcluster-1.3.4/setup.py
taskcluster-1.3.4/taskcluster/
taskcluster-1.3.4/taskcluster/__init__.py
taskcluster-1.3.4/taskcluster/_client_importer.py
taskcluster-1.3.4/taskcluster/auth.py
taskcluster-1.3.4/taskcluster/authevents.py
taskcluster-1.3.4/taskcluster/awsprovisioner.py
taskcluster-1.3.4/taskcluster/awsprovisionerevents.py
taskcluster-1.3.4/taskcluster/client.py
taskcluster-1.3.4/taskcluster/exceptions.py
taskcluster-1.3.4/taskcluster/github.py
taskcluster-1.3.4/taskcluster/githubevents.py
taskcluster-1.3.4/taskcluster/hooks.py
taskcluster-1.3.4/taskcluster/index.py
taskcluster-1.3.4/taskcluster/login.py
taskcluster-1.3.4/taskcluster/notify.py
taskcluster-1.3.4/taskcluster/pulse.py
taskcluster-1.3.4/taskcluster/purgecache.py
taskcluster-1.3.4/taskcluster/purgecacheevents.py
taskcluster-1.3.4/taskcluster/queue.py
taskcluster-1.3.4/taskcluster/queueevents.py
taskcluster-1.3.4/taskcluster/scheduler.py
taskcluster-1.3.4/taskcluster/schedulerevents.py
taskcluster-1.3.4/taskcluster/secrets.py
taskcluster-1.3.4/taskcluster/treeherderevents.py
taskcluster-1.3.4/taskcluster/utils.py
taskcluster-1.3.4/taskcluster.egg-info/
taskcluster-1.3.4/taskcluster.egg-info/dependency_links.txt
taskcluster-1.3.4/taskcluster.egg-info/not-zip-safe
taskcluster-1.3.4/taskcluster.egg-info/PKG-INFO
taskcluster-1.3.4/taskcluster.egg-info/requires.txt
taskcluster-1.3.4/taskcluster.egg-info/SOURCES.txt
taskcluster-1.3.4/taskcluster.egg-info/top_level.txt
taskcluster-1.3.4/test/
taskcluster-1.3.4/test/test_async.py
taskcluster-1.3.4/test/test_client.py
taskcluster-1.3.4/test/test_utils.py

Python 3.6 sdist creation

jhford-macbook:~/taskcluster/taskcluster-client.py $ python3.6 setup.py sdist
running sdist
running egg_info
writing taskcluster.egg-info/PKG-INFO
writing dependency_links to taskcluster.egg-info/dependency_links.txt
writing requirements to taskcluster.egg-info/requires.txt
writing top-level names to taskcluster.egg-info/top_level.txt
reading manifest file 'taskcluster.egg-info/SOURCES.txt'
writing manifest file 'taskcluster.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
creating taskcluster-1.3.4
creating taskcluster-1.3.4/taskcluster
creating taskcluster-1.3.4/taskcluster.egg-info
creating taskcluster-1.3.4/taskcluster/async
creating taskcluster-1.3.4/test
copying files to taskcluster-1.3.4...
copying setup.cfg -> taskcluster-1.3.4
copying setup.py -> taskcluster-1.3.4
copying taskcluster/__init__.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/_client_importer.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/auth.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/authevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/awsprovisioner.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/awsprovisionerevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/client.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/exceptions.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/github.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/githubevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/hooks.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/index.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/login.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/notify.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/pulse.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/purgecache.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/purgecacheevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/queue.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/queueevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/scheduler.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/schedulerevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/secrets.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/treeherderevents.py -> taskcluster-1.3.4/taskcluster
copying taskcluster/utils.py -> taskcluster-1.3.4/taskcluster
copying taskcluster.egg-info/PKG-INFO -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/SOURCES.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/dependency_links.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/not-zip-safe -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/requires.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster.egg-info/top_level.txt -> taskcluster-1.3.4/taskcluster.egg-info
copying taskcluster/async/__init__.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/_client_importer.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/asyncclient.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/asyncutils.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/auth.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/authevents.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/awsprovisioner.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/awsprovisionerevents.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/github.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/githubevents.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/hooks.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/index.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/login.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/notify.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/pulse.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/purgecache.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/purgecacheevents.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/queue.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/queueevents.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/scheduler.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/schedulerevents.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/secrets.py -> taskcluster-1.3.4/taskcluster/async
copying taskcluster/async/treeherderevents.py -> taskcluster-1.3.4/taskcluster/async
copying test/test_async.py -> taskcluster-1.3.4/test
copying test/test_client.py -> taskcluster-1.3.4/test
copying test/test_utils.py -> taskcluster-1.3.4/test
Writing taskcluster-1.3.4/setup.cfg
Creating tar archive
removing 'taskcluster-1.3.4' (and everything under it)

Python 3.6 sdist listing

jhford-macbook:~/taskcluster/taskcluster-client.py $ tar tfz dist/taskcluster-1.3.4.tar.gz
taskcluster-1.3.4/
taskcluster-1.3.4/PKG-INFO
taskcluster-1.3.4/setup.cfg
taskcluster-1.3.4/setup.py
taskcluster-1.3.4/taskcluster/
taskcluster-1.3.4/taskcluster/__init__.py
taskcluster-1.3.4/taskcluster/_client_importer.py
taskcluster-1.3.4/taskcluster/async/
taskcluster-1.3.4/taskcluster/async/__init__.py
taskcluster-1.3.4/taskcluster/async/_client_importer.py
taskcluster-1.3.4/taskcluster/async/asyncclient.py
taskcluster-1.3.4/taskcluster/async/asyncutils.py
taskcluster-1.3.4/taskcluster/async/auth.py
taskcluster-1.3.4/taskcluster/async/authevents.py
taskcluster-1.3.4/taskcluster/async/awsprovisioner.py
taskcluster-1.3.4/taskcluster/async/awsprovisionerevents.py
taskcluster-1.3.4/taskcluster/async/github.py
taskcluster-1.3.4/taskcluster/async/githubevents.py
taskcluster-1.3.4/taskcluster/async/hooks.py
taskcluster-1.3.4/taskcluster/async/index.py
taskcluster-1.3.4/taskcluster/async/login.py
taskcluster-1.3.4/taskcluster/async/notify.py
taskcluster-1.3.4/taskcluster/async/pulse.py
taskcluster-1.3.4/taskcluster/async/purgecache.py
taskcluster-1.3.4/taskcluster/async/purgecacheevents.py
taskcluster-1.3.4/taskcluster/async/queue.py
taskcluster-1.3.4/taskcluster/async/queueevents.py
taskcluster-1.3.4/taskcluster/async/scheduler.py
taskcluster-1.3.4/taskcluster/async/schedulerevents.py
taskcluster-1.3.4/taskcluster/async/secrets.py
taskcluster-1.3.4/taskcluster/async/treeherderevents.py
taskcluster-1.3.4/taskcluster/auth.py
taskcluster-1.3.4/taskcluster/authevents.py
taskcluster-1.3.4/taskcluster/awsprovisioner.py
taskcluster-1.3.4/taskcluster/awsprovisionerevents.py
taskcluster-1.3.4/taskcluster/client.py
taskcluster-1.3.4/taskcluster/exceptions.py
taskcluster-1.3.4/taskcluster/github.py
taskcluster-1.3.4/taskcluster/githubevents.py
taskcluster-1.3.4/taskcluster/hooks.py
taskcluster-1.3.4/taskcluster/index.py
taskcluster-1.3.4/taskcluster/login.py
taskcluster-1.3.4/taskcluster/notify.py
taskcluster-1.3.4/taskcluster/pulse.py
taskcluster-1.3.4/taskcluster/purgecache.py
taskcluster-1.3.4/taskcluster/purgecacheevents.py
taskcluster-1.3.4/taskcluster/queue.py
taskcluster-1.3.4/taskcluster/queueevents.py
taskcluster-1.3.4/taskcluster/scheduler.py
taskcluster-1.3.4/taskcluster/schedulerevents.py
taskcluster-1.3.4/taskcluster/secrets.py
taskcluster-1.3.4/taskcluster/treeherderevents.py
taskcluster-1.3.4/taskcluster/utils.py
taskcluster-1.3.4/taskcluster.egg-info/
taskcluster-1.3.4/taskcluster.egg-info/dependency_links.txt
taskcluster-1.3.4/taskcluster.egg-info/not-zip-safe
taskcluster-1.3.4/taskcluster.egg-info/PKG-INFO
taskcluster-1.3.4/taskcluster.egg-info/requires.txt
taskcluster-1.3.4/taskcluster.egg-info/SOURCES.txt
taskcluster-1.3.4/taskcluster.egg-info/top_level.txt
taskcluster-1.3.4/test/
taskcluster-1.3.4/test/test_async.py
taskcluster-1.3.4/test/test_client.py
taskcluster-1.3.4/test/test_utils.py

Git metadata

jhford-macbook:~/taskcluster/taskcluster-client.py $ git branch
  async
  codegen
  generate-wheels
  master
  mohawk
* python27-exclude-async
  query-params
  read-creds
  shell-login
jhford-macbook:~/taskcluster/taskcluster-client.py $ git id
b3287d48c984fe80e4da668673ccfe33fbc04a7c

@jhford
Copy link
Contributor

jhford commented Aug 8, 2017

I think what would be needed is to figure out of the sdist run right now is being invoked because someone asked for setup.py sdist vs indirectly as a part of setup.py bdist_wheel. If we knew why we were running this code, we could correctly ignore the async package when building an sdist on python 2.7 specifically because we're about to do a bdist_wheel

@edmorley
Copy link
Contributor Author

edmorley commented Aug 8, 2017

Eugh it appears my results were due to yet another over-caching bug with setuptools/...
If I remove taskcluster.egg-info/ between builds, then the contents of the py27 and py36 sdists differ.

A solution could be implemented using if sys.argv[-1] == 'bdist_wheel', but that's a bit hacky so probably not worth it.

@edmorley edmorley closed this Aug 8, 2017
@edmorley edmorley deleted the python27-exclude-async branch August 8, 2017 15:48
@edmorley
Copy link
Contributor Author

edmorley commented Aug 8, 2017

due to yet another over-caching bug with setuptools/...

Possibly this one:
pypa/setuptools#211

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants