Build requirements from setup.py

[tysonclugg]

This PR reads abstract requirements from the install_requires section of setup.py, keeping things nice and DRY.

Previous behaviour is maintained - if requirements.in is present, it will be used by default. If not, we use the new behaviour and read from setup.py.

[dirn]

you should

This is an opinion not everyone would agree with.

[tysonclugg]

@dirn I read the post you reference about an hour ago while looking for a suitable URL for the link in question. I don't think it says we should or should not use setup.py, I think the point was more that you shouldn't do stupid stuff like install_requires=list(open('requirements.txt')). Did I miss something?

[dirn]

The gist is that setup.py is for libraries, requirements.txt is for applications. When installing a library

pip install pip-tools

setup.py is executed. Any requirements.txt (or Pipfile.lock since that's how I found this) would be ignored by Pip.

[tysonclugg]

@dirn I re-read the post. Nowhere does it say that setup.py should not be used with applications. The closest statement I could find was this:

You’ve read this far and maybe you’ve said, ok I know that setup.py is designed for redistributable things and that requirements.txt is designed for non-redistributable things but I already have something that reads a requirements.txt and fills out my install_requires=[...] so why should I care?

It's difficult to know the authors intent, but the statement does not preclude the use of setup.py by applications. I could even argue that the implied link given by the statement "I already have something that reads a requirements.txt and fills out my install_requires=[...]" suggests that applications should have a setup.py. But that would be reading too much into what was written - perhaps @dstufft can weigh in and tell us what was intended?

[dirn]

The two sections immediately before the quote you referenced are Python Libraries and Python Applications. The former covers setup.py and the latter covers requirements.txt. This is where I see the distinction made.

We have, however, digressed from my original point a bit -- probably because I mistakenly replied out of context -- but my objection was to the use of "(you should)." Whether or not Donald was saying that you should or shouldn't use setup.py with your applications[1], I believe his main point was that they serve distinct purposes. I use requirements.in for all of my applications (which have no setup.py) and I have no need for requirements.txt for my libraries.

Perhaps you can explain your use case for having both setup.py and requirements.txt and why you think that's the way people should always do it. If you're installing your application with one of

$ python setup.py install
$ python -m pip install .

the requirements.txt file is ignored. If, on the other hand, you're installing your application with

$ python -m pip install -r requirements.txt

setup.py serves no purpose other than to generate requirements.txt, something you could have already done with a requirements.in file.

[1] Donald later references a sample requirements.txt file for when your library becomes your application. I'd argue that this further supports my interpretation that setup.py is for libraries and not for applications.

[dstufft]

When I wrote that post I struggled (and I still struggle) with trying to accurately describe, in my mind, the line between when you should use setup.py and when you shouldn't. It is not as strict as libraries versus applications (for instance, DevPI is an application but it is distributed on PyPI and is pip installable and uses a setup.py). In reality the difference is more like "thing you will distribute via Python packaging channels to other people" versus "thing you will not". with Pipfile and pip-compile which add a concept of differentiating a fully locked down requirements.txt/Pipfile.lock from the input to that, you end up with three distinctions:

• Thing you will distribute via Python packaging channels.
• Thing that has some semi abstract, semi concrete dependencies that you wish to distribute via non Python packaging channels.
• Thing that has concrete, locked down dependencies, typically resulting from the input of the semi-abstract.

A case where an application may have a setup.py, a requirments.in, and a requirements.txt is say something like Warehouse if we pretended that Warehouse was going to function as an application other people could install on their own infrastructure. In that case you'd have:

• setup.py would list the abstract dependencies that Warehouse actually requires to function.
• requirements.in / Pipfile would reference . through some construct (which would pull in itself, plus all of the abstract dependencies from setup.py as well as any additional things (like maybe gunicorn).
• requirements.txt / Pipfile.lock would be the deployment artifact that the production instance of Warehouse would use to ensure reproducibility.

I'm not sure if that all makes sense, but I don't think that every thing should have a setup.py but I don't think the line are as clear cut as libraries vs applications.

[tysonclugg]

Having abstract requirements in setup.py is encouraged, and having concrete requirements in requirements.txt is also encouraged. This patch maintains use of requirements.in by default over reading from setup.py, the only point of contention (at this early stage) seems to be the inclusion of 2 words and an associated hyperlink in the README file.

I'll admit that this point of contention has been a bug-bear of mine since about 2008. I've never liked how Django messes with sys.path and seems to have an aversion to Python packaging. I think I found why Django does this, it has to do with one of the original Django authors hating on Python packaging back in 2007. Being an influential person, that attitude is still with us across the majority of the Python community nearly 10 years later. And since the internet is an echo chamber, that attitude appears countless times by many individuals without much thought, such as has been demonstrated today.

Frankly, the "my project doesn't need a setup.py" drivel has to stop. Writing a basic setup.py is easy, I've not heard a good reason to avoid doing so.

Some lines from the Zen of Python are relevant here:

Special cases aren't special enough to break the rules.
There should be one-- and preferably only one --obvious way to do it.
Now is better than never.

Applications are not special enough that we should advise that they omit having a setup.py.

The canonical way to define dependencies in Python is via the install_requires argument to setup() in setup.py.

I'm taking a stand on this issue now - the Python packaging ecosystem has come a long way in the last 10 years, it's time to re-evaluate the status quo. Stop referencing any material written on the matter published more than a few months ago - it's likely based on ancient hearsay. The article presented by @dirn as a counter to my comment in the README was written in July 2013 - it's ancient.

[davidovich]

I am also not certain if we need to mandate (or absolutely recommend) the use of setup.py.

requirements.in are sometimes an additionnal set of requirements related to the environment, but not necessarily needed by the package build as described by setup.py. It's not one or the other. @dstufft describes it well: if you need to install a webserver, you will not put that into the setup.py, as setup.py should describe what is needed by the package (be it a library or an application). Put in other words, I see the install_requires param as describing all the imports that will be made directly by running/using the package.

I see this PR more helpful for libraries which typically do not use the requirements.in. It would help in reproducibility in build systems, but then again, this has drawbacks: it would also mask some errors that users of the library will have when using it unpinned (using the pinned requirements.txt for builds will isolate the more general open version usage, which can make subtle variations in sub dependencies on each install). Id rather catch a sub dependency breaking this library than let it be discovered by the user.

[dirn]

Frankly, the "my project doesn't need a setup.py" drivel has to stop. Writing a basic setup.py is easy, I've not heard a good reason to avoid doing so.

What do I gain from shipping around an executable file whose sole purpose is to serve as the basis for the contents of a static requirements.txt? For that purpose, the requirements.in meta file feels like a much saner approach.

Special cases aren't special enough to break the rules.

This is the nature of this entire conversation. Is the rule "everything needs a setup.py" or is it "everything that get distributed through Python packaging channels needs a setup.py and things that don't don't"? I believe it's the latter.

There should be one-- and preferably only one --obvious way to do it.

Again, this calls into question what "it" is here. If it's "everything," then setup.py would have to serve as the TOOWTDI. If, however, it's "everything that doesn't get distributed through Python packaging channels," then setup.py isn't necessarily the TOOWTDI. I'd say that it isn't since the file is never used by the end system/users. setup.py encompasses a lot of functionality and information that would be ignored entirely. To me that feels like trying to make setup.py a one-size-fits-all solution.

I try to write no more code than is necessary. setup.py feels unnecessary here.

The article presented by @dirn as a counter to my comment in the README was written in July 2013 - it's ancient.

I'd argue that the way to do things "since about 2008" may be even less relevant, seeing as "the Python packaging ecosystem has come a long way in the last 10 years."

[dirn]

Here's an example of a situation where I think setup.py is inappropriate:

Ansible modules rely on many packages available on PyPI. When I ship around Ansible playbooks, I will include a requirements.txt that contains things like Ansible itself, pyrax, boto, docker-py, etc. These may all be abstract requirements, but everyone should run the deployment (or whatever else the playbooks do) using the same versions. requirements.txt will contain the concrete requirements. To track the abstract requirements, I could use a setup.py, but seeing as this is neither a library nor an application (really there's almost no executable Python code to found anywhere), introducing Python code feels out of place. This is why I use a requirements.in here.

I prefer the rule

Use requirements.in when you need a requirements.txt and setup.py when you don't.

to

Use setup.py when distributing Python code and requirements.in when you're shipping something else.

The first one feels better to me because, at the end of the day, both scenarios actually require asking "do I need a requirements.txt?" But the first scenario ends there. I have all the information I need to know what file to create. Depending on the answer to the question, though, the second scenario could also require asking "am I distributing Python code or something else?"

Two possible outcomes are easier for me to keep track of than three.

Now I understand that this patch would keep requirements.in as the default and my life will continue on unaffected, but if setup.py is TOOWTDI, shouldn't that become the default? Shouldn't I, as someone doing things in a non encouraged way, have to do more work to get my solution to work?

[tysonclugg]

To track the abstract requirements, I could use a setup.py, but seeing as this is neither a library nor an application (really there's almost no executable Python code to found anywhere), introducing Python code feels out of place. This is why I use a requirements.in here.

I agree, this seems like a valid reason not to use setup.py - you describe a situation where you're not distributing any Python modules.

I think I draw the line at the point where you expect others to import your modules, or you import your own modules. This is the point where namespaces come into effect. Simple scripts such as this serve as a prime example of where a namespsace isn't required, and hence there is no need for setup.py.

I don't consider public vs private distribution of packages to be relevant here, not all packages are uploaded to PyPi. Having installable requirements such as -e git+https://github.com/nvie/pip-tools.git@master#egg=pip-tools (or a private repo URL) is useful, setup.py makes this work.

Would changing those 2 words from "you should" to
"you should if you are distributing a python module" help to settle this? Where would you draw the line and suggest people should use setup.py?

[tysonclugg]

Clearing up my previous comment, I feel the difference between a python script and a python module is that one isn't expected to be imported, the other is.

The whole point of setup.py is to define a distribution, which is a collection of one or more namespaced packages. Once defined, tools such as wheel can build a package which can be distributed in a separate step (eg: using twine). The other things defined in setup.py are things like declaring which versions of Python are supported, how to run tests (with python setup.py test), what the version is, etc. These are all useful in their own right - defining your distribution (including its requirements) is useful, even if a package is not going to be distributed to the general public. As stated before, if you're not using namespacing, you don't need setup.py, and requirements.in serves as a useful proxy for install_requires.

[dirn]

Would changing those 2 words from "you should" to "you should if you are distributing a python module" help to settle this?

At the end of the day, the decision is Vincent's to make. I just didn't agree with what I thought was an overly general recommendation, especially sincerequirements.txt is going to be ignored when you install through setup.py.

[tysonclugg]

I've pushed a new commit, with a significantly toned down version of the offending line in the README. Regarding not using setup.py, it now says "it's easy to write one" instead of "you should", which removes the authoritative tone entirely.

Is it time for @nvie to review? I'm happy to squash the commits in this or a separate branch if that's what is preferred.

[Groxx]

I'll just chime in with generally disagreeing with recommending setup.py. If it's the blanket recommendation, for applications it somewhat suggests "build requirements from installable-thing X, but don't install installable-thing X".
That's sorta like recommending using a loose requirements.txt to build requirements.real.txt and convincing people to install .real.txt instead. Seems prone to accidents.

Plus, for setup.py, reading requirements.in as the install_requires is simple and flexible, while signaling "this project uses pip-compile". It also matches all other requirements-*.in patterns, where using setup.py does not, unless you define complex / possibly-contradictory install extras.

With .in files, it's trivial to define disjoint requirement sets, which can sometimes be useful (for e.g. generating docs, linting with mypy which may require a different version of python altogether, different pip versions via tox, etc). They may not even be capable of executing your code, which doesn't blend well with using setup.py.

[Groxx]

Oh. I do like defaulting to setup.py if there's no requirements.in file - that seems quite convenient for library writers who are already using it, which is great. There's a bit of a conflict when you later define requirements.in and it changes default behavior, but that doesn't seem too bad. So +1 from me for that part of this request!

[tysonclugg]

@Groxx Given you commented after the second commit was pushed, I'm a little confused if you're +1 or -1 given that the recommendation has been removed. Can you please clarify? It looks like you object to the PR as it currently stands...

[Groxx]

Changes to the code: I'm not familiar enough to have an opinion on the actual implementation. The "fall back to setup.py" as a whole sounds convenient though, I like it. Automatically does the right thing for many library projects, which is great.

Changes to the documentation, which prioritize setup.py: yes, somewhat opposed. It's a relatively significant high-level recommendation change, and I'm not convinced it's for an improvement. pip-compile is mostly for baking requirements.txt, which is most-important for applications in a production environment, not libraries (whose requirements.in or .txt will never be seen by the vast majority of users).

[tysonclugg]

@Groxx How do you feel the documentation prioritises setup.py? Can you suggest what I can change to resolve that?

Specifically, are you suggesting I swap the order of the two sub-sections within the Example usage for pip-compile section of the README?

[rpkilby]

I've only read parts of the conversation and skimmed over the rest (there are several words). Sorry if I'm missing it, but what's the actual use case for this PR? When would you need both an install_requires section in setup.py and a requirements.txt? What's the end user's usage supposed to look like?

$ pip install some-package
$ pip install -r path/to/site-packages/some-package/requirements.txt   # ???

[anthrotype]

What'is the status of this? I would like to be able to use the install_requires in my existing setup.py to generate the requirements.txt, instead of having to have a separate requirements.in for pip-tools that duplicate those.
Thanks.

[tysonclugg]

@anthrotype: It seems the conversation about this PR devolved into philosophical discussion on the merits of setup.py vs requirements.in, and when either is appropriate.

Some concern was raised about my proposed changes to the documentation describing the new behaviour, and I made changes in accordance with the concerns raised. However, the detractors against the doc changes (@Groxx, @dirn, @davidovich) have not responded positively to the updated documentation. They haven't responded firmly against either (unless @Groxx eventually makes a response to my question about further changes to the docs), so I'd have to assume the updated docs are probably OK. If not, the detractors better speak up - we can assume consent through silence given they've had over 2 months to respond.

@rpkilby asked a fundamental question:

Sorry if I'm missing it, but what's the actual use case for this PR?

I've decided to leave the answer to this question to people such as yourself. The reality is that this PR isn't likely to be accepted unless @nvie agrees with the changes, and Vincent hasn't made comment yet. Perhaps you can describe why this PR would be useful for yourself, it would clarify that there is indeed a use case for the proposed changes.

[rpkilby]

tl;dr - I do not understand what problem this PR solves. In glossing over the thread, I have not seen an actual use case being discussed. @tysonclugg - can you provide an example of why this is beneficial to you?

---

I've decided to leave the answer to this question to people such as yourself.

I'm confused by this stance. Why would you propose changes without giving a reason as to why the changes are beneficial? Providing this reason seems like the best thing that can be done to advance interest in the PR.

It seems the conversation about this PR devolved into philosophical discussion on the merits of setup.py vs requirements.in, and when either is appropriate.

The debate isn't just philosophical - it's about the practical use case of this PR. As I stated in my comment, I'm confused as to what this PR is trying to accomplish. Having a setup.py module allows a package to be installable. Dependencies are managed through the install_requires attribute. Why would you distribute a separate requirements.txt that duplicates dependency management? Is it expected that users of your package run both:

$ pip install some-package
$ pip install -r path/to/site-packages/some-package/requirements.txt   # ???

Perhaps you can describe why this PR would be useful for yourself, it would clarify that there is indeed a use case for the proposed changes.

Not sure if you're directing this at me, but I plainly stated that I don't understand what problem this PR solves. How would I be able to describe how this PR is useful to me? @tysonclugg - again, can you explain why this PR is useful to yourself?

[anthrotype]

Why would you distribute a separate requirements.txt that duplicates dependency management?

I don't think @tysonclugg was suggesting to distribute the requirements.txt to users.
At least, that's not what I'd want to do. My use case is a library which has some install requirements specified in its setup.py; I also want a requirements.txt with pinned-down versions of all the install_requires (e.g. to be used by tox to run the tests, not by the users who would just pip install my library).
I'd like to keep the requirements.txt up-to-date using pip-compile, without having to have a separate requirements.in file that is just a duplicate of what is in my setup.py. I guess I could also read the content of requirements.in into the setup.py's install_requires, but it feels like a hack.

[majuscule]

@rpkilby in short:

• setup.py hopes to be friendly and optimistic to your users, accepting patch upgrades from dependencies and trying to avoid conflicts with other packages that may be installed.
• requirements.txt is a guaranteed working set of packages and versions.

This abstract/concrete dependency model is the use case that this PR would resolve, and is also discussed in the related issue #331.

I would very much like to use pip-tools, but this is a blocking issue for me as the suggested workaround (mentioned in that same issue) no longer functions.

@davidovich would you reconsider merging the PR if the functionality is only enabled explicitly through a --setuptools flag or similar?

[davidovich]

I think this has waited enough, and I am in favor of merging. Let's put this in the 1.9 release (there are two or three bug fixes I would like to land before incorporating features.

(in retrospect, I was never against this ;-)) I think we have done enough bikeshedding.

Caveat: The only thing missing now is tests. We must strive to augment coverage as this project has shared maintainership.

[majuscule]

I have implemented a basic test for this feature as requested (#481). Please let me know if there are any other cases that you would like covered.

@tysonclugg please let me know if you would prefer to do this yourself, I'm just excited to see your work merged; my branch is built directly on yours.

[davidovich]

Thanks to all. This work was merged in #418, closing.