Replace all urllib / urllib2 imports and uses with requests
#2852
Labels
Lead: @cclauss
Issues overseen by Chris (Python3 & Dev-ops lead 2019-2021) [managed]
Needs: Breakdown
This big issue needs a checklist or subissues to describe a breakdown of work. [managed]
Needs: Help
Issues, typically substantial ones, that need a dedicated developer to take them on. [managed]
Priority: 3
Issues that we can consider at our leisure. [managed]
Type: Feature Request
Issue describes a feature or enhancement we'd like to implement. [managed]
Comments
hornc
added
Type: Feature Request
|
I'm strongly supportive of switching to requests since it has things like built-in (or easy to add) support for things like retries, timeouts, caching, etc. Having said that, there's so much critical work pending that I'm not sure we can afford to being doing "nice to have" work on existing code that works. Refactoring modules as they get worked on seems fine though. |
Agree. |
|
possibly unused |
|
PyCharm identifies them as unused (amongst other things) |
xayhewalo
added
Lead: @cdrini
|
#2894 shows some code smells which need to be avoided when doing any upgrade of this type. If you still have an |
|
%
|
|
Thanks, @cclauss for guiding. I'm on it! |
This was referenced Aug 4, 2020
This was referenced Aug 13, 2020
|
|
This is going to need SEVERAL PRs before it is finally closed. |
This was referenced Jan 7, 2021
|
grep urlopen **/*.py > files.txt with open("files.txt") as in_file:
files = {line.split(":")[0] for line in in_file}
print("\n".join(sorted(files)))
|
|
I don't think this script works any more |
|
Last released in 2004 https://pypi.org/project/PyZ3950 --> http://www.panix.com/~asl2/software/PyZ3950 --> https://github.com/asl2/PyZ3950 (see the PRs) which is far from Python 3 compatible. |
cclauss
changed the title
requestsrequests
|
I don't think this file Should |
This was referenced Feb 13, 2021
cclauss
added
Lead: @cclauss
mekarpeles
added
the
Needs: Help
|
I went ahead and did a little grepping and such to see that these are all the places urllib is used (excluding infogami). I'm not sure which of these functions we should still try to migrate away from.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
urllib2is old, and a bit clunky -- I've found recently it can't do some of things I needed it to do, and it also causes problem for moving to Python3, and generally seems to make the code more complex.Instead of re-using it, and adding new features that use the older mechanisms, can we consider refactoring to use
requestsinstead?My PR where I stripped out
urllib2andStringIOetc to enableHEADrequests: #2838a similar PR that went with using
requeststo avoid old bugs: #2779By using
requests(rather than adding imports fromsix) we should be getting a better tool for making requests, and reducing code complexity, and the number of imports.Relates to #2308 since a common use of
simplejsonis to parseurllib2responses.requestshas a builtinjson()method so an extra import is not required.Describe the problem that you'd like solved
Proposal & Constraints
Additional context
Stakeholders
The text was updated successfully, but these errors were encountered: