-
Notifications
You must be signed in to change notification settings - Fork 272
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Client explicitly asks server to not compress data #1251
Comments
Looks like this session header was introduced (d03dd0f) back in 2013, when adding support for SSL certificate verification, and has since been carried across multiple rewrites of Use of this header was originally copy/pasted from pip (which pip introduced to handle some unexpected behaviour in urllib2). AFAICT from a brief perusal pip now uses the default headers implemented by requests of |
As far as I can tell with some git archaeology, the server configurations in question are those which supply unexpected encodings when I vote that we remove any setting of |
I did not sign that commit, so it might or might not have been me from a different life... |
I lean to the same solution... but I'll still document the alternative though (sechkova already said this but just for emphasis): allow compression only for metadata. We know the metadata is not compressed "on disk" so that should always be safe, even with webservers broken as described in the below comment.
In pip (src/_internal/network/utils.py) the same 'identity' encoding trick with the same reasoning is still used:
This is something I will have to deal with in the pip branch (to make sure metadata does get compressed). |
Another observation is that currently TUF deals with raw response content. IIUC the requests' docs, we would like to deal with raw content for target files but use the requests framework to decode potentially compressed metadata files as shown in the example:
|
|
I'm leaning towards leaving this behaviour as-is for the current codebase, and addressing this in the reference fetcher during our refactor efforts. |
👍 This is fine for the pip use case if we manage to do the network io work -- then I can make sure metadata gets compressed in the pip-specific fetcher. |
This commit tries to deal with two interests: * metadata is highly repetitive and compressible: allowing compression would be good * there may be broken web servers (see https://github.com/pypa/pip/blob/404838abcca467648180b358598c597b74d568c9/src/pip/_internal/download.py#L842) that have problems with compression on already compressed target files We can make things better for that first interest while we have no real data for the second interest -- our current workarounds to avoid compression are based on hearsay, not testing. Now that individual fetchers are possible I suggest we simplify ngclient and allow compression. As an example the pip Fetcher could still use the pip response chunking code with all their workarounds -- pip certainly has better capability to maintain a mountain of workarounds and also has endless amounts of real-world testing compared to python-tuf. Details: * Stop modifying Accept-Encoding (Requests default includes gzip) * Don't use response.raw in RequestsFetcher as there is no need anymore * simple_server.py in tests now supports gzip: this gets used in all tests as the requests default is to ask for "gzip". * Fix issue in test_session_get_timeout(): it's not mocking the error that requests really raises in this case Fixes theupdateframework#1251 Signed-off-by: Jussi Kukkonen <[email protected]>
This commit tries to deal with two interests: * metadata is highly repetitive and compressible: allowing compression would be good * there may be broken web servers (see https://github.com/pypa/pip/blob/404838abcca467648180b358598c597b74d568c9/src/pip/_internal/download.py#L842) that have problems with compression on already compressed target files We can make things better for that first interest while we have no real data for the second interest -- our current workarounds to avoid compression are based on hearsay, not testing. Now that individual fetchers are possible I suggest we simplify ngclient and allow compression. As an example the pip Fetcher could still use the pip response chunking code with all their workarounds -- pip certainly has better capability to maintain a mountain of workarounds and also has endless amounts of real-world testing compared to python-tuf. Details: * Stop modifying Accept-Encoding (Requests default includes gzip) * Don't use response.raw in RequestsFetcher as there is no need anymore * Fix issue in test_session_get_timeout(): it's not mocking the error that requests really raises in this case Fixes theupdateframework#1251 Signed-off-by: Jussi Kukkonen <[email protected]>
This commit tries to deal with two interests: * metadata is highly repetitive and compressible: allowing compression would be good * there may be broken web servers (see https://github.com/pypa/pip/blob/404838abcca467648180b358598c597b74d568c9/src/pip/_internal/download.py#L842) that have problems with compression on already compressed target files We can make things better for that first interest while we have no real data for the second interest -- our current workarounds to avoid compression are based on hearsay, not testing. Now that individual fetchers are possible I suggest we simplify ngclient and allow compression. As an example the pip Fetcher could still use the pip response chunking code with all their workarounds -- pip certainly has better capability to maintain a mountain of workarounds and also has endless amounts of real-world testing compared to python-tuf. Details: * Stop modifying Accept-Encoding (Requests default includes gzip) * Don't use response.raw in RequestsFetcher as there is no need anymore * Fix issue in test_session_get_timeout(): it's not mocking the error that requests really raises in this case Fixes theupdateframework#1251 Signed-off-by: Jussi Kukkonen <[email protected]>
This commit tries to deal with two interests: * metadata is highly repetitive and compressible: allowing compression would be good * there may be broken web servers (see https://github.com/pypa/pip/blob/404838abcca467648180b358598c597b74d568c9/src/pip/_internal/download.py#L842) that have problems with compression on already compressed target files We can make things better for that first interest while we have no real data for the second interest -- our current workarounds to avoid compression are based on hearsay, not testing. Now that individual fetchers are possible I suggest we simplify ngclient and allow compression. As an example the pip Fetcher could still use the pip response chunking code with all their workarounds -- pip certainly has better capability to maintain a mountain of workarounds and also has endless amounts of real-world testing compared to python-tuf. Details: * Stop modifying Accept-Encoding (Requests default includes gzip) * Don't use response.raw in RequestsFetcher as there is no need anymore * Fix issue in test_session_get_timeout(): it's not mocking the error that requests really raises in this case Fixes theupdateframework#1251 Signed-off-by: Jussi Kukkonen <[email protected]>
This commit tries to deal with two interests: * metadata is highly repetitive and compressible: allowing compression would be good * there may be broken web servers (see https://github.com/pypa/pip/blob/404838abcca467648180b358598c597b74d568c9/src/pip/_internal/download.py#L842) that have problems with compression on already compressed target files We can make things better for that first interest while we have no real data for the second interest -- our current workarounds to avoid compression are based on hearsay, not testing. Now that individual fetchers are possible I suggest we simplify ngclient and allow compression. As an example the pip Fetcher could still use the pip response chunking code with all their workarounds -- pip certainly has better capability to maintain a mountain of workarounds and also has endless amounts of real-world testing compared to python-tuf. Details: * Stop modifying Accept-Encoding (Requests default includes gzip) * Don't use response.raw in RequestsFetcher as there is no need anymore * Fix issue in test_session_get_timeout(): it's not mocking the error that requests really raises in this case Fixes theupdateframework#1251 Signed-off-by: Jussi Kukkonen <[email protected]>
This commit tries to deal with two interests: * metadata is highly repetitive and compressible: allowing compression would be good * there may be broken web servers (see https://github.com/pypa/pip/blob/404838abcca467648180b358598c597b74d568c9/src/pip/_internal/download.py#L842) that have problems with compression on already compressed target files We can make things better for that first interest while we have no real data for the second interest -- our current workarounds to avoid compression are based on hearsay, not testing. Now that individual fetchers are possible I suggest we simplify ngclient and allow compression. As an example the pip Fetcher could still use the pip response chunking code with all their workarounds -- pip certainly has better capability to maintain a mountain of workarounds and also has endless amounts of real-world testing compared to python-tuf. Details: * Stop modifying Accept-Encoding (Requests default includes gzip) * Don't use response.raw in RequestsFetcher as there is no need: This was a workaround for false "Content-encoding: gzip" inserted by a broken server -- and the workaround was only possible because we knew we never asked for compression * Fix issue in test_session_get_timeout(): it's not mocking the error that requests really raises in this case Fixes theupdateframework#1251 Signed-off-by: Jussi Kukkonen <[email protected]>
Description of issue or feature request:
After the acceptance of TAP10 the framework no longer handles compressed metadata. Instead,
However, the implementation explicitly asks server to not compress data to avoid potential issues with some server configurations.
Two questions coming from this observation:
Current behavior:
Client explicitly asks server to not compress data
Expected behavior:
Allow handling of compressed metadata at the presentation layer
The text was updated successfully, but these errors were encountered: