Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requests::decompress falsely determines content is not compressed #301

Closed
dustinrue opened this issue Dec 2, 2017 · 1 comment · Fixed by #309
Closed

Requests::decompress falsely determines content is not compressed #301

dustinrue opened this issue Dec 2, 2017 · 1 comment · Fixed by #309
Milestone

Comments

@dustinrue
Copy link

https://github.com/rmccue/Requests/blob/master/library/Requests.php#L823

That function attempts to poke at the data to see if it is actually compressed or not. On a Ubuntu 16.04 system without php-curl installed the fsockopen transport is used instead of curl. When accessing content that is return using encoding deflate or gzip (what I tested) this routine falsely determines that the content is NOT compressed and returns the compressed data leading to failures in dependent software.

My environment consists of the following:

  • WordPress 4.9.1
  • ElasticPress 2.4
  • Ubuntu 16.04 with PHP 7.0 (without php-curl installed)
  • ElasticSearch 5.2.2
  • WP-CLI 1.4

Once configured, using wp elasticpress stats will report that it was unable to contact the ElasticSearch server. This is due to the data coming back compressed (using deflate directly out of ElasticSearch) but being determined as not compressed despite actually being compressed. Not performing the check or installing php-curl on the system will resolve the issue.

@soulseekah
Copy link
Contributor

soulseekah commented Feb 10, 2018

Here's a sample I was able to reproduce: 301.gz from Elasticsearch 5.x

Elasticpress encodes the data using deflate (zlib-encoded) but with a 3 compression level (see elastic/elasticsearch@b6a3952#diff-fc41ed568e8f9fe5f2ffd4a2d410295aR53) resulting in a header with bytes 785e which are disregarded by the Requests::decompress method, which only looks for 789c, 1f8b magic bytes.

We have to support 7801, 785e, 789c and 78da as four valid deflated data markers. https://stackoverflow.com/a/43170354/482864

Patch incoming.

soulseekah added a commit to soulseekah/Requests that referenced this issue Feb 10, 2018
Different compression levels yield a specific second byte in the magic
header for the deflate encoding. All of them can be decoded by the same
functions without any issues. Let them through, as they've been seen in
the wild.

Fixes WordPress#301
@jrfnl jrfnl added this to the 2.0.0 milestone Sep 17, 2021
jrfnl pushed a commit to soulseekah/Requests that referenced this issue Sep 17, 2021
Different compression levels yield a specific second byte in the magic
header for the deflate encoding. All of them can be decoded by the same
functions without any issues. Let them through, as they've been seen in
the wild.

Fixes WordPress#301
jrfnl pushed a commit that referenced this issue Oct 4, 2021
Different compression levels yield a specific second byte in the magic
header for the deflate encoding. All of them can be decoded by the same
functions without any issues. Let them through, as they've been seen in
the wild.

Fixes #301
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants