Add a server side check for mod_deflate to prevent corrupted files

[ghost]

Hi,

as reported in:

owncloud/client#965

it seems that files are corrupted in the sync client when mod_deflate is enabled at the webserver. Adding a server side check for this could prevent such issues.

[karlitschek]

That's a good idea. Can you open a pull request. The check shoud go into lib/util.php like the other checks. Very straight forward.

[karlitschek]

I check can be done like that: http://stackoverflow.com/questions/1862641/compressing-content-with-php-ob-start-vs-apache-deflate-gzip

But I'm unsure if this is still needed. Wasn't the original problem that WebDAV breaks if mod_deflate and ob_start('gz_handler') is both used. So that the output is compressed two times? Nowadays we no longer do compression with php so mod_deflate should be ok. On my test machine mod_deflate is activated and seems to work fine.

@danimo What do you think?

[schmunk42]

👍 Please add a check and warning message when deflate is on. I had some tar.gz files which were just a few bytes missing at the begin of the file, cost me hours to find out why.

[schmunk42]

What's about adding SetEnv no-gzip to the .htaccess fie?

[schmunk42]

@karlitschek I can confirm this 100%(!). I double checked it with Safari, Firefox, but it does not happen with and wget.

If I use SetEnv no-gzip in the .htaccess file I get the exact file back, checked by its md5.
If I don't enable this option get a slightly smaller file:

$ md5 original-file.tar.gz
f7a88e46fb62689142ebf24b3592cc3e

78991231 20 Mai 17:08 with-gizp.tar.gz
79320952 20 Mai 17:07 no-gzip.tar.gz

$ md5 with-gizp.tar.gz
c222245d374d63f6f5560c4ad95541db

$ md5 no-gzip.tar.gz
f7a88e46fb62689142ebf24b3592cc3e

Verified also with wget:

$ wget --no-check-certificate --header\="Accept-Encoding: gzip" "https://somewhere.de/public.php?service=files&t=XXXXXX&download"
$ wget --no-check-certificate "https://somewhere.de/public.php?service=files&t=XXXXXX&download"
$ md5 *
MD5 (XXXXXX) = c222245d374d63f6f5560c4ad95541db
MD5 (XXXXXX.1) = f7a88e46fb62689142ebf24b3592cc3e

As mentioned above, a fix is SetEnv no-gzip in .htaccess.

Also downloading the file from my local network is more than two times faster without gzip.
This only applies for binary files I think, but anyway - this should be fixed.

[schmunk42]

Here's a patch which looks good to me: zuazo/owncloud-cookbook@adce32a

[MorrisJobke]

@LukasReschke Is this possible with our current config check in core?

[ghost]

@MorrisJobke
The main question is: Is this still unsupported?

The check itself could be added for apache mod-php5 like this:

https://github.com/RealRancor/apps/blob/add_adm_deps/admin_dependencies_chk/settings.php#L26

Unfortunately this won't work on php-cgi/php-fpm

An alternative approach was posted by @schmunk42 above: #4783 (comment)

[ghost]

So did some tests on a Apache instance. It seems that mod_deflate is changing the ETag and appending an -gzip to it:

curl -I -sH 'Accept-encoding: gzip,deflate' http://example.com/

HTTP/1.1 200 OK
Date: Thu, 05 Mar 2015 07:17:00 GMT
Server: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.6 OpenSSL/1.0.1f
Last-Modified: Mon, 04 Aug 2014 13:47:08 GMT
ETag: "b3-4ffcdf99c6f00-gzip"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 157
Content-Type: text/html

I don't know if this is a configuration problem on my side but a test.txt file is not shipped compressed/gzipped:

curl -I -sH 'Accept-encoding: gzip,deflate' --user test:test http://example.com/owncloud/remote.php/webdav/test.txt

HTTP/1.1 200 OK
Date: Thu, 05 Mar 2015 07:28:49 GMT
Server: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.6 OpenSSL/1.0.1f
X-Powered-By: PHP/5.5.9-1ubuntu4.6
Set-Cookie: ocde9a1ce42f=lsqa4mdm9kmihcl0mj5slmt6b7; path=/owncloud; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: Sameorigin
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; frame-src *; img-src *; font-src 'self' data:; media-src *
X-Robots-Tag: none
Set-Cookie: ocde9a1ce42f=0esrku7rn4n79g3h0dtqt0uq91; path=/owncloud; HttpOnly
Content-Length: 237678
Last-Modified: Thu, 05 Mar 2015 07:28:44 GMT
ETag: "54f805acc72c1"
Content-Type: text/plain

[LukasReschke]

I should be able to add a check for that on the browser-side. Let me look into that.

[ghost]

Ref for the doc part: owncloud-archive/documentation#992