Chunked transfers lead to 0 byte files

[ylangisc]

We hit the annoying 0 byte files issue when using a PUT operation with a chunked transfer encoding to modify an existing 0 byte file. After successfully executing the PUT the server replies with a 204 but the file is still empty. Not sure if this is a NC or sabre/dav issue.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @report.csv "http://ubuntu.local/remote.php/webdav/ylatst.txt" -u admin:admin
*   Trying 192.168.48.135...
* TCP_NODELAY set
* Connected to ubuntu.local (192.168.48.135) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /remote.php/webdav/ylatst.txt HTTP/1.1
> Host: ubuntu.local
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 204 No Content
< Server: nginx/1.13.8
< Date: Mon, 22 Jan 2018 16:41:33 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=xlmQsVcgE%2FcbE8%2FmB8bvsVG3fGrkDGB%2Fg5wHy5sJqrIjfeVotxGf4US%2BnLdGj6xNmzPjH3NzYgm09Qy%2FnkqKQA59LI9qQEmY0mkZZs17wwCvcubybbqiCS83FHYGdvhe; path=/; HttpOnly
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: nc_sameSiteCookielax=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Content-Security-Policy: default-src 'none';
< Set-Cookie: oc4nuj3qa5xi=upmvsv2vtom8dg52tn72r3g4a0; path=/; HttpOnly
< Set-Cookie: cookie_test=test; expires=Mon, 22-Jan-2018 17:41:33 GMT; Max-Age=3600
< OC-FileId: 00000180oc4nuj3qa5xi
< ETag: "46ca66c845120b954836fb1dc6fb4af9"
< OC-ETag: "46ca66c845120b954836fb1dc6fb4af9"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none

I'm using nginx 1.13.8 as a webserver (nginx 1.10.3 doesn't work either). My nginx configuration is quite default. The relevant part:

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        fastcgi_pass      unix:/var/run/php/php7.0-fpm.sock;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
    }

Some observations:

• changing fastcgi_request_buffering off to fastcgi_request_buffering on solves the issue but this is not an option for large streamed files.
• it does happen for files larger than 7kb in my local environment. Smaller files work fine. For servers not being in my local network the files have to be even larger to show the issue. It's much more difficult to replicate the issue in such an environment. Due to bandwidth limitation and latency the server probably has enough time to process the data and a buffer overflow or similar is less likely to happen.

Background: I'm a developer of Mountain Duck which uses the create/update file pattern described here http://sabre.io/dav/0bytes/. According this document nginx in a newer version should work fine. We have several users hitting this issue though.

Steps to reproduce

1. Upload a 0 byte file
2. Try to update the file with chunked transfer encoding (see above for a curl example)

Expected behaviour

Files has updated content and correct size

Actual behaviour

Remote file is not touched at all and you still see 0 bytes

Server configuration

Operating system:
Ubuntu 16

Web server:
nginx

Database:
MariaDB

PHP version:
7.0.22

Nextcloud version: (see Nextcloud admin page)
12.0.4

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh

Where did you install Nextcloud from:
Official download

[ylangisc]

After some more testing I've noticed that it's even worse. It does not matter if you create a 0 byte file prior updating it. Basically all PUT operations with a chunked transfer encoding lead to 0 byte files in low latency and high bandwidth environments for files that are greater than a certain size (see above). In my local environment it's easy to replicate with a simple curl command. E.g.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @testfile.bin "http://ubuntu.local/remote.php/webdav/testfile.bin" -u admin:admin

[Greek64]

I too am having this issue with webdav.
After disabling encryption and sniffing the traffic going to and from the nextcloud webserver (nginx 1.14.0 in my case), I too came to the conclusion, that after a PUT command with chunked transfer encoding, nextcloud responses with 204 no content and creates only 0 byte files.

I noticed this behavior when trying to export my Kodi library via webdav to nextcloud.

I can also provide a sanitized HTTP communication if needed, but I guess that will not help much.

[jospoortvliet]

I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'jos'
> PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic am9zOmpvcw==
> User-Agent: curl/7.62.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 201 Created
< Date: Thu, 03 Jan 2019 11:22:50 GMT
< Server: Apache
< X-Powered-By: PHP/7.3.0
< Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer
< Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly
< Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600
< OC-FileId: 00001151oczswzpgpvdq
< Content-Length: 0
< ETag: "7ef58d6f933a4e3387fdf1418f188772"
< OC-ETag: "7ef58d6f933a4e3387fdf1418f188772"
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 127.0.0.1 left intact

[bohwaz]

This seems to actually be a bug in mod_fcgid used by FPM. It was fixed in version 2.3.10, but version 2.3.9 is still the only one proposed by Debian. I contacted the Debian maintainer and offered him a bounty to see if the new version could be uploaded to Debian.

In the meantime, NextCloud could implement an error message if this bug is detected, see this example:

		// mod_fcgid <= 2.3.9 doesn't handle chunked transfer encoding for PUT requests
		// see https://github.com/kd2org/picodav/issues/6
		if (strstr($_SERVER['HTTP_TRANSFER_ENCODING'] ?? '', 'chunked') && PHP_SAPI == 'fpm-fcgi') {
			// We can't seek here
			// see https://github.com/php/php-src/issues/9441
			$l = strlen(fread($stream, 1));

			if ($l === 0) {
				throw new Exception('This server cannot accept "Transfer-Encoding: chunked" uploads (please upgrade to mod_fcgid >= 2.3.10).', 500);
			}

			// reset stream
			fseek($stream, 0, SEEK_SET);
		}

For sysadmins, the only solution is to either compile your own mod_fcgid or to use mod_php instead.

[bohwaz]

This is not a patch, but an idea, I'll let you implement it in NextCloud.

[szaimen]

I fear I am not the right guy to implement this. However I suppose it would need to get added somewhere here:

server/apps/dav/lib/Connector/Sabre/File.php

Line 588 in 9e73412

private function createFileChunked($data) {

cc @nextcloud/server-backend

[rfc2822]

Problem occurs here too. Nextcloud 27.0.2, nginx/1.18.0 (Ubuntu) over FastCGI, PHP 8.1 (8.1.22-1+ubuntu22.04.1+deb.sury.org+1)

DAVx⁵ unfortunately has to use chunked transfer because it get its input data as stream, so the problem always occurs with DAVx⁵ (but also with curl when using chunked transfer).

[alx-tuilmenau]

Got the same problem with apache 2.4.57, mpm_event, mod_proxy_fcgi and php8.2-fpm on Debian 12. Temporary? solved by adding SetEnv proxy-sendcl to the Apache configuration to tell mod_proxy to buffer the request and always send the Content-Length header. This may cause spooling requests to disk by apache, but there are no more 0 byte files.

[enoch85]

Got the same problem with apache 2.4.57, mpm_event, mod_proxy_fcgi and php8.2-fpm on Debian 12. Temporary? solved by adding SetEnv proxy-sendcl to the Apache configuration to tell mod_proxy to buffer the request and always send the Content-Length header. This may cause spooling requests to disk by apache, but there are no more 0 byte files.

We use the same solution in the Nextcloud VM: https://github.com/nextcloud/vm/blob/master/lets-encrypt/activate-tls.sh#L124-L129

[bcutter]

How can this still be a thing after 7 (!?!) years?

Ran into 0 byte video files when using chunked upload with PhotoSync app.

[Subito]

It seems to me everyone is busy developing new features. There are other issues like this ("Reliable Up/Download" #11138 for example) where I'm stumped that they were not found, fixed, tested and regression-tested since v1. I really hope the team is going to reverse course some day and develop a solid file-syncing solution.