[DAV] Download fails for big files: stream_copy_to_stream() throws an error

[derkostka]

Steps to reproduce

1. Try to download a big file on Nextcloud, PHP7.0, 32 Bit arm (> 2 GB)
2. Check downloaded file and log
3. File is empty, log shows the error immediately

Expected behaviour

File with size of 9 GB should be downloaded

Actual behaviour

0 Byte file created, no error message but logged error

Server configuration

Operating system:
Ubuntu 16.04

Web server:
Apache

Database:
MySQL

PHP version:
PHP 7

Nextcloud version: (see Nextcloud admin page)
Nextcloud 11.0.1

Updated from an older Nextcloud/ownCloud or fresh install:
Updated from Owncloud 9 -> Nextcloud 10 -> Nexcloud 11 alpha
Where did you install Nextcloud from:
Git
Signing status:
List of activated apps:
The content of config/config.php:
Are you using external storage, if yes which one: local/smb/sftp/...
local
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

Client configuration

Browser: Chrome

Operating system: MacOS 10.12

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"He91ouATEMBQ7rkEw65e","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at \/media\/data\/www\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php#78","level":3,"time":"October 11, 2016 20:58:34","method":"GET","url":"\/remote.php\/webdav\/Media-Fernsehaufnahmen\/arte_HD\/arte_HD_Der_Tag_der_Kr%C3%A4hen20160615_135900.ts","user":"name","version":"9.2.0.4"}
{"reqId":"DKWW6ajns8HKX8pETQS+","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at \/media\/data\/www\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php#78","level":3,"time":"October 11, 2016 20:58:58","method":"GET","url":"\/remote.php\/webdav\/Media-Fernsehaufnahmen\/arte_HD\/arte_HD_Die_wahre_Macht_des_Vatikan_(2_2)20160207_144900.ts","user":"name","version":"9.2.0.4"}

[derkostka]

Update: According to sabre-io/http#61, the issue is located somewhere else. The file size seems to be invalid (NaN) which leads to that error. However, if i apply that patch, it works - but is not solving the root cause as noted in the comments.

Update2: The file breaks at exact 4 GB (real size is 9,1 GB). So there must be an issue somewhere.

Can anybody please test large file downloads in an environment with 32 bit an php 7 ? thx !

[nickvergessen]

Well you said you use 32 bits, the max int there is 2147483647.
From min to max, the number is 4294967294, which is exactly 4GB.
So I'd say try to switch to 64 bits if you want to have so huge files.

I guess the problem is, that before your cast to int (the intval from the patch), the size is an invalid string. casting to int seems to use the max int, when the value is too high:
See that the following 3 numeric strings all result in the same integer:
https://3v4l.org/OMMWp

var_dump((int) '9223372036854775807');
var_dump((int) '9223372036854775808');
var_dump((int) '92233720368547758081231');

So not sure if there is anything we can do.

[go2sh]

There seems to be a general issue with files bigger than 2GB on 32-bit systems. 🙈

[derkostka]

There has been a discussion about that topic, generally, stating that as an alternative either floats may be used (which are essentially 64 bits, even on the 32bit arm platform) or numeric strings are an option:
https://github.com/fruux/sabre-dav/issues/408

var_dump((int) '9223372036854775807');

var_dump((int) '9223372036854775808');

var_dump((float) '92233720368547758081231');

results in:

int(9223372036854775807) int(9223372036854775807) float(9.2233720368548E+22)

... but as far as i understood, floats are not an option for sabre dav, so it possibly won´t change. Nextcloud shall catch the error and throw a warnings prior downloading instead, i guess.

[nickvergessen]

Can you try, whether #1740 fixes the issue?

[nickvergessen]

And by the way, you should not run any nextcloud 11 alpha version yet. We didn't make any release for that Oo

[derkostka]

No, does not seem to solve it. Download is still 0 Byte. Log:

{"reqId":"cIm0o8yJYIzbld34vhUX","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at /media/data/www/nextcloud/3rdparty/sabre/http/lib/Sapi.php#78","level":3,"time":"October 15, 2016 11:24:15","method":"GET","url":"/remote.php/webdav/Media-Fernsehaufnahmen/arte_HD/arte_HD_Der_Tag_der_Kr%C3%A4hen20160615_135900.ts","user":"me","version":"9.2.0.4"}

-> I will investigate and test further by the end of tomorrow.

[derkostka]

The proposed fix did unfortunately not help. So is there really no chance to download files > 4 GB in an 32 Bit Environment ? I am not into the internals of NC, but i think a download is "streamed" in some kind of chunks, so Filesize should not matter for the download itself, correct ? Is it needed to know the actual filesize (as variable) before downloading ?

[nickvergessen]

Well that seems to exactly be the problem. Before streaming the size is given. But in your case the size is not a number, but a string (because its bigger than the max int) and therefor the error occures.

The problem is, that the log is not really helpful. Let me check if I can find out where this comes from and whether you can add some debug calls for us, that help us finding the real issue.

[nickvergessen]

Can you make the following changes, try an download and then revert the changes again (otherwise your log will become very huge quickly), and then post the log here?

File 3rdparty/sabre/dav/lib/DAV/CorePlugin.php
find (line 795):

$propFind->handle('{DAV:}getcontentlength', [$node, 'getSize']);

before add:

\OC::$server->getLogger()->error('Debugging propFind(): ' . json_encode($node->getSize()));

File apps/dav/lib/Connector/Sabre/Node.php
find (line 201):

return $this->info->getSize();

before add:

\OC::$server->getLogger()->error('Debugging getSize(): ' . json_encode($this->info->getSize()));

File lib/private/Files/FileInfo.php
find:

return isset($this->data['size']) ? $this->data['size'] : 0;

before add:

\OC::$server->getLogger()->error('Debugging FileInfo::getSize(): ' . json_encode($this->data['size']));

[derkostka]

Here are the Results:

File 1, exact Filesize is: 9.085.604.216 Byte
"message":"Debugging propFind(): 9085604216"
"message":"Debugging FileInfo::getSize(): 9085604216"
"message":"Debugging getSize(): 9085604216"
"message":"Debugging FileInfo::getSize(): 9085604216"
--> is correct !

FIle 2, exact Filesize is: 9.585.365.368 Byte:
"message":"Debugging propFind(): 9585365368"
"message":"Debugging FileInfo::getSize(): 9585365368"
"message":"Debugging getSize(): 9585365368"
"message":"Debugging FileInfo::getSize(): 9585365368"
--> is correct !

[nickvergessen]

Can you try whether https://github.com/nextcloud/server/pull/1902/files helps?

[LukasReschke]

Also #1890

[derkostka]

Sorry, neither #1890, nor #1902 solve this

PHP_INT_SIZE is 4, I double checked this, just in case ...

[derkostka]

Just for your information. The same issue occurs on the last stable version of owncloud as well as the daily build (which includes the mentioned patches).

[derkostka]

Perfect @rikmeijer @nickvergessen , this fixed the issue. I will 👍 on the change, as i do not see any side effect. Please vote for this change in order to get the 32 Bit download right.

[staabm]

Filed a php bug https://bugs.php.net/bug.php?id=74395

[nickvergessen]

Oh nice work @rikmeijer

[staabm]

Why do you use 32bit php in the first place?

[derkostka]

Most SBCs such as Raspberry Pi and Odroid devices still rely on 32 Bit. Therefore this change is a real big thing for those.

[hrlmartins]

YEP! Just stumbled in this right now. Got really sad :(

[staabm]

just released https://github.com/fruux/sabre-http/releases/tag/4.2.3 which contains a workaround for the 32bit file size problems

[derkostka]

I just checked on daily, big downloads on 32 Bit work perfectly now (6,8 GB file in my testcase), the md5 sum matches and network speed did not slow down (tested in my local lan).

Thank you - Let consider this fixed with Nextcloud 13 💯 👍

[sturze]

i use Nextcloud 12.0.3 on a 32Bit 16.04 Ubuntu (from a Webhoster) with PHP 7.0.22
i am facing the issue that the zip generated by the Download-Button crashed downloading arround 4GB.

i also tried the sabre-http fix (copied the lib to nextcloud/3rdparty/sabre/http/lib) without success 😢
what did i do wrong?!