broken upload on shared folder by URL

[nielsd]

Steps to reproduce

1. create a folder and create a share url
   2 .set the share writable (read + write)
2. open the URL (in another browser)
3. upload a file > 100 MB (or less)

Expected behaviour

The upload should process and success. The file should get listet / available in the current folder.

Actual behaviour

Upload progress bar "stucks" and timer rises against infinite (?) over time or it falls back before complete and starts new (loops).

see log output "expected filesize 78097358 got 23084040" (not in GUI)

This affects all (updated) nc installations from (at least) 15.x and even the 16.0.0.9 (where we had applied the fix to get loaded the "marked.min.js" instad of getting: "Could not find resource core/vendor/marked/marked.min.js to load". So i assume this still affects 16. upstream.

We tried to disable cachiung (redis ( APCu) too.

Uploads from autorized users are working so far, after we compiled PHP with inifile support to support the user.ini from NC.

Server configuration

Operating system:
(Gentoo) Linux 4.x kernel

Web server:
Apache 2.4.39 + mod_php

Database:
SQLite3

PHP version:
PHP 7.1 (incl. inifile option)

Nextcloud version: (see Nextcloud admin page)
15.0.7.0
but even on 16.0.0.9 (with fixed JS load)

Updated from an older Nextcloud/ownCloud or fresh install:
updated multiple times from earlier versions

Where did you install Nextcloud from:
initial with tarball - updates by updater

Signing status:

Signing status

https://cloud.syndicat.com/pub/index.php/settings/integrity/failed

List of activated apps:

App list

Enabled: - accessibility: 1.1.0 - activity: 2.8.2 - cloud_federation_api: 0.1.0 - comments: 1.5.0 - dav: 1.8.1 - encryption: 2.3.0 - federatedfilesharing: 1.5.0 - files: 1.10.0 - files_pdfviewer: 1.4.0 - files_sharing: 1.7.0 - files_texteditor: 2.7.0 - files_versions: 1.8.0 - files_videoplayer: 1.4.0 - firstrunwizard: 2.4.0 - gallery: 18.2.0 - logreader: 2.0.0 - lookup_server_connector: 1.3.0 - nextcloud_announcements: 1.4.0 - notifications: 2.3.0 - oauth2: 1.3.0 - password_policy: 1.5.0 - provisioning_api: 1.5.0 - serverinfo: 1.5.0 - sharebymail: 1.5.0 - support: 1.0.0 - survey_client: 1.3.0 - systemtags: 1.5.0 - theming: 1.6.0 - twofactor_backupcodes: 1.4.1 - updatenotification: 1.5.0 - workflowengine: 1.5.0 Disabled: - admin_audit - audioplayer - bruteforcesettings - federation - files_external - files_trashbin - user_ldap

Nextcloud configuration:

Config report

{ "system": { "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwritewebroot": "\/pub", "dbtype": "sqlite3", "version": "15.0.7.0", "installed": true, "maxZipInputSize": 1047527424, "allowZipDownload": true, "mail_smtpmode": "smtp", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpauth": 1, "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "tls", "log_type": "owncloud", "logfile": "\/var\/log\/cloud\/pub_cloud.log", "loglevel": 3, "forcessl": "true", "versions": "false", "cloud_name": "***REMOVED SENSITIVE VALUE***"", "instanceid": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "theme": "", "default_language": "de", "trusted_domains": [ "cloud.syndicat.com" ], "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***"\/pub", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "trashbin_retention_obligation": "auto", "singleuser": false, "filelocking.enabled": "true", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.local": "\\OC\\Memcache\\APCu", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "dbindex": 0, "password": "***REMOVED SENSITIVE VALUE***", "timeout": 2 }, "updater.release.channel": "production" } }

.user.ini:
upload_max_filesize=2.9G
post_max_size=2.9G
memory_limit=512M
mbstring.func_overload=0
always_populate_raw_post_data=-1
default_charset='UTF-8'
output_buffering=0

Are you using external storage, if yes which one:
no

Are you using encryption:
no

Are you using an external user-backend, if yes which one:
no

Client configuration

Browser:

tested in:
Firefox
Google Chrome

Operating system:
Linux, Windows, Mac

Logs

Web server error log

Web server error log

empty

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"XNQdt-um4LfDBCYaQ8zN0AAAAAs","level":4,"time":"2019-05-09T12:32:11+00:00","remoteAddr":"217.235.176.6","user":"--","app":"webdav","method":"PUT","url":"\/pub\/public.php\/webdav\/portage-20170705.tar.bz2","message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"expected filesize 78097358 got 23084040","Code":0,"Trace":[{"file":"\/xxx\/cloud\/pub\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"\/xxx\/cloud\/pub\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":1096,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["portage-20170705.tar.bz2",null]},{"file":"\/xxx\/cloud\/pub\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php","line":525,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["portage-20170705.tar.bz2",null,null]},{"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https:\/\/yyy\/pub\/public.php\/webdav\/portage-20170705.tar.bz2","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/xxx/cloud\/pub\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPut"],[{"absoluteUrl":"https:\/\/yyy\/pub\/public.php\/webdav\/portage-20170705.tar.bz2","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/xxx\/cloud\/pub\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PUT",[{"absoluteUrl":"https:\/\/yyy\/pub\/public.php\/webdav\/portage-20170705.tar.bz2","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"\/xxx\/cloud\/pub\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\/yyy\/pub\/public.php\/webdav\/portage-20170705.tar.bz2","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/xxx\/cloud\/pub\/apps\/dav\/appinfo\/v1\/publicwebdav.php","line":107,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/home\/cloud\/pub\/public.php","line":79,"args":["\/xxx\/cloud\/pub\/apps\/dav\/appinfo\/v1\/publicwebdav.php"],"function":"require_once"}],"File":"\/xxx\/cloud\/pub\/apps\/dav\/lib\/Connector\/Sabre\/File.php","Line":222,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:66.0) Gecko\/20100101 Firefox\/66.0","version":"15.0.7.0"}

Browser log

Browser log

JQMIGRATE: Migrate is installed, version 1.4.0

https://xxx/pub/public.php/webdav/portage-20170705.tar.bz2
Anfragemethode:PUT
Referrer Policy:no-referrer-when-downgrade

headers:

Host: xxxxx
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: /
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-crossover-bz2
If-None-Match: *
Authorization: Basic a0dtRmJLbTk5aXQ1N0FhOg==
Content-Disposition: attachment; filename="portage-20170705.tar.bz2"
requesttoken: 2v9jt/syI3T5IZRxMs3OETLhHMLk/bojj+koPzcrdtw=:vo8x9M1cThysV9YpA4yPdgSnaJKMseJZycJEdXBIG6U=
OCS-APIREQUEST: true
X-Requested-With: XMLHttpRequest
Content-Length: 78097358
Connection: keep-alive
Cookie: nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; 511223a9b3115=ofhu9kko892ajor3kim0g2g6an; oc_sessionPassphrase=DXbBZmGhzJRvHek1Hled5wf1eojNECxryv73jXQqLQK5fgYQ3O8ZxHBxTDU27EOzsIlLMx0OfJfbPt7jkuTyHLLTsfLvN0EavF4x%2BKdDYtXU7KwhLP7UBZpGLmpIdFTU; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; _ga=GA1.2.1759605856.1427960753

Hope this helps in any way. Any help from your side is very welcome.

many thanks for your time in advance.

niels.

[kesselb]

Looks like #15095

[nielsd]

Looks like #15095

unfortuantely not - uploads work with sync client as for authorized users.

[kesselb]

You disabled mod_reqtimeout?

[nielsd]

You disabled mod_reqtimeout?

wow! Many thanks. This seems to work / solve this issue.

many thanks!

btw: I've read the docs but stumbled over this multiple times - as it worked with erlier versions of NC or apache revisions.
sorry for the noise.

[nielsd]

solved

[stfast]

The problem is tmpfs size for PHP-FPM. Since user is anonymus over shared link, temporary upload file isn't written to Nextcloud user directory (over PHP session), but is written to system tmpfs (/tmp) instead, which is 2Gb by default, thus upload file over shared link is being truncated:

[root@nextcloud ~]# df -h|grep tmp
tmpfs 2.0G 0 2.0G 0% /dev/shm

Our soution is to change PHP-FPM tmpdir settings:

[root@nextcloud ~]# cat /etc/php-fpm.d/custom.conf
[www]
env[TMP] = /Nextcloud_Data/upload_tmp_dir
env[TMPDIR] = /Nextcloud_Data/upload_tmp_dir
env[TEMP] = /Nextcloud_Data/upload_tmp_dir

[root@nextcloud ~]# systemctl restart php-fpm

Another possible solution would be tmpfs resize, but actually this is not a good idea, since tmpfs is really RAM allocation/consumption.

https://wiki.archlinux.org/index.php/Tmpfs

[kesselb]

Thank you for digging @stfast 👍

If anyone of you want to try this please validate first that you are really affected by this. You might create a file with dd dd if=/dev/zero of=/tmp/test.dd bs=1M count=3500 and check if this has some effect on df -h|grep tmp.

I don't have time to take a closer look at this but changing to TMP to the upload_tmp_dir within your data folders looks dangerous. Maybe nextcloud does some housekeeping and remove files from this folder.

[stfast]

PHP setting upload_tmp_dir in global /etc/php.ini does not have affect for PHP-FPM.

I've tried that on the half path, but didn't try all settings variations (you are welcome!) like

php_value[upload_tmp_dir] = /some/tm/pdir

in /etc/php-fpm.d/www.conf.

PHP-FPM tmpdir is under php-fpm service (systemd) control and . You will see changing temporary file in /tmp/systemd-private--php-fpm-*/ directory.