Federation: Upload reports insufficient storage but we are far from any limit

[mmattel]

Steps to reproduce

1. having a federation between 10.0.2 remote (shares the folder) - 10.0.3 local (recieves the folder)
2. try uploading a file (browser or client) with 24MB to the federation recieved folder at 10.0.3

Expected behaviour

No errors

Actual behaviour

Error reported about insufficient storage via oC client or browser
Upload to the local 10.0.3 instance or 10.0.2 instance works (no federation)
Upload to the federation target (10.0.2) directly, works and is distributed afterwards to 10.0.3 and W10
Free space local: 37GB
Free space remote: >200GB
No change if set in config.php quota_include_external_storage = false
No local/remote user quota limit reached
Creating 2 nested directories with some smaller files in but in total 33MB (which is of course higher than the single 24MB file) than the upload via 10.0.3 to the federation works = single filesize matters.
It worked before in this combination and suddenly stopped. The federation accepted filesizes of 350MB+ without any problems.

Server configuration

Operating system: Ubuntu 16.04

Web server: nginx

Database: mysql

PHP version: 7.0

ownCloud version: 10.0.3

Updated from an older ownCloud or fresh install: updated

Where did you install ownCloud from: tar

Signing status (ownCloud 9.0 and above): no problems

The content of config/config.php:
config_report_20171008.txt

List of activated apps:

Enabled:
  - activity: 2.3.6
  - comments: 0.3.0
  - configreport: 0.1.1
  - dav: 0.3.0
  - federatedfilesharing: 0.3.1
  - federation: 0.1.0
  - files: 1.5.1
  - files_external: 0.7.1
  - files_external_dropbox: 1.0.0
  - files_external_ftp: 0.2.0
  - files_pdfviewer: 0.8.2
  - files_sharing: 0.10.1
  - files_texteditor: 2.2.1
  - files_trashbin: 0.9.1
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 16.0.2
  - market: 0.2.2
  - music: 0.5.2
  - notifications: 0.3.1
  - provisioning_api: 0.5.0
  - systemtags: 0.3.0
  - templateeditor: 0.1
  - updatenotification: 0.2.1
Disabled:
  - encryption
  - external
  - files_antivirus
  - theme-example
  - user_external

Are you using external storage, if yes which one: smb, federation

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Opera

Operating system: W10x64

ownCloud log (data/owncloud.log)

"user":"xyz","app":"webdav","method":"MOVE","url":"\/remote.php\/dav\/uploads\/xyz\/1488037006\/.file","message":"Exception: {\"Message\":\"HTTP\\\/1.1 507 Insufficient Storage\",\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\InsufficientStorage\",\"Code\":0,\"Trace\":\"
#0 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/QuotaPlugin.php(105): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\QuotaPlugin->checkQuota('\\\/abc...', 25116827)\\
n#1 [internal function]: OCA\\\\DAV\\\\Connector\\\\Sabre\\\\QuotaPlugin->handleBeforeMove('uploads\\\/xyz...', 'files\\\/xyz\\\/...')\\
n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\
n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(633): Sabre\\\\Event\\\\EventEmitter->emit('beforeMove', Array)\\
n#4 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpMove(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\
n#5 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\
n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:MOVE', Array)\\
n#7 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\
n#8 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Server.php(234): Sabre\\\\DAV\\\\Server->exec()\\
n#9 \\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\
n#10 \\\/var\\\/www\\\/owncloud\\\/remote.php(165): require_once('\\\/var\\\/w...')\\
n#11 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/QuotaPlugin.php\",\"Line\":170,\"User\":\"xyz\"}"}

[PVince81]

Not sure yet but could be related to #29103 which I haven't been able to reproduce yet

[PVince81]

note: it's a chunked upload on new DAV endpoint (see the .file MOVE operation)

[PVince81]

Added an automated test to upload to a received federated share when quota is applied, the test passes: #29193

Need further research, I think the scenario here is a bit different anyway

[mmattel]

The incident first occurred was on Friday night with many activities for client syncing.
About every 5-10min a file with about +20MB was changed a little bit and saved on the W10 client which was then synced to the federation. It continued to be problematic over the weekend - no changes made on both side of the federation ! - The last file upload attempt was on Sunday afternoon. On Monday morning - again - no changes made - changing the file, syncing automatically succeeded...
The limiting upload bandwidth is 5Mbit/s which is less than a minute needed.

One idea is, and maybe I am wrong, that oC gets confused when a file that is currently beeing in process of federation sync gets a new version via the client autosync. And somehow there is a internal job running regularly that cleans up whatever was messed up. Just my 2c

[PVince81]

Hmmm not sure... now that you talk about stuff "being in process" this reminds me of part files.

Can you check if the folder in question contains any stray ".part" files ? I suppose this could skew quota calculation as these files are not indexes, so their size does not count.

But my gut feeling says the problem is somewhere else.

[PVince81]

The problem is reproducible, but only with the desktop client upload:

1. Setup two servers OC A and OC B
2. From OC B share a folder "test" with admin@OC_A
3. Login on OC_A as admin
4. Accept the federated share
5. Set the quota of the admin@OC_A user to "1 MB"
6. Upload a 100 MB file into "test" with the web UI: works (network console show chunking is used)
7. Setup desktop client for admin@OC_A
8. Put another 100 MB file "test/data2.dat" in the local folder of the desktop client
9. Wait for sync

Expected: sync goes through
Actual: 507 on the final MOVE

I suspect that the difference is that the desktop client might set more headers in the final MOVE, one related to the final size or something, and somehow this seems to make the quota check bug.

I'll debug this...

[PVince81]

oh come on... I thought this was fixed before: QuotaPlugin does call getFreeSpace('/test') and receives -3 because no quota is set (unlimited) on the remote. But this special case isn't detected. I thought we fixed that already.

[PVince81]

Ok now I also see the error in the web UI... weird...

Nevertheless, here is the task list:

• adjust integration tests to be able to reproduce this case
• add check for -3 value to ignore quota check on federated share

[PVince81]

Fix is here: #29220

[mmattel]

Thanks for the fix

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.