OCA\DAV\Connector\Sabre\Exception\FileLocked - Redis race condition #9001
Comments
|
I have the same problem with 2 servers on Debian 9. |
|
Exact same problem here as well. |
|
I have had the same problem as well. We have remote offices uploading over 5mbps lines and after several attempts to upload 1/2GB files we had to give up. We have stopped using NextCloud and will have to wait until this is fixed. |
|
Same here. |
|
Same problem. Large files over a slow link cause file locking - the same upload over a fast network will successfully upload. I would love a fix for this. |
|
We have started experiencing this more than in the past ... I can't prove correlation to the upgrade from 12 to 13, but that's what it feels like. |
|
Hello, one more example. As described here https://docs.nextcloud.com/server/12/user_manual/files/access_webdav.html#accessing-files-using-curl i use curl to upload a 4,4GB file. The first from a Hetzner root-server with 100MB/s connection: The second with a 2MB/s upload cable-connection: The complete error from nextcloud.log: My config: If any informations missing, please ask. I hope, someone can help. This problem is so frustrating. |
|
It is very frustrating... |
|
Same issue after manual 13.0.1 upgrade. |
|
Because of this problem I had to install vsftpd today :-( (and I thought with nextcloud I'd never have to do that again). some seconds later: And the nextcloud.log: |
|
I've tried 12-13 and all the same as above. |
|
me too :D |
|
I remove this options from my config: upload a file again and it is the same problem. So redis should not be the cause. Are there any ideas I could still test? |
|
Is this a duplicate of #8239 ? there seems to be quite a few affected by this just searching on google. best workarounds imply disabling file locking altogether. |
|
'filelocking.enabled' => false, is not a solution. |
|
Issue still existing in 13.0.2.
👍 |
|
For me, the problem also occurs during large file uploads. Uploading +40GB virtual disk files always fails, despite a 40-60Mbps throughput. This worked fine in v12. |
|
Sorry ... @MorrisJobke any idea how to solv this problem? |
|
Some additional infos from my side. For testing, I set 'filelocking.enabled' to false and copied a whole folder to the remote location (I use the german "Magentacloud"). So I wanted to verify that the issue is not situated at the storage provider. Then I copied (cp -rv) many files within many folders to the remote location and back (I used the nextcloud source folder 😉 ) So for me it seems to be an issue in the "Sabre" library itself. And disabeling file locking is (for me) also not working. If I can help out with testing, just let me know. EDIT: Same in 13.0.3. A colleague in my company told me that he made the same experience in another use case (not nextcloud). There the cause was situated in the frequency the files were pushed to the remote. The client pushed the files to fast for the server. This may be also the cause here, as larger files that take more time to transmit were no problem here while high amount of small files lead to the above error messages. |
|
Same problem, came with 13.0.4, swift primary storage, Redis used for filelocking and memory cache as well. |
|
Yes, the problem still exists (have the same version). I have the impression that a lot of new stuff is being developed in nextcloud, but the known bugs are not being fixed. |
|
@BurtGummer @vpecinka Does it happen reliably for you or only from time to time? |
|
cc @icewind1991 |
|
And if so: could you enable the locking debug log in the config.php? See config.sample.php for the description. |
|
Hey @MorrisJobke thank you very much for your response. Yes, the problem happens reliably every time, the upload connection is slow and the file is big. In #9001 (comment) i had 'filelocking.debug' => true, filelocking.debug was enabled all the time, I forgot to disable this option. |
no, thats wrong. please read my tests, i upload one file with 4,4GB and the error occurs. Until now, I have some smaller files with a bigger total size not uploaded, but i'll try it. |
|
This does happen reliably for me as well. Not sure if this is too vague to be useful, but I've found anything below 2mbit/s for files 2.5GB or more will fail 100% of the time. |
|
I've tested it from several different locations with different upload speeds, from 2 till 100 MBit/s. Internet connections with better performance are able to upload bigger files. The file size does not matter, more the consequence. I can upload 18GB big files, when its fast enough (internal network). In my test it seems that all uploads fail, when upload time is longer than one hour. The upload goes till 99 percent and at the 100 percent mark the file locking issue appears. |
|
I can confirm that this is also happening on my Nextcloud 14 instance. It seems that if the upload over WebDAV takes longer than an hour, the transferred file disappears from the server and it logs two errors like: Fatal webdav OCA\DAV\Connector\Sabre\Exception\FileLocked: "xxx" is locked I am using APCu for caching and have not explicitly configured memcache.locking Has anyone found a workaround for this? |
|
issue still persists in Nextcloud 15.0.2 :/ |
|
this issue is not redis related. It's file locking in general. I setup a new nextcloud installation without redis and file lock enabled. Issue is still the same --> upload longer than 1 hour leads to 0kb file. With file lock disabled everything is fine. |
|
This bug is a endless story... :-( should have the label "High"... |
|
Please see if #13739 fixes the issue for you |
|
Still an issue. This should be marked as high - we stopped using NextCloud in production because of this issue. |
Did you tested the PR from @icewind1991? Did you also waited long enough so that the locks are not there from before the patch was applied. The locks have a TTL of 1 hour. So either clear them (with the risk of running into a concurrent request) or wait for at least 1 hour. |
|
I have to wait until the next version 15.0.3 because I don't know how to apply the patch. I can hardly wait. |
|
And I've already had hope.... install 15.0.3 and upload a big file: nextcloud.log: :-( |
skjnldsv
added
the
0. Needs triage
|
I also struggled quite a while with this, it was a showstopper and a huge PITA because testing any changes would always take at least an hour and it's not very obvious why there should be a locking problem in this situation. I used to test this with root@uploadtest:~# time curl -u "SomeUser:SomePassword123" -T 500MB.zero --limit-rate 100k "https://myserver/remote.php/webdav/SomeDir/500MB.zero"But after the transfer was done (about 85 minutes), I always got: <?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
<s:exception>OCA\DAV\Connector\Sabre\Exception\FileLocked</s:exception>
<s:message>"SomeDir/500MB.zero" is locked</s:message>
</d:error>During the transfer I could see the file grow in I also tried with I tried the other options mentioned here and in other issues, such as disabling file locking, but when setting After exhausting the other options, I finally stumbled upon a parameter I was using the latest stable snap (15.0.8snap3) so ran: # set it to 12 hours
/snap/bin/nextcloud.occ config:system:set filelocking.ttl --value=43200
snap restart nextcloudAnd now it works!Also with root@uploadtest:~# time curl -u "SomeUser:SomePassword123" -T 500MB.zero --limit-rate 100k "https://myserver/remote.php/webdav/SomeDir/500MB.zero"
real 85m20.745s
user 0m2.131s
sys 0m3.961s
I noticed in older threads that people have tweaked // no php execution timeout for webdav
if (strpos(@ini_get('disable_functions'), 'set_time_limit') === false) {
@set_time_limit(0);
}However, in /**
* Set the lock's time-to-live in seconds.
*
* Any lock older than this will be automatically cleaned up.
*
* Defaults to ``60*60`` seconds (1 hour) or the php
* max_execution_time, whichever is higher.
*/
'filelocking.ttl' => 60*60,So if I understand this correctly, even if I tweaked $ttl = $config->getSystemValue('filelocking.ttl', max(3600, $ini->getNumeric('max_execution_time')));The serverinfo page shows the <max_execution_time>3600</max_execution_time>So perhaps setting the The relevant options in my config when this thing started to work were: There may be more issues related to this but in my case this ultimately seemed to solve the "uploads over 1 hour fail" problem. Hope this helps. |
|
Hello joukos, thank you for your work. I have set: 'filelocking.ttl' => 43200, in my config.php and upload a large File: yes, it works! But now the question is, is this solution safe? |
Good question. It is save to raise the TTL IMO, but you buy in another downside, that if a process crashes for some reason and does not clean the locks you need to wait longer to access the resources again. :/ |
skjnldsv
added
1. to develop
|
I have the same or similar issue. Redis is enabled but "xxx is locked" appears after one minute of uploading many small files/folders via webinterface. PHP max_exec is 3600. Is there anything I can check? NC: 18.0.4 Details here: nextcloud-snap/nextcloud-snap#884 (comment) Thanks in advance! |
|
Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you! |
szaimen
added
needs info
0. Needs triage
and others
Steps to reproduce
Expected behaviour
When the upload finishes (monitoring the last seconds) the file should appear on filesystem and in webinterface.
Actual behaviour
The last Bytes are transmitting and then appears the "locked" message in Webbrower. The message
OCA\DAV\Connector\Sabre\Exception\FileLocked will be logged in Adminpage. It happens when somebody uses a not so fast Internetconnection and is uploading for longer than one hour (f.e. 3GB, over 5 MBit/s, over 70-80 minutes), but somebody who is able to upload 4.5 GB in 12 minutes has success.
Server configuration
Operating system: Ubuntu 16.04.4 LTS
Web server: apache2 (2.4.18-2ubuntu3.5)
Database:
PHP version: php7.0 7.0.28-0ubuntu0.16.04.1 [1.290 B]
Nextcloud version: 12.0.3, 12.0.6 (actually), 13.0.0
Updated from an older Nextcloud/ownCloud or fresh install: First 12.0.3 than updated over Adminpage.
Where did you install Nextcloud from: Ubuntu repository
Signing status:
Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: Yes, SMB.
Are you using encryption: No internal/build-in. Only encrypted filesystem on Ubuntu-erver himself and Bitlocker on Fileserver.
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Internet Explorer 11, Firefox 59.0.2
Operating system: Windows 7 Professional x64
Logs
Web server error log
Web server error log
Nextcloud log (data/nextcloud.log)
Nextcloud log
The text was updated successfully, but these errors were encountered: