Deleting big files within webinterface takes long time

[disy-mk]

I have OC running on a Server (s1) using nginx; the data directory resides on another server (storage - s2) and is mounted via NFS.
Deleting a big file, e.g. of 2.5GB size, takes quite long; investigating this behavior I realized, that utilization is very high, but no cpu is used (top/htop).
iotop (on s1) shows initial writing activity for the first 20 secs, than nothing; no cpu then as well, but load climbs as high as >8.0, but is <0.1 in idle.

Just checked with a folder containing 2 files, 2.2GB in sum and it tool 6-7 mins to "delete" the file via webinterface (different devices showed the file(s)/folder being still available (webinterface) for that time.

Connection betw. s1&s2 is 100mbit and usually is not fully utilizable. So, it might be possible that the file "gets fetched" before it gets deleted.

What is happening in background there?

Both (s1&s2) are running in VMs on different phys. hardware. oc vers. btw. is up2date.(7.?)

[karlitschek]

Can you post a full bug report using the issue template please?
https://github.com/owncloud/core/blob/master/CONTRIBUTING.md

Do you use encryption?

[disy-mk]

Detailed template cough
Will hand it in tomorrow. ;)

No enc used.

[DeepDiver1975]

Assuming trashbin and/or versions are enabled: the reason is that files are copied around - moving them would be faster but has other limitations.

@schiesbn

[disy-mk]

This indeed might be the reason.

So, NFS RPC is used, but files are intentionally copied to the trash and deleted afterwards?
Or does this action additionally create 4 gigs of network traffic while deleting 2 gigs?

If so, maybe the solution for me might be to use (a modified version of) the file move plugin and move the files to the bin. Writing a plugin, that deletes files skipping the trash bin also might be a solution.

Thanks for your thoughts, @DeepDiver1975!

[DeepDiver1975]

we really should address this issue on the trashbin and versions app

@schiesbn has to have a look at this - THX

[schiessle]

Versions aren't a problem on delete operations. The problem is the trashbin. I would love to improve this. IMHO the best solution would be to implement the trashbin as a storage wrapper which moves the deleted files to either a ".trash"-folder on the same storage or keep them where they are and only rename them to something like from file.txt to file.txt.d<timestamp> and just hide the .d<timestamp> files. This way we could also keep all meta information like encryption keys, thumbnails, share information, etc. at the same place without moving them around... But we still need to agree on a solution which is acceptable from a (external-)storage point of view.

[DeepDiver1975]

But we still need to agree on a solution which is acceptable from a (external-)storage point of view.

for sure - and this brings back the ideas of having storage based implementations. We should schedule this together with @karlitschek and @MTRichards for the next release after OC8

[disy-mk]

Either of the two suggestions would be a massive improvement imho, although only affecting the minor part of the community; in e.g. in my case with the mounted NFS data storage this led to massive traffic; having 2 TB of monthly free traffic this issue bumped my traffic usage astronomically. The culprit of my current solution is that every byte gets doubled due to my setup consisting of a separate data storage.
So uploading 500 GB results in 1TB traffic for me. This in mind I know when and how much traffic to cause/use. In my case the scenario was(is as follows:
I start with 2 TB of free traffic.
I uploaded 600 GB, causing 1.2 TB of traffic, leaving 800 GB of free traffic.
I then downloaded half of it which results in 200 GB of free traffic. This 10% was assumed to be enough traffic for my customers as this is twice as much as they usually cause.
I then deleted 300 GB of the data, which resulted in 400 gigs of traffic for me to pay additionally, not to mention the additional traffic caused by my customers :-(
I already talked to my provider which I have a very good business relationship with and he assured me to only charge me with 30% of the traffic, which at 12 Ct. per GB still results in at least 15,- EUR -> no bone breaker, to be honest, but still happy that it wasn't 800 GB of upload and 400 GB of download/deleting, which would have resulted in at least 1.2 TB of traffic to be payed. ( 150,- EUR in worst case, if my provider would not grant me a discount).

For now I will disable the trash bin plugin before deleting big files (at least in the current monthly period), but this is annoying... have to

• log in as admin
• disable the app
• logout
• login as user
• delete
• logout
• login as admin
• enable the plugin
• logout

I want to point out that I am not angry by any means, because as admin it's me who is responsible for anything and alarm bells should automatically have rung... but there was no mechanism yet to indicate this circumstance...

So this issue at least should be pointed out publicly.

[MorrisJobke]

ref #12274 and #12487

@PVince81 @schiesbn Is this issue covered in one of the above tickets? Or is there a separate "implement trashbin as a move instead of a copy command"?

[PVince81]

This isn't fully finished.
The first step of making a move was done with the move of the trashbin to use storage wrapper #13377. Later on, @icewind1991 will provide further changes that should hopefully do the same in case the files are from a shared storage, see #13360

I suggest to keep this ticket open for now.

[PVince81]

On another note, here are some fixes for the trashbin expiration which was also slow: #13974 (comment)

[MorrisJobke]

Files are now moved to the trash bin: #13377

@PVince81 @schiesbn This looks like it can be closed now, right?

[DeepDiver1975]

@PVince81 @schiesbn This looks like it can be closed now, right?

yes

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.