big folder owncloud/data/<user>/cache #9513
Comments
|
Please try switching to system mode cron in the admin page, then setup the crontab as specified here: http://doc.owncloud.org/server/6.0/admin_manual/configuration/background_jobs.html#cron There is a cleanup routine that auto-deletes abandoned chunks that are older than one day. Not sure why you have that many abandoned (cancelled) chunks in the first place. |
|
I discussed the issue with you yesterday on #IRC, i have setup the croon on system but still getting the same problem with my users. What the routine cleanup? you mean using find and deleting the files? |
|
No, I mean there is already code here https://github.com/owncloud/core/blob/master/lib/private/cache/file.php#L118 to garbage collect. And now that I see it, it seems to be only triggered when the user logs in again... and the sync clients keep the current session. I'll double check, I did remember seeing a background job that was supposed to call that function somehow. |
|
CC @icewind1991 for clarification |
|
@mzaian in the meantime you could try doing it with a shell script run through a cron job. |
|
I will clean up the cache dirs manually until i have a fix for this issue |
|
Let us know whether that worked. |
|
Background job is not cleaning up the bad old chunks its not the solution, can anyone confirm if we are hit by a bug or what is the root cause of this problem? |
|
@mzaian I don't understand. Do you mean you setup a cron job to clean up old chunks (ideally once a day) but the cache folder is still full of abandoner chunks ? Something else to check maybe: have a look at a specific known files. Now to find out whether the remaining chunks were from an abandoner/cancelled upload, check whether the total size of the chunks matches the size of the known file. If the size matches: it's a bug, because the chunks must be deleted after the final file was assembler. |
|
Doing a cleanup using find and rm manually or using a cronjob is fine. This is an example for a specific file which is totally :- Now to the user's cache directory in the server:- root@owncloud:/data/owncloud/data/eomar/cache# ls -ltrah oaj2se* |
|
No updates regarding the issue is confirmed or not? |
|
I cannot confirm you issue as I don't have an environment where the issue happens. From what I see from your last info it seems the chunk number Can you provide a similar listing but where the time appears ? (there's only date) Also would be good to check owncloud.log and the web server error log to see if there are errors related to such files / repeated uploads. Also you can check the sync client log from that same time as the server to see whether the client was getting errors which made it retry multiple times. |
|
The issue is reported in the forums from many users i will post file log time stamp and the same from the user's sync client. |
|
Can you post a link to the forum topics you had in mind so I can have a look ? |
|
Here the file chunks from server side:- root@owncloud:/data/owncloud/data/mhazem/cache# ls -ltrah --full-time * Here are the sync client log:- #=#=#=# Syncrun started 2014-07-16T12:29:28 until 2014-07-16T12:29:39 (10856 msec) |
|
Okay thanks. Interesting, every five minutes. And always only the first chunk. Have you ever been able to upload a file bigger than 10 MB ? Now let's see if we find clues in your logs about those aborted chunks. |
|
What it might be: the file is always changing on the client side, and the client has detected it has changed between chunk, so it aborts the upload after the first chunk. Maybe the server could detect that the client is starting uploading the file from scratch and delete the stale chunks. But that's hard to do if we don't want to break the use case of two client uploading in parallel the same file (someone need to win the race eventually and we should not end up in a live lock) |
|
@ogoffart I think we should be able to find this in the sync client logs, right ? Find whether the file has been changed. I agree that the server cannot safely delete chunks for the current file. Still hoping to find the core issue. |
|
well, there is the error "Local file changed during sync." which one can see in the log. I think the client does the right thing. If there was an API to cancel the chunked upload, the client could also use it. |
|
I mean, if the file is actually NOT changed on the local file system, then there is a bug in the client, yes. |
|
One idea would be for the client to send the previous transaction ID with the new transaction. Yes, we still need to find out why the client thinks the file changed that often. |
|
you mean a header like |
|
Yes 😄 |
|
I'll make a separate ticket for that. Let's keep @mzaian's ticket here to investigate why the sync client is repeatedly sending the same file. |
|
See #9676 for discussing the new header. |
|
Do you want me to provide any other logs or information? |
|
@mzaian yes, please provide the owncloud.log and sync log of a failing file. Goal is to find out why the sync client is re-sending the same chunk over and over again instead of finishing the file. |
|
Some of the broken files are created months ago, files are synced with +1 day to the server if you check the provided info above. |
|
As you can see in the discussion of #9781 it seems the Owncloud Client changes the timezone in use by the client computer to UTC or something else when syncing some files, breaking the sync operation. I did a more extensive test and reported in #9781. Hope this gets fixed soon. This is a serious bug. |
|
Hi everyone, I just had a look on my ownCloud server and saw 13.6 GiB of chunk files in my cache folder. I do remember very well what happened when I tried to upload these files, because I created an isse for that reason. The upload using the desktop client for OS X failed. Please see issue #2042: owncloud/client#2042. Maybe these two problems correlate? |
|
Stray chunks are there because of failed upload. Note that this isn't related to what was discussed with @ogoffart above where the client would send the old transaction id. That part would be a secondary mechanism to expire chunks even earlier. |
|
running owncloud 7.0.3 and the latest desktop clients: the clean up does not happen. |
|
Does the cleanup happen when the user logs in over the web UI ? You can delete older files manually, yes |
|
no, cleanup doesn't happen on user login, and not through cron job, either. |
|
We're looking into ways to directly write chunks into the final file and remove the need of the cache folder. (#4997) |
|
Cleanup of chunks will now happen as part of a background job as per #14500 (CC @icewind1991) |
|
Which owncloud server version im currently using 7.0.3 |
|
It will be in the upcoming 8.1 |
|
I see it was backported, so will probably be in 8.0.3 |
Hello,
I'm having a big issue with my owncloud setup at work i recently upgraded the server to 6.04 and the latest client 1.6.1.
Some owncloud users have a big cache directory and still creating chunks of 10mb in their respective directory i tried to delete these files manually but its hitting back again and my storage is growing since i'm having the issue with half of my users.
I need to solve it as soon as possible.
Thanks,
The text was updated successfully, but these errors were encountered: