Hige idle cpu when external storage is added

[ghost]

it seems my owncloud install has some problems with hige cpu usage when adding external storage in the gui. i have tried with both cifs, ftp and sftp and all 3 types generates steady hige cpu usage all the time the share is mounted. removing the external storage and cpu usage returns to normal.

debian 7 wheezy
owncloud 7
openfiler as external storage (cifs,ftp and sftp)

[ghost]

Owncloud put file trees in mysql after a external storage is scanned, how long should a search like that take?

Could it be that this scanning of the external storage is the problem cousing hige cpu usage.

external storage is close to 600 000 files.

is there a way to check this scanning progress.
top shows apache2 is using the cpu at around 50% for as long as the storage is mounted

Mysql is showing alot of traffic from owncloud: row, oc_filecache is almost 250 000 entries and 120mb.

so it seems that the cpu is scanning through the file paths on the external storage and is caching them in DB.

is this normal behavior?

[karlitschek]

hmm. hard to say but I guess this is normal for 600.000 files

[ghost]

I doubt that its normal, there cant be a depth, file limit for owncloud. So you say if i have a owncloud cluster there is a limit to how many files it can handle?

If this is in fact the normal behavior i think owncloud has fundamental design flaws.

I did read a bit about issues related to the scan function for storage, it seems its plagued with bugs.

Also deleting external storage should clean out the db for cached paths related to that storage. Its not.

i find there is alot of bugs in owncloud.

1. Document in /data is not removed when source is removed.
2. Regarding cifs external storage there is issues with it, when a client downloads a file from owncloud, file is first copyed from cifs to internal storage, first after this process is complete the download from the client continues. If file is large copy from ext to int takes long time, client wil time out before copy is done and made available for download.
3. There is no cleanup crons, local storage and db needs cleanup as alot of cached files is never removed.
4. Sms and news app is full of annoying bugs.

So my conclusion is owncloud is still in early beta/alpha regarding code stability. Its inefficient code. But its far better then a couple of years ago when i last tried it. Its even usable now if the external storage can be fixed.

[PVince81]

For 600 000 files the file scanner might take quite a long time to index all files.

Deleting an external storage will cleanup properly as per #10820

As per your points:

1. Can you raise this as a separate bug ?
2. Some external storages do not support streaming yet, so files are downloaded/uploaded with an intermediary step in a temp folder. This will be solved with the new SMB/CIFS implementation: Use native library for SMB mounts #9480
3. You are probably refering to update detection ? Whenever a file was changed remotely on an external storage SMB/CIFS (without going through ownCloud), the internal cache/index needs to sync itself. It usually does this on demand. But there is work planned to improve update detection in general here: Get remote update detection issues for files_external sorted out #11797
4. Please raise tickets in the respective app repos (these are 3rd party apps): https://github.com/owncloud/SMStorage (not sure if it's this one) and https://github.com/owncloud/news

Thanks!

[PVince81]

CC @icewind1991 @butonic for the scanner issue, did we have a ticket to look into improving scanner performance ?

[PVince81]

The scanner has been moved to a background job (cron) in 9.0, so while it might still use CPU for the actual scanning task, it won't be triggered when a user is inactive in the web browser.