Create a Way to Disable Parallel Uploads

[dragotin]

For various reasons we can not enable the parallel upload by default. We need to disable the parallel upload and only switch it on by a flag if admins decide that. Even if that results in a performance drop.

e.g. #2724 #1866 (comment) #2401

[dragotin]

This is not only about uploads but also all other operations. Disable parallelism by default.

[guruz]

OK... check for status.php version to be >= 8.1 ?

Or check the capabilities API for a maxparallel value.
curl -X GET 'http://root:admin@localhost/owncloud7/ocs/v1.php/cloud/capabilities'

[PVince81]

As discussed on IRC we could add a new value "maxparallel" to that capabilities call with the following logic: if SQLite is used, "maxparallel" would be 1, else 0 (infinite)

[PVince81]

In general I think we should start using this "capabilities" endpoint more often to share server info with clients.

[dragotin]

agree with @PVince81

[guruz]

We also had the idea to check all error messages for SQLSTATE and then force paralleism down to 1

[PeterPablo]

Would disabling parallelism also prevent wrong/missing ctimes and mtimes (confer the problem described in the referenced issue #2401).

[ogoffart]

@PeterPablo: no, it would just make it less likely.

sqlite should be avoided on the server if possible.

[PeterPablo]

In my opinion users should be made aware of this in a more pressing alert than described in the current documentation (confer #2401 (comment)). I still consider this loss of data and as such a bug that should be tackled.

[radry]

I agree with this issue! On my low memory owncloud php runs out of memory when uploading many files with the windows client because of parallel uploads!! Uploading via a browser works perfectly fine. This is a major issue since there are no (free) alternative clients which can sync (instead of "mounting" the webdav which I don't want).

[dragotin]

We need to know to which extend the server fixed the underlying problems to be able to decide what to do.

[PVince81]

Parallel upload should work now on OC 8 RC 2, the major database is locked issues were fixed.
It worked fine in my various tests (and I had a slow HDD / VM where the issue could be reproduced consistently before the fix)

[PVince81]

For OC 7, backports are still required:

• [stable7] Close cursor early in calculateFolderSize core#13832
• [stable7] Close cursor when setting preference value core#13831

[radry]

@PVince81

Great but what about php running out of memory on low end servers? I realize that is not the problem of owncloud per se but the parallel upload is still to blame, so there MUST be an option to disable parallel upload in the client or the server manually.

[guruz]

@radry I believe that problem should be fixed because the 1.8 client only uses a chunk size of 5MB (vs 20MB in 1.7.x and 10 MB in 10.6.x)

[ogoffart]

So according @PVince81 in #2724 (comment)
recommand disabling parallelism for anything less than owncloud 8.0

Should we disable only parallel chunk upload (which is new in 1.8), or any kind of parallelism?
Don't we risk users complaining about preformence regression in 1.8?

Another option we could take is to disable the parallelism once we detect the 500 error.

[moscicki]

I would like to say it again: what about that the server tells the client what kind of features it supports? In this way you may avoid any hardcoded assumptions and also avoid to manually re-adjust the client when the new versions of the server develop new capabilities (which is anyway impossible because you still have old versions of the servers around).

The same also for the size of the chunk. Changing from 10MB -> 20MB -> 5MB looks a bit like a random walk for me. I KNOW what is the averge file size in my service, so why would not you let me configure the server with the correct number for MY service.

And above all it all may be achieved by adding extending simple JSON reponse of status.php. Why make it any more complicated?

kuba

On Feb 6, 2015, at 1:00 PM, Olivier Goffart [email protected] wrote:

So according @PVince81 in #2724 (comment)
recommand disabling parallelism for anything less than owncloud 8.0

Should we disable only parallel chunk upload (which is new in 1.8), or any kind of parallelism?
Don't we risk users complaining about preformence regression in 1.8?

Another option we could take is to disable the parallelism once we detect the 500 error.

—
Reply to this email directly or view it on GitHub.

[ogoffart]

@moscicki I agree with you, and that is already implemented by d7c2b66 for the amount of parallelism, but the server do not expose this after all.

[PVince81]

I'm fine with adding this back or making this configurable, but not sure if others agree: @karlitschek @DeepDiver1975
The ticket for "max_parallels" was this one: owncloud/core#13628

[luciamaestro]

It works fine version 1.8.0-beta1