files_external loads all users on admin page, which opens extremely slow with large user base

[blizzz]

Steps to reproduce

1. set up owncloud with a large user base (e.g. using LDAP)
2. log in as admin user
3. open admin page

Expected behaviour

Admin page opens quickly

Actual behaviour

Admin page takes long to load, because all users are fetched

Quick fix: load users via Ajax. Open admin page immediately, only files_external configuration will take longer until it is ready

Better fix: as above, but do a server side search if users and groups like the sharing box does.

Server configuration

ownCloud version: (see ownCloud admin page)

stable6, most likely master too

[PVince81]

Wow, I didn't realize that this app would do that.
Yes, your quick fix makes sense.

[PVince81]

I had a quick look and it seems we're using the "chosen" library to display the list, which itself doesn't support ajax load and would require some trickery.
We could switch from "chosen" to "select2" which is a fork of chosen that supports ajax calls (even pagination!)

[ser72]

@blizzz @PVince81
Any updates on this? I have a user experiencing this issue as well.

[blizzz]

The workaround is to disable files_external if it is not needed.

[gig13]

@blizzz @PVince81
This has been identified with several instances that have a large number of users. We should be able to scale to a large number of users with either the search box type feature, scrolling between pages of users, or have an alphabetical set of pages between users.

So far, the search box type feature sound the most promising.

[butonic]

I have already worked with select2 and will have a look.

[gig13]

@butonic
Any updates?

[butonic]

@DeepDiver1975 pointed me to https://github.com/owncloud/core/tree/user-jquery where an ajaxified version of the user management happens. I am preparing a PR for the files_external app, based on current master. We only need the filter of the userlist anyway.

[ser72]

@craigpg @karlitschek

Can we get someone to review this fix so we can get it in 6.0.4.

[DeepDiver1975]

@ser72 the fix is in #8507 - would be great if you can test this and give feedback - once merged into master we can discuss the backport to OC6 - THX

[ser72]

@DeepDiver1975

I don't have an LDAP large enough to reproduce the issue. It was originally reported by @blizzz
and I have several end users seeing the issue, but I was never able to reproduce in my lab.

[PVince81]

@ser72 I think you don't necessarily need to reproduce the slowness issue, but just test whether the PR works with many pages of users (like 200-300 users and many groups) and doesn't omit any.

The PR makes the results paginated in some way.

I just tried it and the result set is messy, so that will need fixing first (see my PR comment)

[craigpg]

Moving to 7.0.1

[PVince81]

The core issue that makes this one hard is that the results from the server are NOT sorted.
Even if we fix the JS code to properly group the results, it means that when scrolling down multiple pages, if an user from "group1" pops up in the result, that one will be inserted on page 1 while you were on page 10, which makes this unusable.

I'm not sure it's worth to spend time to try working this around with JS because it will still be unusable.

@blizzz is it possible to get sorted results from LDAP, assuming that we only use a single user backend ? (I understand that sorting + pagination is not possible with multiple backends).

[ser72]

@blizzz Any updates here

[ser72]

@blizzz
Currently have a reproduction in the lab in oC6 if you have any fixes that you want tested.

[blizzz]

@blizzz is it possible to get sorted results from LDAP, assuming that we only use a single user backend ?

No, unfortunately not. Only without Paged Search, but this is not an option for various reasons.

[blizzz]

@ser72 i need to continue work on it, however I have fixed deadlines for other things atm.

[ser72]

@blizzz

A user tried oC7 and said it is much faster to load the admin page. Has this fix been placed into oC7?

[blizzz]

no

[PVince81]

Another idea: how about using the exact same mechanism like the share dropdown ?
The admin would need to type at least 1-2 letters of a user/group before getting results. This means that we don't need to group the results or provide infinite scrolling in such case. If there are too many results, the list is truncated to the first X results. (not sure how many are shown in the share dropdown)

[blizzz]

I had the same thought once upon a time, but why did i not follow it? Because it makes absolutely sense. And not only here but also when it is about picking groups only, e.g. in the apps when limiting access.

[butonic]

@blizzz @PVince81 if you are busy with other things I'll take care of this next week.

[blizzz]

@butonic This would be awesome

[ser72]

@butonic
I see a lot of commits above. Status on this please?

[butonic]

@ser72 #8507 is only waiting for @blizzz or maybe your review with lots of users