Unusable for large collections

[RandolfCarter]

I know, the app is still in a preview state, but I wanted to give it a try; here's my report that alas, with a big music collection, the app is unusable at the moment.

On first access to the app, the page loading takes forever (actually it never finishes). I heard the hard drive being accessed so I thought I'd let it run. Then the access to the hard drive stopped suddenly. The server log then showed a segmentation fault in some child process; the apache and mysql process still showed high CPU usage. Only a restart of apache helped restore the ownCloud interface. And when trying to go again to the music app, the page loads forever, this time without any hard drive access, again locking up apache and mysql.

Please realize that people will have more than a few songs, and that therefore

1. Progress report during scan is crucial
2. The whole scan possibly cannot fit into one script run, or might be aborted in the middle, so there has to be a way to (ideally automatically) resume the scan
3. The scan not finishing should not break the music app; and even more importantly, should not prevent any other ownCloud access by leaving apache/mysql running at high CPU cycles...

Environment:
Ubuntu Server 12.04
ownCloud 5.0.10
music: tried with latest master (65f4678) and "various-fixes" branch (d75a4ff)
Client: Firefox 23.0 running on Ubuntu 13.04

[butonic]

@icewind1991 I had to find a solution for the 'scan a lot of files' problem when working on search_lucene.

IMHO The files app has the best scanning mechanism because it saves the scan status in the filecache table. Can we find a general solution for scanning mechanisms? How do we mark a file as scanned by search_lucene (I currently join my own status table with the filecache table ... but meh) or by music (or by pictures or by videos)? All of them could extract meta data. I personally would like to explore zend_lucene for searching through that ... (cc @andrewsbrown) ... but I may be biased. Perhaps we can come up with a more general search API that can use either the database or zend lucene ....

Anyway. I propose the following approach: we create a new onscan event/hook that apps that want to extract meta data can register for. Correct me if I am wrong but the files app currently determines the mime type and basic file meta data. music could listen for audio files and then use getid3 to extract more specific meta data. search_lucene likewise. Unfortunately, this naive implementations will cause synchronization to take longer because the file app scans the file on write causing an upload via webdav to wait for all meta extraction to be complete. Therefore we need to separate "file app"-scanning and meta data indexing from each other. This is where background processes come into play. I just don't know yet if adding a separate job for each file makes sense or if just adding a "reindex" job is more elegant. The problem with a "reindex" job is that you don't want to reindex all files but only the ones that are new. In search_lucene I use a status table to keep track of this. Music could use a flag in the songs table but then we are duplicating the mechanism. Single jobs will fill the jobs table quite quickly ...

@icewind1991 @karlitschek @kabum @DeepDiver1975 @bartv2 opinions? ideas? we need a more general solution for this?

[Tronic69]

I can confirm, that it's not usable for large collections and for me it seems to scan the collection every time again, when I open the musics app.

I opened the app and then it shows "Loading ...", but that never changes (not even if an error occurs)!
While the music app (tells me it) is loading my whole owncloud instance is unusable. 23 seconds after opening the music app I tried to access the admin settings to check for an error in the logs, but nothing happened until a PHP error occured and the script was "killed":

[Tue Dec 10 08:12:15 2013] [error] [client 46.5.241.128] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 77 bytes) in /srv/www/owncloud6_RC3/apps/appframework/db/entity.php on line 58, referer: https://oc.vtux.de/index.php/apps/music/

As you can see in the log lines below after nearly 2 minutes (when script exhausted memory) the admin request was processed. So you can see that everything else on the owncloud instance was blocked.

I haven't checked the code, but I think it would be a better idea to do just a rescan for all files where filecache.fileid > music_tracks.file_id (for current user) and then in (background) scan for deleted files which never exists in filecache. This would be unique, too.

Regards,
Tom

[10/Dec/2013:08:10:04 +0100] "GET /cron.php HTTP/1.1" 200 20 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"
[10/Dec/2013:08:10:04 +0100] "GET /index.php/apps/music/api/artists?fulltree=true HTTP/1.1" 500 20 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"

[10/Dec/2013:08:12:16 +0100] "GET /data/htaccesstest.txt HTTP/1.0" 403 1860 "-" "-"
[10/Dec/2013:08:12:17 +0100] "PROPFIND /remote.php/webdav/ HTTP/1.1" 401 291 "-" "-"
[10/Dec/2013:08:10:27 +0100] "GET /index.php/settings/admin HTTP/1.1" 200 6294 "https://<my_oc_url>/index.php/apps/music/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"

[RandolfCarter]

Any progress on the app in general? Will it be included in OC6? Or will there be no music app in OC6?

[MorrisJobke]

@RandolfCarter It will not be included into the OC6 package, but can be installed from the app store. Therefore the app could be updated individually. I'm quite busy with university at the moment.

[giovanisp]

will this app ever be included back in the OC cloud core ?

[MorrisJobke]

will this app ever be included back in the OC cloud core ?

cc @karlitschek

[karlitschek]

@MorrisJobke We can ship it with the next release if you want. Or we distribute it via the appstore independently. Or we add the option to update shipped apps with versions from the appstore in owncloud 7. But this still have to be implemented. :-)

[LukasReschke]

@karlitschek Could we please discuss privately the process of inclusion of new applications? - Adding new applications to the packages without at least a short security review is something that we should avoid at all costs.

Even a simple grep (which has cost me one second) has shown one potential attack vector to me: JamesHeinrich/getID3@dc85490

The recent problems within the documents app has shown that this is something where we have really place to improve.

[leizh]

Hi,

I have worked on my fork on some features to improve large collection support. I will explain them here.

• Register a command to rescan the music files from the owncloud console. #158 (already merged) :
  Register a command to rescan the music files from the owncloud console (wiki). This is usefull when the collection is too large to be scanned by the first request to the music app.
• Search provider #159 (already merged) : Allows to search in the collection in the OwnCloud search area. To implement this, I have redirected the play requests to changes of the hash. I have removed the routing mechanism of angular because it implied to reload all the scopes.
• leizh/owncloud collection-path (merged in Collection path #165) : Allows to filter the collection by a path specified in the user settings (because I don't want not well tagged tracks in my collection). In this branch, I have done huge changes to how the collection is retrieved. A new REST handler 'collection' has been created which retrieves the tree of all the collection in 3 sql statements (instead of 1 + nbArtists + nbAlbums + nbTracks) and with a minimal data amount. Results for 7000 tracks : 21 sec => 6 sec, 3 MB => 1.5 MB.
• leizh/owncloud collapsed-artists (opened PR Collapsed artists #166 - triaging state): Show/hide the albums by click on the artist name. I wanted animations so I upgraded angular to the last stable version (1.2.14) and included the animate-module.

At this point, I stopped to create branches for each features because it was very difficult to merge.

• 65445ca (merged 6a35f48...601e299): I have more than one image in my album directories. This commit allow to select in first place the images named "front_" and "cover_". I haven't took the time to create a setting entry for this.
• 1317f93 (merged de85928): I have remove the bindings to the current position. This caused huge lags on the web interface. I supposed that angular tested all the binding updates every second.

Not related to large collections :

• 8cec86b (merged 2a64d51): Better duration display.
• 62533d6 (merged 5b24129): Allows seeking by click on the progress bar.

I will stop to work on this app for a while. Feel free to merge this features.

[MorrisJobke]

@leizh You are so awesome! I will try to merge as many as possible. Thanks.

[MorrisJobke]

@leizh How can I give you credits for this changes? Feel free to open a PR for https://github.com/owncloud/music/blob/master/AUTHORS

[leizh]

(I have updated my comment while you're wrote yours).

This is not very important to me. You can add a "Leizh [email protected]" line in AUTHOR if you want.

Take the time to review the commits, some can not please you and some require settings entries (collapsed-authors and the album cover filenames for example).

[MorrisJobke]

I added checkmarks to your comment to mark merged stuff

[MorrisJobke]

@leizh 7 out of the 8 changes are merged into master. The 8th is the collapsing of the artists and maybe needs a polishing but I prepared a PR for this.

[leizh]

Thanks for the merging of all of this.

[HLFH]

@MorrisJobke Any updates? :)

[MorrisJobke]

Sadly not ... I currently have not enough time for this :(

[HLFH]

@MorrisJobke @butonic Is there a way to merge https://github.com/libasys/audios and ownCloud Music? ownCloud Music has a working Ampache backend but is unusable for large collections, Audios ownCloud plugin is usable for large collections but does not have any Ampache backend.

[MorrisJobke]

We plan to join forces.

[pellaeon]

@MorrisJobke Any updates? How do you plan to solve problems with large collection? It's really a pain for me, so I want to help.
Currently I find 2 bottlenecks, one is /api/collection and another /api/scan?dry=0, both seems to be caused by slow database queries. Perhaps it could be solved by memcache?

[arpee]

Would also like to bump on this issue.

I get unresponsive pages since the application finished scanning my collection.
(Stuck with a loading gif.)

[tetzlav]

+1
unusable slow with 10000+ songs... :(

[HLFH]

@tetzlav You should use Plex or Emby for multimedia (movies, tv shows, music) as @MorrisJobke does. For this moment, it's the better way.

[ghost]

You do not need 10k songs to slow down this. I have 3k and it's unusable. Press play on a song and wait minutes to play. UI takes minutes to complete scanning (which it does too often). Gap of several minutes between songs. No way to browse by album. Would love to see this app spread it's wings.

[bee272]

Great app but....
+1
unusable with 10000+ songs... :(

[paulijar]

The PR #564 improving the performance especially with large collections was merged yesterday. If you are willing to test with development version, I would be interested to hear how it works now with large real-life music collections and what is the practical collection size limit for usability. I would recommend using Chrome browser with large music collections.

[paulijar]

The Music app version 0.4.1 containing #564 has been released to ownCloud marketplace and Nextcloud app store. There's always room for improvement on performance, but IMHO, the app is now quite usable at least with several thousands of tracks. Obviously, it will still probably become unusable if you add e.g. 100k tracks. I'm closing now this very old issue. Feel free to report a new issue if you still find it unusable with your music collection.