Fixing preview caching

[oparoz]

This is a meta ticket used to reference all the various problems that need sorting so that preview generation and caching works quickly in all situations. The current solution is unbearably slow as soon as large images are involved.

Current situation

Asking for the first preview

This can be a Files thumbnail or a Gallery image per example, depending on how the picture was uploaded

The preview class collapses #13493

• 9.0 if applying a simple local fix.
• 9.0 if waiting on the move of the preview system to the AppFramework use appframework for preview system #13610. This would not solve all problems as some apps use the AppFramwork's PreviewManager internally

Large previews are returned by some preview providers #13607

• 8.1 Introduce maximum dimensions in the config (PR Introducing the maximum size preview #13674)
• 8.1 Force bitmap provider to use that setting (PR The bitmap preview class now takes maxX and maxY into consideration #13635)

Thumbnails of large files result in a time-out #13218

• 8.0 Introduce a max file size setting (PR add config-option for an image's maximum filesize when generating previews #13449)
• 8.0 Fix the PHP Fatal error (PR remove insane debug-log from OC_Image #13614)

Save memory by streaming large previews straight from disk #13611

• 9.0 The preview system needs to move to the AppFramework (PR use appframework for preview system #13610)
• 8.1 Waiting for StreamedResponse to land in AppFramework (PR AppFramework StreamResponse #13616)
• Use StreamedResponse

Asking for a new preview of different size

Caching seems to not be working #13516 #8196

Many PRs touching on thumbnail generation performance are affected by this, but the people most affected are the ones using large images, especially in formats which need to be converted by ImageMagick

• 8.1 Cache the largest possible preview and use that for all subsequent resizing requests (PR Introducing the maximum size preview #13674)
• 9.0 Use preview categories to fulfil requests

How things will work

Once all those issues are fixed, we should have something that works like this

Asking for the first preview

• Is size less than 50MB? Only process those
• Do we have a max preview? If not, ask the provider for a 2048x2048 preview and it will return exactly that
• Store that max preview, resize it to fulfil the request and send it back

Asking for a new preview of different size

• Is a preview of that exact dimension stored already? Use that
• If not, do we have a preview for this preview category? Use that
• If not let's create a new preview for that category from the max preview and store it in the cache

@georgehrke Feel free to add any missing issues and PRs.
People who might be interested in this: @libasys @icewind1991

[georgehrke]

Once enhancement we could add:

• OC\Preview\Provider::getThumbnail will get a new parameter: a file handle

Instead of returning the OC_Image object, write the preview to the handle and just return true/false.

Any opinions?

[georgehrke]

Another enhancement would be stream-based response types, that won't require us to load the cached preview into memory.
This was also requested here: #12988, not sure if there is a dedicated issue yet.

[georgehrke]

• use app framework for preview system

[oparoz]

Added StreamedResponse to the OP.
Regarding the handle, I'm not sure it would make a difference since the first thing the Image class is doing is loading the file to process it? What am I missing?

[georgehrke]

I'm not sure it would make a difference since the first thing the Image class is doing is loading the file to process it?

• There are preview generators, e.g. movies, that generate the png not using php.
• Even if the preview generators use OC_Image, this way (combined with StreamedResponses) we would release the used memory earlier.

[oparoz]

I guess if OC_Image doesn't need to load the data in memory to do its resizing, it would be a good thing to have, yes. We can already do that for some providers by using OC_Image::loadFromFileHandle no?

[oparoz]

Previews generated by the bitmap preview class are now limited in their maximum dimensions: #13635

Affects:

• PDF
• Photoshop
• Illustrator
• Postscript
• TIFF

[oparoz]

Most of it should be done by the time 8.1 launches if all the PR get merged, but some changes will have to wait for 8.2

[PVince81]

@oparoz move to 9.0 as we're past feature freeze ?

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[PVince81]

@DeepDiver1975 since you were working on preview stuff.

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[stale]

This issue has been automatically closed.