Use web-thumbnailer to retrieve thumbnails #687

ArthurHoaro · 2016-11-09T18:24:22Z

Disclaimer:

this a proposition. ping @virtualtam @nodiscc

it requires PHP 5.6 (see Drop PHP 5.3/5.4 compatibility #599)

Hi guys! I've tried to refactor thumbnails, and then end up writing web-thumbnailer. It goes a bit beyond Shaarli needs but I still think we could use it as a dependency to retrieve thumbnail - see the readme for more info.

This PR shows how to integrate web-thumbnailer within Shaarli. So, what changes:

a lot more thumbnails are retrieved
use blazy on linklist to lazy load images
thumbnails can be disabled
thumbs size is now 120x120
thumbs are now cropped to fit the expected size
again, requires PHP 5.6

Fixes #345 #425 #487 #543 #588 #590

PS: a few things still need to be fixed, like a release on the WT repo.

EDIT: Fixes #712 - all thumbnails are now cached, no more hotlinking.
Fixes #1056

nodiscc · 2016-11-11T01:44:05Z

I have just read the patch comments, had a quick look at the implementation in Shaarli, and at the readme in https://github.com/ArthurHoaro/web-thumbnailer, and I would very much welcome this change. Lacking time to do a full code review though :/

Thanks

kalvn · 2017-11-02T19:50:44Z

May I suggest to add a way for templates to retrieve thumbnails direct URL? Or an array with several useful parameters such as what computeThumbnail() function does.

ArthurHoaro · 2017-11-03T16:49:02Z

@kalvn What's your use case?

kalvn · 2017-11-03T17:40:45Z

@ArthurHoaro the idea would be to allow more flexibility in the way templates display the thumbnail (with background-image instead of for example). The current way is a bit rigid.

ArthurHoaro · 2017-11-11T15:44:48Z

@kalvn Each links will have a thumbnail field containing the thumbnail URL, or false if there isn't any.

PR updated and ready for v0.10!

nicolasdanelon · 2017-11-13T13:33:48Z

tpl/default/js/shaarli.js

@@ -321,8 +321,8 @@ window.onload = function () {
    /**
     * bLazy trigger
     */
-    var picwall = document.getElementById('picwall_container');
-    if (picwall != null) {
+    var blazyElements = document.getElementsByClassName('.b-lazy');


be careful ..
https://caniuse.com/#feat=getelementsbyclassname :)

Good to know... I've used it multiple times in this file.

nicolasdanelon · 2017-11-13T13:34:58Z

tests/ThumbnailerTest.php

+
+        $this->thumbnailer = new Thumbnailer($conf);
+        // cache files in the sandbox
+        \WebThumbnailer\Application\ConfigManager::addFile('tests/utils/config/wt.json');


maybe you should use a use keyword? like you do in ConfigManager

nodiscc · 2017-11-13T17:50:08Z

Each links will have a thumbnail field containing the thumbnail URL, or false if there isn't any.

Very nice.

I think we should document the list of fields somewhere, perhaps as part of a https://github.com/shaarli/Shaarli/blob/master/doc/md/Directory-structure.md update (new Datastore structure section). I thought there already was info on this somewhere, but can't find it. As far as I can tell currently Shaarli handles the following fields

created (??)
description (string)
id (integer)
private (boolean)
shorturl (string, 6-char long)
tags (array of strings)
title (string)
updated (???)
url (string)

add thumbnails to that.

ArthurHoaro · 2017-11-14T17:58:43Z

@nodiscc Good idea, I'll add that. And dates are now stored as DateTime objects.

nodiscc · 2017-11-16T17:36:44Z

index.php

@@ -768,22 +769,51 @@ function renderPage($conf, $pluginManager, $LINKSDB, $history, $sessionManager)
    // -------- Picture wall
    if ($targetPage == Router::$PAGE_PICWALL)
    {
+        if (! $conf->get('thumbnails.enabled')) {
+            header('Location: ?');


Maybe display a warning message to the user Picture wall unavailable (thumbnails are disabled). Or even better, hide the Picture wall button from the toolbar.

Good idea. I did both.

* requires PHP 5.6 * use blazy on linklist since a lot more thumbs are retrieved * thumbnails can be disabled * thumbs size is now 120x120 * thumbs are now cropped to fit the expected size Fixes shaarli#345 shaarli#425 shaarli#487 shaarli#543 shaarli#588 shaarli#590

Upgrade web-thumbnailer and display thumbs right after download

…plates

…dia websites

ArthurHoaro · 2018-07-05T18:39:28Z

I think I'm good here. I've:

Fixed the issue with Apache userdir
Auto-generate a .htaccess file in thumbs/ cache folder which grants direct access
Move the sync to a dedicated page through AJAX requests (the picwall no longer trigger retrieval)
Added a common mode which only retrieve thumbs from a list of websites
Rebased the branch

As usual, clear the cache folder to test it. Regarding the milestone, since we didn't released v0.10 yet, maybe we can merge it before (depending on the feedback review of course).

EDIT: if you already ran the updater with previous commits, you might need to remove thumbnails section from the config file also.

virtualtam · 2018-07-05T23:49:07Z

Neat, I'll test it to(day|morrow) :)

virtualtam · 2018-07-10T13:57:11Z

Currently testing :)

thumbs are correctly displayed when serving in a user-specific path (http://domain.tld/~user/shaarli)
the new option to define which thumbnails are fetched is convenient, maybe it should default to "Only common media hosts" to stay consistent with the current behaviour
the thumbnail update page is a nice addition, that should appear more prominently:
- as a button in the Tools page (I actually missed it in its current form :p )
- as a button in the Picture Wall page if the picture cache is empty
the thumbnails seem to be fetched only when browsing the linklist; browsing the picture wall has no effect

The following is most likely due to removing the thumbnails section under data/config.json.php:

the Picture Wall looks nicer with square thumbnails
on the linklist, the thumbnail div does not respect its parent container's geometry, see screenshot below:

Here is the resulting configuration section, I think it misses the width/height settings:

    "thumbnails": {
        "mode": "common"
    }

ArthurHoaro · 2018-07-10T16:22:09Z

the new option to define which thumbnails are fetched is convenient, maybe it should default to "Only common media hosts" to stay consistent with the current behaviour

I'm not sure. The goal behind this PR, beside having a cleaner solution, is to provide a more complete feature. I feel like this option is nice to have for people who want a lighter solution, but displaying thumbnails for any website which provides an OpenGraph image is a good default option.

I don't really know what users would prefer, and this can easily be changed though, so any other opinion is welcome here.

the thumbnail update page is a nice addition, that should appear more prominently

It should display a big pure-alert warning when you run the update or change the setting.

as a button in the Tools page (I actually missed it in its current form :p )
as a button in the Picture Wall page if the picture cache is empty

Sure, I'll add those.

For the last part:

I'll check the size settings update. I believe I never reset them after the update. And you probably already ran the update during your previous tests.
I'll fix the CSS issue if the thumb is too high.

Thanks again for the quick test 👍

nodiscc · 2018-07-12T20:33:11Z

index.php

+            $thumbnailsMode = extension_loaded('gd') ? $_POST['enableThumbnails'] : Thumbnailer::MODE_NONE;
+            if ($conf->get('thumbnails.enabled', Thumbnailer::MODE_NONE) !== Thumbnailer::MODE_NONE) {
+                $_SESSION['warnings'][] = t(
+                    'You have enabled or changed thumbnails mode. <a href="?do=thumbs_update">Please synchonize them</a>.'


s/synchonize/synchronize

nodiscc · 2018-07-12T20:38:15Z

inc/web-thumbnailer.json

+{
+  "settings": {
+    "default": {
+      "download_mode": "HOTLINK",


Why not set the download mode to DOWNLOAD ?

Good catch, there was actually an issue with WT where this setting would be ignored.

- add a default thumb size value (125x90px) - improve private vertical bar visual, especially with thumbnails - translations - add a sync thumbs button in tool and empty picwall page - fixes WT download mode in JSON config

ArthurHoaro · 2018-07-17T11:18:10Z

Bunch of improvements for thumbnails integration:

add a default thumb size value (125x90px)
improve private vertical bar visual, especially with thumbnails
translations
add a sync thumbs button in tool and empty picwall page
fixes WT download mode in JSON config

I've changed a bit the way the orange bar for private links is displayed, mostly because it was truncated with floating thumbnails and no description. I went for an easier solution, and IMO it looks better. Example:

virtualtam

Great work @ArthurHoaro 🎉

Thumbnail generation and display went smoothly, I think we can merge this PR and handle improvements in future issues/PRs

ArthurHoaro · 2018-07-28T07:41:14Z

Thank you @virtualtam and everyone for your time and reviews on this looong PR. Let's merge it! 🎉

nicolasdanelon · 2018-07-30T18:35:37Z

hi, I have a problem with thumbnails..

my list of links and the detail of a link refers to public_html/links/cache/thumb/14dd5266c70789bdc806364df4586335/3a9726225cd4ed259458afa90126bb1509eea4a01259010.jpg

the link should be links/cache/thumb/14dd5266c70789bdc806364df4586335/3a9726225cd4ed259458afa90126bb1509eea4a01259010.jpg

my shaarli is updated like this

$ cd /home/username/public_html/links
$ git checkout master
$ git pull --prune
$ composer install --no-dev --prefer-dist
$ yarn install
$ yarn run build

nginx  1.12.1
php-fpm 5.6.36-1

$ node -v
v6.14.2
$ yarn -v
1.9.2
$ npm -v
3.10.10

ArthurHoaro added 3rd party interoperability with third-party platforms discussion feedback needed in progress template HTML rendering thumbnail media thumbnails labels Nov 9, 2016

ArthurHoaro added this to the 0.10.0 milestone Nov 9, 2016

ArthurHoaro self-assigned this Nov 9, 2016

ArthurHoaro force-pushed the web-thumb branch from 48f0865 to 69eaabe Compare December 8, 2016 19:54

nodiscc mentioned this pull request Dec 14, 2016

Option to block non-https thumbnails #712

Closed

virtualtam mentioned this pull request Jan 6, 2017

Cleanup: use safe boolean comparisons #748

Merged

ArthurHoaro force-pushed the web-thumb branch 5 times, most recently from 97db222 to 9b49f9b Compare November 11, 2017 15:42

ArthurHoaro added in review and removed discussion feedback needed in progress labels Nov 11, 2017

nicolasdanelon reviewed Nov 13, 2017

View reviewed changes

nodiscc added the documentation label Nov 13, 2017

nodiscc reviewed Nov 16, 2017

View reviewed changes

nodiscc added this to the 0.11.0 milestone Jun 21, 2018

ArthurHoaro force-pushed the web-thumb branch from bcfd2bd to 2911136 Compare July 5, 2018 18:30

ArthurHoaro added 8 commits July 5, 2018 20:31

ConfigManager: add a method to remove an entry

a372471

Update thumbnail integration after rebasing the branch

e85b7a0

Add Link structure page to the documentation

8b5b7dc

Take code review into account

787faa4

Upgrade web-thumbnailer and display thumbs right after download

Add a page to update all thumbnails through AJAX requests in both tem…

28f2652

…plates

Bump WT version

fcba541

Thumbnails: add a common mode to only retrieve thumbs from popular me…

b302b3c

…dia websites

ArthurHoaro force-pushed the web-thumb branch from 2911136 to b302b3c Compare July 5, 2018 18:34

nodiscc reviewed Jul 12, 2018

View reviewed changes

Bunch of improvement for thumbnails integration:

7b4fea0

- add a default thumb size value (125x90px) - improve private vertical bar visual, especially with thumbnails - translations - add a sync thumbs button in tool and empty picwall page - fixes WT download mode in JSON config

ArthurHoaro force-pushed the web-thumb branch from 72303fb to 7b4fea0 Compare July 17, 2018 11:17

virtualtam approved these changes Jul 27, 2018

View reviewed changes

ArthurHoaro merged commit ad5f47a into shaarli:master Jul 28, 2018

ArthurHoaro deleted the web-thumb branch July 28, 2018 07:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use web-thumbnailer to retrieve thumbnails #687

Use web-thumbnailer to retrieve thumbnails #687

ArthurHoaro commented Nov 9, 2016 •

edited

Loading

nodiscc commented Nov 11, 2016

kalvn commented Nov 2, 2017

ArthurHoaro commented Nov 3, 2017

kalvn commented Nov 3, 2017

ArthurHoaro commented Nov 11, 2017

nicolasdanelon Nov 13, 2017

ArthurHoaro Nov 14, 2017

nicolasdanelon Nov 13, 2017

nodiscc commented Nov 13, 2017 •

edited by ArthurHoaro

Loading

ArthurHoaro commented Nov 14, 2017

nodiscc Nov 16, 2017

ArthurHoaro Nov 17, 2017

ArthurHoaro commented Jul 5, 2018 •

edited

Loading

virtualtam commented Jul 5, 2018

virtualtam commented Jul 10, 2018 •

edited

Loading

ArthurHoaro commented Jul 10, 2018

nodiscc Jul 12, 2018

nodiscc Jul 12, 2018

ArthurHoaro Jul 17, 2018

ArthurHoaro commented Jul 17, 2018

virtualtam left a comment

ArthurHoaro commented Jul 28, 2018

nicolasdanelon commented Jul 30, 2018 •

edited

Loading

Use web-thumbnailer to retrieve thumbnails #687

Use web-thumbnailer to retrieve thumbnails #687

Conversation

ArthurHoaro commented Nov 9, 2016 • edited Loading

nodiscc commented Nov 11, 2016

kalvn commented Nov 2, 2017

ArthurHoaro commented Nov 3, 2017

kalvn commented Nov 3, 2017

ArthurHoaro commented Nov 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nodiscc commented Nov 13, 2017 • edited by ArthurHoaro Loading

ArthurHoaro commented Nov 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArthurHoaro commented Jul 5, 2018 • edited Loading

virtualtam commented Jul 5, 2018

virtualtam commented Jul 10, 2018 • edited Loading

ArthurHoaro commented Jul 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArthurHoaro commented Jul 17, 2018

virtualtam left a comment

Choose a reason for hiding this comment

ArthurHoaro commented Jul 28, 2018

nicolasdanelon commented Jul 30, 2018 • edited Loading

my shaarli is updated like this

ArthurHoaro commented Nov 9, 2016 •

edited

Loading

nodiscc commented Nov 13, 2017 •

edited by ArthurHoaro

Loading

ArthurHoaro commented Jul 5, 2018 •

edited

Loading

virtualtam commented Jul 10, 2018 •

edited

Loading

nicolasdanelon commented Jul 30, 2018 •

edited

Loading