Toughts on all new Jekyll pagination system

[gynter]

The post below is long, longer than I tought it'll be, but please read it all 😄

I hope that others including the maintainers of the project, have similar toughts on the topic and see it as a way to go forward with the project making it more feature rich and customizeable even for more complex web pages. My intention is not to fragmentize the community, thats why I posted my ideas here instead of creating another fork and just doing "my stuff".

Why I'm doing this?

Hello, like others here I'm dedicated Jekyll user and have built many websites, including my own, on Jekyll.

The most time consuming part was to find a suitable solution to produce multilingual site with customized and filtered pagination pages. Currently Jekyll pagination has no support for that. Yes, there are a lot of alternatives, each of them have different goal, but any of them exactly suitable for my needs - lack of documentation and sometimes hard or even overkill to implement.

I propose to write all new Jekyll pagination system, from scratch. Some of You might ask, why write here, why I don't just fork it and make my changes? Well I kind of did that already, but this is not the solution.

In my opinion projects like this should be kept together and developed as one which guarantees one good source of the software.

My idea is to start from scratch (we could reuse useful code), create an experimental repository, but first we should map our needs, toughts and other hardcore awesome ideas.

The ideas listed below are my personal toughts on the subject, feel free to express Your toughts. The ideas are based on my experience and from issues of the project.

If maintainers think that I'm overstepping my boundaries (I hope that I'm not 😃), feel free to close and lock this issue.

What I would like to achieve?

Architecture

Current architecture is limited only to posts. Referring to #22 I think it should be able to paginate all Jekyll content types: pages, posts and collections.

Code clarity

Make classes independent (#26). Currently there's huge hell in Pager class. The class should be completely independent and all variables should be passed to it as separate and well defined arguments.

Layouts/templates

Jekyll has a well defined layout system and for clarity we should use it instead of defining html files directly and relative of the root directory. Defining layout like myblog whould use template _layouts/myblog.html.

Pemalinks

With this way we should also separate layout from permalinks (#22). This would make a life lot of easier for multilingual page. Default permalink should point to the layout, if layout is myblog then permalink is /myblog:num. The first page is /myblog.html unless, there is a missing index.html in the same directory, then the first page is /index.html. Other pages would still be /myblog2/index.html, /myblog3/index.html etc.

Filtering

When converting my site to multilingual I researched some other plug-ins which added support for pagination by category. Since Jekyll is in my opinion pretty abstract we shouldn't focus on one filter variable category, but it should be configureable (for example, I filter my posts using language). We should also keep in mind that the frontmatter variables can have multiple values so support for lists as filter value is needed (eg filter posts by language values [ "et_EE", "en_US" ]). Also support for both include and exclude filters would be a good feature (include all category X, but exclude if also in category Y).

Sorting

Custom sorting would be awesome feature. The default should be same behavior as now (by date descending), but we should make it configurable, so we could use other fontmatter variables and values for sorting, try to detect automatically if alphabetically or chronologically. For pages and collections the default should by filename alphabetically ascending.

Avoiding existing mistakes

Both open and closed issues should be taken in consideration when building new system so we don't step on the same rake over and over again.

Compatibility

Everything must be backwards compatible. The new code must support the old config layout otherwise a lot of web pages will break and that's not acceptable.

Proposed config syntax (based on #22)

I'm proposing the configuration syntax below.

pagination:
  # Allow setting global default limits for all paginated pages
  limit: 0 # Limit how many content objects to paginate (default: 0, means all)
  page_limit: 10 # How many objects per page, used to be `paginate` (default: 0, means all)
  filter:
    # If filter is not specified or is null then all objects are
    # considered included
    include: 
      locale: en_US

  posts:
    # The global settings `limit` and `page_limit` are included here
    # but can be overwritten
    - page_limit: 15
      layout: blog # Which layout to use from `_layouts/`
      # permalink: /blog:num # Permalink syntax, used to be `paginate_path` (the default is taken from layout)
      filter:
        include:
          # en_US is included from global filters settings, et_EE and 
          # de_DE are appended.
          locale:
            - et_EE
            - de_DE
        exclude:
          category:
            - drafts
    - layout: blog_et_EE # Another posts paginator.
      permalink: /et/blog/:num # Note: /et/blog/:num and /et/blog:num are behaving differently
      filter:
        include: et_EE
        exclude: en_US
      sort:
        date: asc

  pages:
    - layout: pages
      permalink: /pages/:num
      sort:
        title: desc

  collections:
    # The collections are defined here not on the root node, because
    # this behavior would eliminate conflicts with global pagination
    # settings
    - authors:
        limit: 50
        # layout: authors # By default, taken from collection name
        # permalink: /authors:num # By default, taken from collection name
        filter: null # Disable filtering, otherwise global filters will apply

Note, that if posts, pages or collections are not defined as lists, then those objects are not considered as multipagination (if you only want one pagination page, then You can use associative arrays directly as those node's values:

pagination:
  posts:
   layout: blog

The code above would also be the shortest code to enable pagination for posts. There is no need to define permalink here and if You don't have /index.html then default permalink is considered to be that, but if You do, then first page will be /blog.html (see the Permalinks paragraph).

How could I achieve that?

As stated in the introduction, the idea is to create an experimental repository (milestone 2.0.0?) which would contain all the new Jekyll pagination. We all have our daily jobs to do, so this probably won't be finished this month, not even next or even might not be in this year, but we would have a straight goal and sight where to go next. All useful contributions from everyone would be welcome 😄 .

How can You help?

I'm slightly new to Ruby world, but I have extensive and long time knowledge of PHP, Python, various *nix shells' script languages etc. To be honest those two commits from my fork's experimental branch are basically my first serious contact with Ruby world.

Note: I don not encourage nor recommend to use this code in production, it's not a solution to existing issues, it's a hack which I quickly made to add multilingual support to my site.

Yes, I can write the code but I have no knowledge of other Ruby stuff including inline documenting, unit testing, packaging and distribution - so thats where the most of help would be needed.

[gynter]

Just to clarify, my intention is not to hijack or maintain the project, but if You share my ideas then would be good if the maintainers would create some new branch (eg milestone-2.0.0) and then merge my PRs to it.

[feryardiant]

👍

[parkr]

Feel free to build it, test it, and share widely. I worked on an idea in #22 but I just can't get excited about pagination. My vote is to kill off this version and move in a new vision that is as easy as possible to use, but still flexible.

[gynter]

Imo the hardest part with the pagination is the templating which gets pretty complicated if You want to create more awesome paginators. But I can't see any other way than making lots of examples, to make it easier.

[d3netxer]

so there is no way right now to currently to paginate posts in certain categories without a plug-in correct?

[gynter]

@d3netxer, correct.

[d3netxer]

well, if this was built-in to Jekyll it probably would have save me 6 plus hours. What I ended up doing was making a json feed of the posts with the category of 'news'. I then used Javascript to create html templates of the posts from the json feed and display them. I used a little bit more javascript and Jquery to load more entries in an AJAX fashion.

The good news is that it is nice to have a json feed anyway of my posts, and I like how the AJAX responds.

[entozoon]

Anyone know a good plugin that achieves the above problem (paginate posts in certain categories) ?

[gynter]

@entozoon https://github.com/octopress/paginate

[stevecrozz]

@gynter are you still actively working on this?

[neortls007idev]

I think this would work best if its done something like this but it requires more documentation for newbies.
http://anjesh.github.io/2015/01/25/collection-pagination-working-github-pages/

just so that if the link gets moved.

I enabled the collection in the _config.yml and created the folder _tripcollection in the root.

collections:
tripcollection:
output: true
permalink: /:collection/:title/
I added posts in the tripcollection folder and added the following code to get the previous and next links in the collection post in the trip layout file.

{% for c in site.tripcollection %}
{% if c.title == page.title %}
{% assign thisPost = c %}
{% if forloop.index == 1 %}
{% assign prevflag = 0 %}
{% assign nextflag = 1 %}
{% elsif forloop.index == forloop.length %}
{% assign prevflag = 1 %}
{% assign nextflag = 0 %}
{% else %}
{% assign prevflag = 1 %}
{% assign nextflag = 1 %}
{% endif %}
{% endif %}
{% endfor %}

{% for c in site.tripcollection %}
{% if c.title == page.title %}
{% assign prevflag = 0 %}
{% endif %}
{% if prevflag == 1 %}
{% assign prevPost = c %}
{% assign page.previous = c %}
{% endif %}
{% endfor %}

{% if nextflag == 1 %}
{% for c in site.tripcollection %}
{% if foundPost == 1 %}
{% assign getNext = 1 %}
{% endif %}
{% if c.title == page.title %}
{% assign foundPost = 1 %}
{% endif %}
{% if getNext == 1%}
{% assign nextPost = c %}
{% assign page.next = c %}
{% assign foundPost = 0 %}
{% assign getNext = 0 %}
{% endif %}
{% endfor %}
{% endif %}

{% if prevPost.url %} < {{prevPost.title}} {% endif %} {% if nextPost.url %} {{nextPost.title}} > {% endif %}

I see you raised your eyebrows looking at the code above. It works and got it working after number of hit and trials. It’s just pure logic, it looked complicated because of lack of language constructs in liquid template.

Now you get previous and next links in your collection posts as well. See the working demo here.
http://pagination-demo.github.io/tripcollection/trip-1/

[Joshfindit]

It's hinted at in the suggestion, but it sounds like realistically @gynter is suggesting a new version of Jekyll that's a rewrite.
I support this idea.

Edit: I mistakenly called this "Jekyll 2.0" referring to the type of versioning system where major point releases are complete rewrites. Jekyll, of course has an actual v2.0 that is not based on this system.

[neortls007idev]

No you see the the pagination feature is completely disabled and only pure logic is used so if they give a standard for pagination using some more predefined variables with a highly explained documentation in layman terms it should be easy.

I don't know how it was for jekyll 2.0 i have used jekyll only for the last two weeks.

[sverrirs]

Hey everybody, I know this is an old thread but I wrote a pagination plugin for Jekyll that supports all of the features described in this post. Hoping that it might be a viable alternative for the old jekyll-paginate gem https://github.com/sverrirs/jekyll-paginate-v2

[stevecrozz]

@sverrirs I think this is worth exploring.

But your new gem has a bit of chicken-and-egg problem. Its hard to get people to care about a new (better) gem that nobody uses when we could use a (maybe worse) gem that everyone uses. Can you show some real public web sites that are currently using your pagination plugin? Its a really hard sell without that.

Are you really committed to pouring in the time it takes to make this effort successful? Lots of folks pop up on github with something half-baked and 6 months later abandon their project. You could literally be signing up for years of finding/fixing annoying little bugs and arguing with other people on github.

If you're still on board, I'd love for you take this on and I'll test your plugin with my own jekyll site and give you feedback.

[sverrirs]

@stevecrozz Thanks for the comment and you're right in a sobering way :)

My motivation for building this plugin/gem is to be able to provide a viable migration alternative from Wordpress for a client of mine (pitching Jekyll as the alternative). So I have a vested interest in seeing this through. I currently have my own personal blog using this paginator logic but it doesn't really have a multi category structure or some of the more advanced features this plugin supports. But at least one system has it, but then again I only completed the paginator code this morning.

I am committed to building a viable production-ready alternative to the old paginator but my ultimate goal is to eventually hand this repo over to the Jekyll team if they deem it worthy (and hopefully gather some contributors along the way). But I'm in for the long haul.

At this point my main focus is on hardening the code and the feature set that it currently has. Then publish the first version of the gem. Unless critical to the plugin I would like to put additional "nice-to-have" features on standby until these initial goals are properly completed.

I'd be glad and grateful to have you help me testing this.

[parkr]

@sverrirs Once you feel your updated version is stable and ready for prime-time, I could be convinced to give you the keys to this repo (jekyll/jekyll-paginate) and/or allow you put your repo in this project and make it the new default Jekyll pagination solution. We could even see about getting it onto GitHub Pages to replace the current jekyll-paginate if it's compatible with jekyll-paginate.

[sverrirs]

@parkr Thank you Parker. I will be in touch.
I just managed to package this whole thing as a gem now but I need to work on tests to avoid regressions before I can confidentially say that we're ready for the feature/bug/issue deluge that would be jekyll/jekyll-paginate hehe ;)

[sverrirs]

@stevecrozz,. @parkr and everybody else.
I've just finished producing a fully backwards compatible version (v1.3.1) of the new pagination gem. It is now a zero-config in place replacement for the old jekyll-paginate gem. I would really welcome any and all testing that you guys could help me out with both the new and more importantly the old compatibility logic. https://rubygems.org/gems/jekyll-paginate-v2/versions/1.3.1

The latest stable release before compatibility code was added is v1.2.0 (https://rubygems.org/gems/jekyll-paginate-v2/versions/1.2.0)

[stevecrozz]

@sverrirs I got around to upgrading my blog to Jekyll v3 and included switching to jekyll-paginate-v2 in the process. There were plenty of issues I encountered during the upgrade, but jekyll-paginate-v2 was one of the simplest parts of the process. I did make a few mistakes and if you're open to discussing it, we could talk over an idea I've got on how to make debugging a little easier. You can send me an email if you want to talk that over.

One thing you might find interesting is my implementation of tag indexes (https://github.com/stevecrozz/lithostech.com/blob/4f6ac029b71a01b459a860739c5e4297664fd9e1/_plugins/tag_indexes.rb) which I thought would be one of the trickier parts of the process because of the way I had been abusing the old jekyll-paginate. But my new solution ended up being even simpler than the previous version. I even found that this new tag index implementation based on jekyll-paginate-v2 fixes a bug that hadn't noticed before.

Granted, my blog is a pretty simple project, but my results were good and I plan to keep it on jekyll-paginate-v2. Thanks for sharing your project and being willing to support a new pagination system for jekyll!

[sverrirs]

@stevecrozz Thank you for the kind words. I'm really glad that the experience with the pagination code was good.

I would really love some feedback on the whole project. If you feel that your comments are appropriate for the general public then we can discuss it openly at https://github.com/sverrirs/jekyll-paginate-v2/issues otherwise I am more than willing to have a one-on-one through email. Ping me your thoughts at [email protected].

Also, thank you for the tag_indexes.rb link. It is a really neatly implemented solution 👍 I have been thinking about how to do this cleanly for categories/tags/collections incorporating the paginate-v2 logic. Do you mind if I take this code and expand on it and perhaps publish it as an accompanying gem for the pagination-v2 gem?

[stevecrozz]

@sverrirs You're certainly welcome to take and use/extend/publish any code you find in my blog. Cheers!