Multi-level navigation

[pdmosses]

This PR is intended to supersede #192, and incorporates #448. It aims to fix #219, #447, and to respond to the discussion in #425 and in the comments on #192.

A summary of the background for this PR, and an overview of the code, are available here.

The rec-nav-2 branch used for this PR is based on the latest an outdated release of the theme (v0.3.3, October 2020).

To test rec-nav-2 locally as a remote theme:

• Specify in _config.yml:

  remote_theme: pdmosses/just-the-docs@rec-nav-2
  

• Specify in Gemfile:

  gem "just-the-docs", github: "pdmosses/just-the-docs", branch: "rec-nav-2"
  

• Run:

  bundle update
  bundle exec jekyll serve
  

The proposed updated documentation pages for the theme navigation structure are available here.

To serve the rec-nav-2 documentation pages locally and display the results of the regression tests:

• Download the rec-nav-2 branch
• Run:

  bundle update
  bundle exec jekyll serve --config _config.yml,_tests.yml
  

[pdmosses]

The last three commits above remove the need for avoiding a special character (^) in titles, and add regression tests where a page title is a prefix or suffix of a potential ancestor title.

[jtebert]

From a quick test, I believe that this also fixes the issue I opened yesterday (#492) about incorrect breadcrumbs and table of contents in collections.

[pdmosses]

@jtebert thanks for testing this PR with multiple collections – I will add them to the regression tests.

[pdmosses]

N.B. If further collections are added, the TOC/breadcrumbs links in the Tests collection will be incorrect, as reported in #492.
The added tests for multiple collections confirm that this fixes #492.

[pyorot]

If I rocket-react then hopefully this gets reviewed sooner :D
I'd be interested to know the roadmap for the project, what the priorities are, and if it's still active; maybe I should @ the maintainer?
Also interested in if there's an example website based on this PR, and how ready for use it is. I was hoping it'd be merged before I needed it, but it's been 5 months since the last commit.

[adfoster-r7]

I haven't investigated the specific performance bottlenecks; but I wondered if the performance could be improved with a custom opt-in _plugin. i.e. the slow parts could be handled directly in Ruby instead of liquid. It would have to be opt-in functionality though if the plugin worked - as I believe by default plugins aren't supported on Github's default gh-pages, but they're available in Github actions, folk's own CI pipelines, or local dev environments

[pdmosses]

@adfoster-r7 thanks for the suggestion. However, I'm not a Ruby programmer, so I wouldn't develop such a plugin myself.

I'd like to update this PR to use v0.4, then start investigating whether it still suffers from the reported performance issues. But that will have to wait until I get time for it (hopefully from late April).

[mattxwang]

Just circling back - slowly looking to devote more time on bigger JtD projects (instead of just maintenance). Anything I can do to help get this PR up to speed?

[pdmosses]

@mattxwang

Anything I can do to help get this PR up to speed?

I think it would be better to open a new PR for supporting multi-level navigation, rather than continuing this one. The conversation here has become too long for comfort. Having to scroll down through a lot of (mostly resolved) comments would surely discourage discussion and reviews.

Moreover, some sites may have been using this PR branch for the past year or two. To avoid the risk of breaking those sites, I suggest to leave the PR branch as it is now. We can also leave the PR itself open until a new PR for multi-level navigation has been approved and merged.

@peterchenadded and @adfoster-r7 have made some suggestions for reducing the build times for sites using my current implementation of multi-level navigation (which can be painfully long, even when using Jekyll 4). Version 0.4 of JtD significantly reduced the build times for some larger 3-level sites; it's conceivable that incorporating similar optimisations would also help with multi-level sites.

However, we might also consider a more radical redesign of how we specify navigation in JtD sites. I recently took a closer look at Hugo, which seems to be generally regarded as the fastest SSG currently available. My curiosity had been piqued by some (partly misleading) comments about slow builds with JtD on Jekyll Talk.

I also noticed a Hugo theme called Docsy, which supports several features that we have yet to implement in JtD – including multi-lingual sites, versioned docs, and multi-level navigation! Using the folder structure of the source files to specify the navigation structure (apart from the relative order of pages at the same level) might allow JtD to reduce multi-level build times significantly, as well as circumventing the current need of the present PR for specifying section or ancestor to disambiguate parent title resolution.

So instead of trying to "get this PR up to speed", I'd prefer us to open a fresh issue (or simply a new discussion) to decide how best to support multi-level navigation in JtD – paying special attention to build times – before opening a PR to implement that support. Would that be okay?

[adfoster-r7]

@pdmosses Whatever you think is the best approach to move forwards with things works for me - I'm happy to test any new structure/format for bugs or performance comparisons as well 👍

Just as an extra datapoint; docusaurus also relies on the folder structure without needing extra frontmatter to be present when generating its sites:

• Example page: https://docusaurus.io/docs/advanced/routing
• Frontmatter: https://github.com/facebook/docusaurus/blob/41a52161fd3df5433ec2ed5e013743cec80a2286/website/docs/advanced/ssg.mdx

They do provide an escape hatch for specifying a slug to ignore the folder and generate a custom path though example

[peterchenadded]

If we are talking about performance, we need to also assess the underlying technology being used.

For example, are people actually using liquid in the markdown files?

If we instead use markdown-it an in browser js markdown converter and change to single page design, the pages will load much much faster.

We had actually gone down this path already and the build and page load performance is light and day for our growing 2.5k page site.

[adfoster-r7]

are people actually using liquid in the markdown files?

I can't speak for the majority; For our docs we definitely use liquid for generating the links to other pages (example. I imagine folk are using different liquid filters. For our docs we also register a custom liquid filter to use.

If we instead use markdown-it an in browser js markdown converter and change to single page design, the pages will load much much faster.

I believe that might impact SEO and search engine indexing, as well as potentially screen readers. In my experience markdown parsers all have different semantics too, so the in-browser rendering might behave differently to Jekyll's parser - which would introduce regressions in behavior

I imagine that could probably be added on as a separate feature from multi-level navigation support either way, but I'll defer to the experts here

[peterchenadded]

I can't speak for the majority; For our docs we definitely use liquid for generating the links to other pages (example. I imagine folk are using different liquid filters. For our docs we also register a custom liquid filter to use.

I think we can divide parts of the page that need liquid and parts that do not need it. There is no way to remove liquid for layout files which includes for example header, nav, footer and aux links you mentioned. The actual content shown probably does not need it and pure markdown syntax is sufficient.

I believe that might impact SEO and search engine indexing, as well as potentially screen readers. In my experience markdown parsers all have different semantics too, so the in-browser rendering might behave differently to Jekyll's parser - which would introduce regressions in behavior

True, but for us our site is private and SEO was not important. Though, we still have the full.md files and should still work with SEO, if we really wanted it to.

Your right about the rendering being different. But actually markdown-it has good number of extensions. So for example if you wanted number lines for your code, you just use the extension.

Agree there will be regressions and some features will change but if build speed and page load performance is of concern, moving to SPA and not using liquid where it is not needed should be considered.

I imagine that could probably be added on as a separate feature from multi-level navigation support either way, but I'll defer to the experts here

Yes, nav is a different area and definitely needs to be cached for bigger sites. Just following the structure of the file system was good enough for us, but many will probably want front matter to define different titles. It was the worse performing when profiling but after it is fixed it is almost mandatory to fix the other two issues.

Forgot to mention search, which makes the page unresponsive for a second as it tries to load in search-data.json on every page you visit. We actually had to setTimeout and chunk the loading of that file to make the UI responsive.

[mattxwang]

Hi all, thanks for the lively discussion! Bigger picture, I think a new issue could be appropriate; I could also put some work in simply porting this PR over to main (on a new branch, to avoid disrupting this one).

Let me address a few quick conversation topics:

• personally, I think rendering markdown client-side is significantly out of scope for this project. There are other solutions that work quite well already, and in those cases, you probably shouldn't be using Jekyll; it is intentionally a static-site generator. The "hybrid" approach that's being alluded to - code splitting and doing the equivalent of SSR/islands for SSG - is way out of scope of most Jekyll themes. I think that would be more appropriate for the modern JS stack for this, e.g. Astro/Next/11ty and associated tooling.
• file-based navigation is certainly appealing! I will point out that we are somewhat limited by Jekyll itself and that our theme is intended to work out-of-the-box with GitHub Pages (not necessarily a bad thing); since file-based navigation is not first-class, we'd have to do some hackiness to get it to work. The tradeoff is that Hugo is arguably less portable (depends on who you ask). Docusaurus is becoming the gold standard for OSS docs (Stylellnt has been using it for a while, for example), so it definitely makes sense to look at what they're doing.
• I do think an opt-in Ruby plugin is within the cards; pretty sure that we can have users opt-in with a _config.yml parameter and only load in the plugin/file at runtime (i.e. when compiling). This preserves backwards compatibility.

Of course, these are all my opinions, and I'm happy to hear out some other perspectives!

In terms of my personal roadmap as a maintainer, there are some prerequisites I'll do before I work on this:

• merging in PRs Fix font-size scaling for text-related CSS properties by using rem instead of fixed px values; deprecate $root-font-size #1169 and Add visual tests with Storybook & Chromatic #1236
• guiding in automatic theme switching (many open PRs for this, or I'll write my own)

After that, I'm happy to then dedicate my maintenance time to pushing this PR forward.

In the meantime, if someone would either like to:

• fork this branch and open an up-to-date PR
• or, experiment with any of the above suggestions (with the caveat that I may deem an approach out of the spirit of JtD)

feel free to do so - happy to review!

[pdmosses]

Bigger picture, I think a new issue could be appropriate; I could also put some work in simply porting this PR over to main (on a new branch, to avoid disrupting this one).

In any case, I think I first should move all the multi-level tests from this PR branch to (a branch of) just-the-doc-tests.

BTW, I'm currently using the combination-rec-nav branch of my fork of JtD to publish a draft proposal for multi-level navigation docs and some accompanying notes. The published website is clearly labeled as a fork of a particular branch.

Let me address a few quick conversation topics:

• personally, I think rendering markdown client-side is significantly out of scope for this project.

I know nothing about it, and I wouldn't be able to contribute.

• file-based navigation is certainly appealing! I will point out that we are somewhat limited by Jekyll itself and that our theme is intended to work out-of-the-box with GitHub Pages (not necessarily a bad thing); since file-based navigation is not first-class, we'd have to do some hackiness to get it to work. The tradeoff is that Hugo is arguably less portable (depends on who you ask). Docusaurus is becoming the gold standard for OSS docs (Stylellnt has been using it for a while, for example), so it definitely makes sense to look at what they're doing.

I've also looked at Material for MkDocs, which has already implemented many of the features that we intend to support in JtD (multi-level navigation, i18n, versioned docs, toggled between color schemes, …). It seems that the navigation hierarchy there can either be left implicit in the directory structure, using file names as titles – with optional front matter to override the URL and/or the title displayed in the navigation panel – or specified in a single YAML file that associates titles with file names in the intended order and nesting. Jekyll provides variables such as page.path, and I don't see why any hacking would be needed to implement the same idea in JtD. There's also Docsy Jekyll, a port of the Hugo theme Docsy.

• I do think an opt-in Ruby plugin is within the cards; pretty sure that we can have users opt-in with a _config.yml parameter and only load in the plugin/file at runtime (i.e. when compiling). This preserves backwards compatibility.

I've never programmed in Ruby, and I wouldn't be able to contribute to developing such a plugin. If we'd need to maintain consistency between a plugin and a Liquid implementation of multi-level navigation, some other maintainers would need to take responsibility for that.

Of course, these are all my opinions, and I'm happy to hear out some other perspectives!

In terms of my personal roadmap as a maintainer, there are some prerequisites I'll do before I work on this:

• merging in PRs Use rem instead of px for all text related CSS properties #1169 and Add visual tests with Storybook & Chromatic #1236
• guiding in automatic theme switching (many open PRs for this, or I'll write my own)

After that, I'm happy to then dedicate my maintenance time to pushing this PR forward.

I have a pressing conference submission deadline at the end of June. After that, I should have time to participate in further discussion and potential implementation of multi-level navigation.

In the meantime, if someone would either like to:

• fork this branch and open an up-to-date PR

I think we need an in-depth discussion of what we would like to implement before opening a new PR on this topic.

There's an open discussion #425 with the question "Navigation levels limitation?". My response is now outdated, but I think it would be easy enough to update it, as a starting point for a fresh discussion of our current ideas for multi-level navigation.

An alternative might be to revisit the open issue #219 on "Navigation doesn't work with grand-grand-children". However, the linear structure of an issue doesn't seem as convenient as a discussion with threads of comments on potential answers, and discussions have the possibility of sorting them by the date of latest activity.

[ragalgut]

It would be fantastic if this went ahead. A recursive navigation.

[pdmosses]

@ragalgut Thanks for your comment!

PR #1397 should make it much easier to add recursive navigation (by reducing dependence on the parent, grand_parent, and has_children relations).

I expect to have time this month to investigate whether the directory structure could determine the navigation hierarchy (apart from the title and nav_order, which would still be specified in front-matter). I think that would have some significant benefits, although there might also be some drawbacks.

I suggest to continue the discussion in #1277, to make it more visible.

[didacrios]

Hi @pdmosses is there any consensus achieved related to this point? Seen the branch from the proposal far from the current main branch of the project and conflicts raised related to this https://just-the-docs.com/migration/#moved-include

[pdmosses]

This PR is to be superseded by #1431.

Users with websites built with the PR branch rec-nav-2 should consider migrating to the multi-level branch, as rec-nav-2 will not be updated, and may eventually be deleted.

[pdmosses]

Hi @pdmosses is there any consensus achieved related to this point?

@didacrios PR #1431 generalises the latest JtD release to allow any number of navigation levels. However, we cannot claim that a consensus has been achieved, as there has not been any further discussion of the adopted approach.

[pdmosses]

Superseded by #1431