Add option to localize books in multiple languages

[Ruin0x11]

Referencing the ideas at #5 (comment), I took a shot at implementing the localization of books.

• Add a [language] table to book.toml. Each key in the table defines a new language with various properties for localizing the metadata of the book.

[language.en]
name = "English"

[language.ja]
name = "日本語"
title = "なんかの本"
description = "何の役にも立たない本"
authors = ["Ruin0x11"]

• Change the directory structure of localized books. If the [language] table exists, mdBook will now assume the src/ directory contains subdirectories named after the keys in [language]. The behavior is backwards-compatible if you don't specify [language].

├── book.toml
└── src
    ├── en
    │   ├── chapter
    │   │   ├── 1.md
    │   │   ├── 2.md
    │   │   └── README.md
    │   ├── README.md
    │   ├── SUMMARY.md
    │   └── untranslated.md
    └── ja
        ├── chapter
        │   ├── 1.md
        │   ├── 2.md
        │   ├── 3.md
        │   └── README.md
        ├── README.md
        └── SUMMARY.md

• Specify which language of book to build using the -l/--language argument to mdbook build and similar, or omit to build all translations at once and add a menu in the output pages for switching between them.
  
• Specify the default language by setting book.language. This language gets used for page fallbacks.
• Each language has its own SUMMARY.md. It can include links to files not in other translations. If a link in SUMMARY.md refers to a nonexistent file that exists in the default language's file structure, the book loader will gracefully degrade the link to the default language's file. If it still doesn't exist, the config's build.create-missing option will be respected instead.
• Link redirection also works for missing links in the Markdown itself. If the corresponding file exists in the default translation, it will be rewritten to point there instead. This way a translation can share things like images from the default translation. (This won't work if --language is passed, since there will be no default page in the generated output to redirect to.)

Closes #5.

TODO

• Remove multilingual property in [book] config section
• Support translation of book title/description
• Make mdbook init generate books with the new localized directory structure
• Make mdbook serve aware of multiple language root directories
• Add docs in mdBook manual about localization
• Render entire book in all languages at once and allow switching between them in the generated pages?
• Support graceful fallback of missing non-Markdown files?
• Support graceful fallback of missing {{#include ...}} files?
• More tests

[MarcoIeni]

I don't know if this has been already discussed, but instead of having default=true wouldn't it be easier to say that the first language that has been declared is the default one?

In this case this constraint goes away by design:

Exactly one language must have default set to true if the [language] table is defined.

[Ruin0x11]

@MarcoIeni Yes actually, I updated the code to use book.language which was already there and removed default, I just didn't update the description yet.

I'm not sure about order because HashMap doesn't preserve insertion order.

[MarcoIeni]

Oh, ok. book.language looks good if it's not possible to keep the order :)

[Ruin0x11]

I'm trying to test a real-world example by adding the Japanese and French translations of The Rust Programming Language into the base repository.

https://github.com/Ruin0x11/book/tree/translated

A few things I'm noticing immediately:

• There's no way to tell that a page is not available for a given language if it is missing from the translation's SUMMARY.md. For example, the Japanese translation is nearly complete, but the French translation is missing all the chapters past Chapter 8 in its summary. Viewing one of the missing pages in English and then switching to French gives a 404. The same goes for a language in the translation but not in the default. I would expect to be able to see a copy of the English page under fr/ with a notice in French saying the page hasn't been translated yet instead of a 404.
• Pages missing from the translation's SUMMARY.md will not appear in the TOC even if there is an equivalent page in the default language.
• The need to update SUMMARY.md to reference missing chapters feels unnecessary. Ideally things should just be drop-in without having to edit the summary to reference the default language's new pages.
• If a page is not translated, there's no indication in the TOC or the output HTML that the page hasn't been translated to the user's native language yet. We might have to add translations to mdBook for all the languages needed to allow generating the message.

It could be more helpful to have the missing chapters automatically added regardless of whether or not they're in the summary with a notice about the missing translation, like this page: https://docs.microsoft.com/vi-vn/azure/guides/developer/azure-developer-guide. And for everything else, just grey out the language in the dropdown. But this would also have to handle the translations diverging, to prevent two copies of the same page with different names from showing up in the TOC - one for the default language which was moved, and one for the translation that hasn't been updated yet.

Maybe what we need to do is use a unique identifier of some kind in each summary item in addition to the human-readable text, and then each translation's summary would reference the same identifier to indicate what item it corresponds to. We could reuse the filename for this unique identifier. The default language's summary would define the order of each item. If a translation-local page is found, it would appear immediately after the summary item before it. Anything that is missing in the order defined in the default language's summary would get copied into the translation's summary and file structure.

[damirka]

@MarcoIeni @Ruin0x11

What needs to be done for this PR to be included into upstream? Can I help anyhow?

[MarcoIeni]

Hi damirka, I just looked at the code out of curiosity and added a comment. Then GitHub added me as a reviewer, but I am not one of the maintainers of this library (I haven't done a single commit 😅). I hope this PR will receive some love from the maintainers soon anyway :)

[Ruin0x11]

@damirka Mostly usability issues. There should be a way to define a "base" language structure that is shared between all languages. Then if you switch to another language, you should be able to easily tell which pages in the translation are not available, maybe copying the structure of the base language's outline and greying out the sections that are missing, showing a message, and possibly redirecting or replacing the page with the base language. (See the Microsoft docs example in my previous post.) Currently the outline is not shared between languages, so you can't tell what is missing and get a 404 instead if you switch anyway. Also this has to account for pages that are not in the base language but are in the translation, I think they can just be hidden if you're not viewing the translation.

I think that as far as the requirement to switch languages goes it's usable, but since it requires breaking changes to the structure of the books it might be helpful to get it (mostly) right the first time. That might mean having to change the meaning of SUMMARY.md to implement the shared outline feature, for example.

See my previous post for other gotchas, like needing to handle translations diverging.

[damirka]

@Ruin0x11

Currently, mdbook creates an empty page with a title if you've created link to this page in SUMMARY. Why can't translations do the same? Create title-only structure same as the non-translated version of the engine does?

Another question comes to mind - how to translate summary? Should it follow the original structure? So duplicating link blocks is the only way?

[Dylan-DPC-zz]

@Ruin0x11 can you resolve the conflicts? we can move this forward from there

[Ruin0x11]

@Dylan-DPC Yes, when I can get some cycles I will look at it.

[mahidul-islam]

Hi, @Ruin0x11 what is the update about this feature..

[Nutomic]

Thanks for this work @Ruin0x11! For your information, we are already using this in production at join.lemmy.ml/docs (rebased on a later mdbook version). Hopefully this can be completed and merged soon.

For some feedback, I think it would be good if the select language button was more prominent, right now it seems very easy to miss.

[DarkStar1]

It would be nice if this can make it into upstream, this is one feature that I am eagerly awaiting 😄 .

[almereyda]

I think it would be good if the select language button was more prominent, right now it seems very easy to miss

Maybe it's good to put it to the most right, next to the printing button? This way it declutters the available options of icons on the left side (even the theming button feels off there, given how often it will be used in comparison with the hamburger and the search magnifier), and gets one of the most prominent positions on the screen:

People are already conditioned to check the top-right for a login button, why a 🌐 there might as well be recognised.

---

@Nutomic Do you maintain a public repository of your fork, which can be used as guidance for the rebase which is left to happen here? The README in your lemmy-docs repository suggest you are effectively using this feature branch, and did not maintain a separate fork, which leaves me a little confused.

cargo install mdbook --git https://github.com/Ruin0x11/mdBook.git \
    --branch localization --rev d06249b

[Ruin0x11]

Done.

[rzerres]

Seems this branch will introduce a race:

i did crosscheck, that rendering on stock 0.4.12 is working out fine, including the mermaid preprocessor.

$ cd $ORBTK_BOOK_ROOT
$ MDBOOK_BOOK__src=src/en mdbook build

2021-09-21 17:00:20 [INFO] (mdbook::book): Book building has started
2021-09-21 17:00:20 [INFO] (mdbook::book): Running the html backend

Reason

I have tried to figure out, why mdbook::utils complains about a missing field.
The chapter_titles is marked as skip, so I'm not sure what's happening inside the produced serde stream. This seems to be an issue, if you include it to another preprocessor. At least throwing the error with mdbook-mermaid.

Describe the bug

When including mdbook-mermaid v0.8.3 (or latestest git version) as a valid preprocessor in the render chain controlled with mdbook v0.4.12 the process returns with ERROR.
I have checked with orbtk-book .

To Reproduce

Steps to reproduce the behavior:

• compile mdbook-mermaid

$ git clone https://github.com/badboy/mdbook-mermaid.git
$ cd ./mdbook-mermaid
$ cargo update; cargo install --path `pwd`

• add preporcessor to the book

$ cd $ORBTK_BOOK_ROOT
$ mdbook-mermaid install

inside $ORBTK_BOOK_ROOT/book.toml the output and preprocessor tables are extended as expected. The mermaid[-init, .min].js are included.

• calling the book build process

$ cd $ORBTK_BOOK_ROOT
$ RUST_LOG=info  mdbook build --dest-dir=$XDG_RUNTIME_DIR/book --language en

$ 2021-09-20 11:49:40 [ERROR] (mdbook::utils): Error: Unable to parse the preprocessed book from "mermaid" processor
$ 2021-09-20 11:49:40 [ERROR] (mdbook::utils): 	Caused By: missing field `chapter_titles` at line 1 column 83887```

Expected behavior

The application should render the book with the mermaid source.

Bug solved

Race condition is solved with latest commits (0.4.15)

[Ruin0x11]

The mentioned issue should be fixed, although the book in question needed some fixes for compatibility with the feature. I also merged the latest master.

@ehuss Could this perhaps see some more traction soon?

[rzerres]

Are we going to collect guide translations?

Where to submit the PR's?
I did translate the given guide to german. It is availabe as a PR inside Ruin0x11:localization, as well as from my Repo rzerres:wip_german

[mgeisler]

Hi @Ruin0x11,

(I'm just a random person with some experience with translating software.)

I wrote a lengthy comments on #5 about why I think it would be better to support translations with dedicated tooling. Basically, while it seems useful at first to use parallel directories, I don't think it's practical for the poor people who will have to maintain the translations. Translating software has been a solved problem for 20+ years (GNU Gettext was apparently released in 1995). I would advice building on top of the tooling which has already been developed in this field.

Now, the PR adds a lot of great stuff, which would still be needed if mdBook starts using Gettext: the language selector is important, as is the per-language configuration in book.toml.

[buttle]

mdBook is really good, and this branch makes it complete for us. Thanks!

[jkelleyrtp]

Some new features got added to mdbook making this branch somewhat behind. Are you able to update the branch or do you need any help?

[Ruin0x11]

@jkelleyrtp I did update the branch a few months ago, but it didn't get merged at the time. It seems like the feature may need a redesign according to @mdinger. Is there something else that would prevent this from being merged if I were to go back and update the branch again?

[Tealk]

i could also really use this feature, is there any progress?

[trdthg]

Hi, can't build on rustc 1.64. I have tested it using rust-toolchain to specify an older rustc(1.63, 1.32 ...), and all of them are successful.
Maybe this error comes from rust itself ?, and a pr to mdbook before 1.64 updates seems fixed it.

// src/renderer/html_handlebars/helpers/navigation.rs
// Before
_h.template()
    .ok_or_else(|| RenderError::new("Error with the handlebars template"))
    .and_then(|t| {
        let local_ctx = Context::wraps(&context)?;
        let mut local_rc = rc.clone();
        t.render(r, &local_ctx, &mut local_rc, out)
    })?;

Ok(())
// After
let t = _h
    .template()
    .ok_or_else(|| RenderError::new("Error with the handlebars template"))?;
let local_ctx = Context::wraps(&context)?;
let mut local_rc = rc.clone();
t.render(r, &local_ctx, &mut local_rc, out)

Would you like to fix this bug?