Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to localize books in multiple languages #1306

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

Ruin0x11
Copy link

@Ruin0x11 Ruin0x11 commented Aug 28, 2020

Referencing the ideas at #5 (comment), I took a shot at implementing the localization of books.

  • Add a [language] table to book.toml. Each key in the table defines a new language with various properties for localizing the metadata of the book.
[language.en]
name = "English"

[language.ja]
name = "日本語"
title = "なんかの本"
description = "何の役にも立たない本"
authors = ["Ruin0x11"]
  • Change the directory structure of localized books. If the [language] table exists, mdBook will now assume the src/ directory contains subdirectories named after the keys in [language]. The behavior is backwards-compatible if you don't specify [language].
├── book.toml
└── src
    ├── en
    │   ├── chapter
    │   │   ├── 1.md
    │   │   ├── 2.md
    │   │   └── README.md
    │   ├── README.md
    │   ├── SUMMARY.md
    │   └── untranslated.md
    └── ja
        ├── chapter
        │   ├── 1.md
        │   ├── 2.md
        │   ├── 3.md
        │   └── README.md
        ├── README.md
        └── SUMMARY.md
  • Specify which language of book to build using the -l/--language argument to mdbook build and similar, or omit to build all translations at once and add a menu in the output pages for switching between them.
    2020-08-28-032545_1366x750_scrot
  • Specify the default language by setting book.language. This language gets used for page fallbacks.
  • Each language has its own SUMMARY.md. It can include links to files not in other translations. If a link in SUMMARY.md refers to a nonexistent file that exists in the default language's file structure, the book loader will gracefully degrade the link to the default language's file. If it still doesn't exist, the config's build.create-missing option will be respected instead.
  • Link redirection also works for missing links in the Markdown itself. If the corresponding file exists in the default translation, it will be rewritten to point there instead. This way a translation can share things like images from the default translation. (This won't work if --language is passed, since there will be no default page in the generated output to redirect to.)

Closes #5.

TODO

  • Remove multilingual property in [book] config section
  • Support translation of book title/description
  • Make mdbook init generate books with the new localized directory structure
  • Make mdbook serve aware of multiple language root directories
  • Add docs in mdBook manual about localization
  • Render entire book in all languages at once and allow switching between them in the generated pages?
  • Support graceful fallback of missing non-Markdown files?
  • Support graceful fallback of missing {{#include ...}} files?
  • More tests

@MarcoIeni
Copy link
Member

I don't know if this has been already discussed, but instead of having default=true wouldn't it be easier to say that the first language that has been declared is the default one?

In this case this constraint goes away by design:

Exactly one language must have default set to true if the [language] table is defined.

@Ruin0x11
Copy link
Author

Ruin0x11 commented Aug 29, 2020

@MarcoIeni Yes actually, I updated the code to use book.language which was already there and removed default, I just didn't update the description yet.

I'm not sure about order because HashMap doesn't preserve insertion order.

src/config.rs Outdated Show resolved Hide resolved
@MarcoIeni
Copy link
Member

Oh, ok. book.language looks good if it's not possible to keep the order :)

@Ruin0x11 Ruin0x11 force-pushed the localization branch 2 times, most recently from c8101cb to a06aafd Compare August 29, 2020 23:12
@Ruin0x11
Copy link
Author

Ruin0x11 commented Aug 30, 2020

I'm trying to test a real-world example by adding the Japanese and French translations of The Rust Programming Language into the base repository.

https://github.com/Ruin0x11/book/tree/translated

A few things I'm noticing immediately:

  • There's no way to tell that a page is not available for a given language if it is missing from the translation's SUMMARY.md. For example, the Japanese translation is nearly complete, but the French translation is missing all the chapters past Chapter 8 in its summary. Viewing one of the missing pages in English and then switching to French gives a 404. The same goes for a language in the translation but not in the default. I would expect to be able to see a copy of the English page under fr/ with a notice in French saying the page hasn't been translated yet instead of a 404.
  • Pages missing from the translation's SUMMARY.md will not appear in the TOC even if there is an equivalent page in the default language.
  • The need to update SUMMARY.md to reference missing chapters feels unnecessary. Ideally things should just be drop-in without having to edit the summary to reference the default language's new pages.
  • If a page is not translated, there's no indication in the TOC or the output HTML that the page hasn't been translated to the user's native language yet. We might have to add translations to mdBook for all the languages needed to allow generating the message.

It could be more helpful to have the missing chapters automatically added regardless of whether or not they're in the summary with a notice about the missing translation, like this page: https://docs.microsoft.com/vi-vn/azure/guides/developer/azure-developer-guide. And for everything else, just grey out the language in the dropdown. But this would also have to handle the translations diverging, to prevent two copies of the same page with different names from showing up in the TOC - one for the default language which was moved, and one for the translation that hasn't been updated yet.

Maybe what we need to do is use a unique identifier of some kind in each summary item in addition to the human-readable text, and then each translation's summary would reference the same identifier to indicate what item it corresponds to. We could reuse the filename for this unique identifier. The default language's summary would define the order of each item. If a translation-local page is found, it would appear immediately after the summary item before it. Anything that is missing in the order defined in the default language's summary would get copied into the translation's summary and file structure.

@Ruin0x11 Ruin0x11 force-pushed the localization branch 2 times, most recently from 910c342 to d06249b Compare August 30, 2020 01:32
@damirka
Copy link

damirka commented Dec 15, 2020

@MarcoIeni @Ruin0x11

What needs to be done for this PR to be included into upstream? Can I help anyhow?

@MarcoIeni
Copy link
Member

Hi damirka, I just looked at the code out of curiosity and added a comment. Then GitHub added me as a reviewer, but I am not one of the maintainers of this library (I haven't done a single commit 😅). I hope this PR will receive some love from the maintainers soon anyway :)

@Ruin0x11
Copy link
Author

Ruin0x11 commented Dec 16, 2020

@damirka Mostly usability issues. There should be a way to define a "base" language structure that is shared between all languages. Then if you switch to another language, you should be able to easily tell which pages in the translation are not available, maybe copying the structure of the base language's outline and greying out the sections that are missing, showing a message, and possibly redirecting or replacing the page with the base language. (See the Microsoft docs example in my previous post.) Currently the outline is not shared between languages, so you can't tell what is missing and get a 404 instead if you switch anyway. Also this has to account for pages that are not in the base language but are in the translation, I think they can just be hidden if you're not viewing the translation.

I think that as far as the requirement to switch languages goes it's usable, but since it requires breaking changes to the structure of the books it might be helpful to get it (mostly) right the first time. That might mean having to change the meaning of SUMMARY.md to implement the shared outline feature, for example.

See my previous post for other gotchas, like needing to handle translations diverging.

@damirka
Copy link

damirka commented Dec 16, 2020

@Ruin0x11

Currently, mdbook creates an empty page with a title if you've created link to this page in SUMMARY. Why can't translations do the same? Create title-only structure same as the non-translated version of the engine does?

Another question comes to mind - how to translate summary? Should it follow the original structure? So duplicating link blocks is the only way?

@Dylan-DPC-zz
Copy link

@Ruin0x11 can you resolve the conflicts? we can move this forward from there

@Ruin0x11
Copy link
Author

@Dylan-DPC Yes, when I can get some cycles I will look at it.

@mahidul-islam
Copy link

Hi, @Ruin0x11 what is the update about this feature..

@Nutomic
Copy link

Nutomic commented May 13, 2021

Thanks for this work @Ruin0x11! For your information, we are already using this in production at join.lemmy.ml/docs (rebased on a later mdbook version). Hopefully this can be completed and merged soon.

For some feedback, I think it would be good if the select language button was more prominent, right now it seems very easy to miss.

@DarkStar1
Copy link

It would be nice if this can make it into upstream, this is one feature that I am eagerly awaiting 😄 .

@almereyda
Copy link

almereyda commented Sep 3, 2021

I think it would be good if the select language button was more prominent, right now it seems very easy to miss

Maybe it's good to put it to the most right, next to the printing button? This way it declutters the available options of icons on the left side (even the theming button feels off there, given how often it will be used in comparison with the hamburger and the search magnifier), and gets one of the most prominent positions on the screen:

People are already conditioned to check the top-right for a login button, why a 🌐 there might as well be recognised.


@Nutomic Do you maintain a public repository of your fork, which can be used as guidance for the rebase which is left to happen here? The README in your lemmy-docs repository suggest you are effectively using this feature branch, and did not maintain a separate fork, which leaves me a little confused.

cargo install mdbook --git https://github.com/Ruin0x11/mdBook.git \
    --branch localization --rev d06249b

@Ruin0x11
Copy link
Author

Done.

@rzerres
Copy link

rzerres commented Sep 21, 2021

Seems this branch will introduce a race:

i did crosscheck, that rendering on stock 0.4.12 is working out fine, including the mermaid preprocessor.

$ cd $ORBTK_BOOK_ROOT
$ MDBOOK_BOOK__src=src/en mdbook build

2021-09-21 17:00:20 [INFO] (mdbook::book): Book building has started
2021-09-21 17:00:20 [INFO] (mdbook::book): Running the html backend

Reason

I have tried to figure out, why mdbook::utils complains about a missing field.
The chapter_titles is marked as skip, so I'm not sure what's happening inside the produced serde stream. This seems to be an issue, if you include it to another preprocessor. At least throwing the error with mdbook-mermaid.

Describe the bug

When including mdbook-mermaid v0.8.3 (or latestest git version) as a valid preprocessor in the render chain controlled with mdbook v0.4.12 the process returns with ERROR.
I have checked with orbtk-book .

To Reproduce

Steps to reproduce the behavior:

  • compile mdbook-mermaid
$ git clone https://github.com/badboy/mdbook-mermaid.git
$ cd ./mdbook-mermaid
$ cargo update; cargo install --path `pwd`
  • add preporcessor to the book
$ cd $ORBTK_BOOK_ROOT
$ mdbook-mermaid install

inside $ORBTK_BOOK_ROOT/book.toml the output and preprocessor tables are extended as expected. The mermaid[-init, .min].js are included.

  • calling the book build process
$ cd $ORBTK_BOOK_ROOT
$ RUST_LOG=info  mdbook build --dest-dir=$XDG_RUNTIME_DIR/book --language en

$ 2021-09-20 11:49:40 [ERROR] (mdbook::utils): Error: Unable to parse the preprocessed book from "mermaid" processor
$ 2021-09-20 11:49:40 [ERROR] (mdbook::utils): 	Caused By: missing field `chapter_titles` at line 1 column 83887```

Expected behavior

The application should render the book with the mermaid source.

Bug solved

Race condition is solved with latest commits (0.4.15)

Nutomic added a commit to LemmyNet/lemmy-docs that referenced this pull request Oct 22, 2021
Nutomic added a commit to LemmyNet/lemmy-docs that referenced this pull request Oct 22, 2021
dessalines pushed a commit to LemmyNet/lemmy-docs that referenced this pull request Oct 25, 2021
* Generate config docs from lemmy code

* fix mkdir error in build.sh, update lemmy submodule

* fix drone config

* init git submodule

* install git

* apt update

* -y

* dont do submodule init

* x

* w

* update gitignore

* xz

* remove submodule

* readd submodule

* step

* update rust

* deps

* use defaults.hjson from lemmy without building, also include apub examples

* use updated mdbook from upstream pr

rust-lang/mdBook#1306 (comment)

* Download embeds over http instead of using git submodule

* install curl

* apt update

* typo

* -y
# Conflicts:
#	.gitignore
#	guide/src/en/cli/completions.md
#	guide/src/en/format/images/rust-logo-blk.svg
#	guide/src/en/format/markdown.md
#	guide/src/en/misc/introduction.md
#	src/renderer/html_handlebars/hbs_renderer.rs
#	src/utils/mod.rs
@Ruin0x11
Copy link
Author

Ruin0x11 commented Feb 25, 2022

The mentioned issue should be fixed, although the book in question needed some fixes for compatibility with the feature. I also merged the latest master.

@ehuss Could this perhaps see some more traction soon?

teobouvard added a commit to syrac-org/syrac-org.github.io that referenced this pull request Mar 19, 2022
@rzerres
Copy link

rzerres commented Mar 21, 2022

Are we going to collect guide translations?

Where to submit the PR's?
I did translate the given guide to german. It is availabe as a PR inside Ruin0x11:localization, as well as from my Repo rzerres:wip_german

@mgeisler
Copy link
Contributor

mgeisler commented Jun 2, 2022

Hi @Ruin0x11,

(I'm just a random person with some experience with translating software.)

I wrote a lengthy comments on #5 about why I think it would be better to support translations with dedicated tooling. Basically, while it seems useful at first to use parallel directories, I don't think it's practical for the poor people who will have to maintain the translations. Translating software has been a solved problem for 20+ years (GNU Gettext was apparently released in 1995). I would advice building on top of the tooling which has already been developed in this field.

Now, the PR adds a lot of great stuff, which would still be needed if mdBook starts using Gettext: the language selector is important, as is the per-language configuration in book.toml.

@buttle
Copy link

buttle commented Jun 11, 2022

mdBook is really good, and this branch makes it complete for us. Thanks!

@jkelleyrtp
Copy link

Some new features got added to mdbook making this branch somewhat behind. Are you able to update the branch or do you need any help?

@Ruin0x11
Copy link
Author

Ruin0x11 commented Jul 14, 2022

@jkelleyrtp I did update the branch a few months ago, but it didn't get merged at the time. It seems like the feature may need a redesign according to @mdinger. Is there something else that would prevent this from being merged if I were to go back and update the branch again?

@Tealk
Copy link

Tealk commented Sep 24, 2022

i could also really use this feature, is there any progress?

@trdthg
Copy link

trdthg commented Sep 27, 2022

Hi, can't build on rustc 1.64. I have tested it using rust-toolchain to specify an older rustc(1.63, 1.32 ...), and all of them are successful.
Maybe this error comes from rust itself ?, and a pr to mdbook before 1.64 updates seems fixed it.

// src/renderer/html_handlebars/helpers/navigation.rs
// Before
_h.template()
    .ok_or_else(|| RenderError::new("Error with the handlebars template"))
    .and_then(|t| {
        let local_ctx = Context::wraps(&context)?;
        let mut local_rc = rc.clone();
        t.render(r, &local_ctx, &mut local_rc, out)
    })?;

Ok(())
// After
let t = _h
    .template()
    .ok_or_else(|| RenderError::new("Error with the handlebars template"))?;
let local_ctx = Context::wraps(&context)?;
let mut local_rc = rc.clone();
t.render(r, &local_ctx, &mut local_rc, out)

Would you like to fix this bug?

mirrorcult added a commit to space-wizards/mdBook-spacewizards that referenced this pull request Sep 28, 2023
commit 8664faea083017b1ec7c9d811be28427b8408bef
Author: Kara <[email protected]>
Date:   Thu Sep 28 10:47:37 2023 -0500

    Update for new mdbook version

commit 1b45e7a7a6521b4df6d441788a7fff105eba9240
Merge: e74fdb1 79edc75
Author: Kara <[email protected]>
Date:   Thu Sep 28 10:03:55 2023 -0500

    Merge branch 'master' into localization

    # Conflicts:
    #	Cargo.lock
    #	Cargo.toml
    #	src/book/book.rs
    #	src/book/init.rs
    #	src/book/mod.rs
    #	src/cmd/build.rs
    #	src/cmd/clean.rs
    #	src/cmd/serve.rs
    #	src/cmd/test.rs
    #	src/cmd/watch.rs
    #	src/config.rs
    #	src/preprocess/links.rs
    #	src/renderer/html_handlebars/hbs_renderer.rs
    #	src/renderer/markdown_renderer.rs
    #	src/utils/mod.rs
    #	tests/init.rs

commit e74fdb1
Author: Ruin0x11 <[email protected]>
Date:   Fri Feb 25 14:30:38 2022 -0800

    Make `chapter_titles` optional in Book

commit 7305e8c
Merge: 9d8147c 5921f59
Author: Ruin0x11 <[email protected]>
Date:   Fri Feb 25 14:13:22 2022 -0800

    Merge remote-tracking branch 'upstream/master' into localization

    # Conflicts:
    #	.gitignore
    #	guide/src/en/cli/completions.md
    #	guide/src/en/format/images/rust-logo-blk.svg
    #	guide/src/en/format/markdown.md
    #	guide/src/en/misc/introduction.md
    #	src/renderer/html_handlebars/hbs_renderer.rs
    #	src/utils/mod.rs

commit 9d8147c
Author: Ruin0x11 <[email protected]>
Date:   Wed Sep 15 21:49:58 2021 -0700

    Remove extra `localization.md`

commit 56e72a2
Author: Ruin0x11 <[email protected]>
Date:   Wed Sep 15 15:33:28 2021 -0700

    [localization] rustfmt

commit 92ec3dd
Author: Ruin0x11 <[email protected]>
Date:   Wed Sep 15 15:25:31 2021 -0700

    [localization] Fixes for latest master

commit d6c27ab
Author: Ruin0x11 <[email protected]>
Date:   Sat Aug 29 16:11:47 2020 -0700

    Implement translation fallback of files included with preprocessing

commit 5fed5e8
Author: Ruin0x11 <[email protected]>
Date:   Wed Sep 15 14:29:30 2021 -0700

    Update mdBook manual to have information about translations

commit 09a8b66
Author: Ruin0x11 <[email protected]>
Date:   Sat Aug 29 14:41:08 2020 -0700

    Improve robustness of link rewriting

commit 8d1c086
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 16:33:02 2020 -0700

    Fix {{#include}} directives for default language

commit 98c3a04
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 16:11:21 2020 -0700

    Move example book to multilingual structure

commit c72ce18
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 14:50:04 2020 -0700

    Rewrite links in Markdown to point to fallback if missing in translation

    It will follow relative links to other pages and embedded images.

commit ee740ac
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 12:26:08 2020 -0700

    Remove 'default' property on languages, use book.language instead

commit a042cfc
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 11:35:42 2020 -0700

    Make `mdbook init` output multilingual structure

commit 5e223e0
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 03:17:26 2020 -0700

    Support localizing book title/description

commit e17ce64
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 02:29:07 2020 -0700

    Fix test using create_missing

commit 282fdaa
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 02:05:21 2020 -0700

    Redirect to a 404 page when serving translated

    We can't redirect in warp based on the URL, so redirect to the default
    language's 404 page instead.

    See: seanmonstar/warp#171

commit 85ab4d3
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 01:36:22 2020 -0700

    Redirect to translation index page in serve command

commit 8869c2c
Author: Ruin0x11 <[email protected]>
Date:   Fri Aug 28 00:24:33 2020 -0700

    Build multiple books from localizations at once

    Changes how the `book` module loads books. Now it is possible to load
    all of the translations of a book and put them into a single output
    folder. If a book is generated this way, a menu will be created in the
    handlebars renderer for switching between languages.

commit 96d9271
Author: Ruin0x11 <[email protected]>
Date:   Thu Aug 27 19:44:24 2020 -0700

    Specify language for book in command line args

    - Add a [language] table to book.toml. Each key in the table defines a
    new language with `name` and `default` properties.
    - Changes the directory structure of localized books. If the [language]
    table exists, mdBook will now assume the src/ directory contains
    subdirectories named after the keys in [language]. The behavior is
    backwards-compatible if you don't specify [language].
    - Specify which language of book to build using the -l/--language
    argument to `mdbook build` and similar, or omit to use the default
    language.
    - Specify the default language by setting the `default` property to
    `true` in an entry in [language]. Exactly one language must have `default`
    set to `true` if the [language] table is defined.
    - Each language has its own SUMMARY.md. It can include links to files
    not in other translations. If a link in SUMMARY.md refers to a
    nonexistent file that is specified in the default language, the renderer
    will gracefully degrade the link to the default language's page. If it
    still doesn't exist, the config's `create_missing` option will be
    respected instead.

commit 3049d9f
Author: Ruin0x11 <[email protected]>
Date:   Thu Aug 27 16:35:00 2020 -0700

    Actually, don't change source root

    The book paths have to gracefully degrade to the default language if
    they aren't available.

commit 24e6d6b
Author: Ruin0x11 <[email protected]>
Date:   Thu Aug 27 16:26:07 2020 -0700

    Change book source root depending on language

commit e4b443c
Author: Ruin0x11 <[email protected]>
Date:   Thu Aug 27 13:27:47 2020 -0700

    Add language config section

    Referencing rust-lang#5 (comment).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add multilingual support