Switch to Hugo

[XhmikosR]

OK... this is probably something we might never get to do it due to the amount of changes.

That being said, Hugo is blazingly fast, has no dependency like Jekyll has to Ruby, and is very extensible too.

If anyone starts working on it, please let me know.

[Johann-S]

Without dependency 😲 love it 👍

[mdo]

has no dependency

You know the way straight to my heart 😻

[zebapy]

I've wanted to ask about using something like Gatsbyjs as an alternative or is that a crazier ask than Hugo? At least it'd be just node dependency instead of using brew to install Hugo. I'd work on something like that.

[XhmikosR]

Personally, I'd like to skip using React, Preact et all. Thus my choice is to look into Hugo; no extra client side dependencies, no dependencies to run either.

I started working on the Hugo switch, it won't be an easy task, but we'll see how it goes.

https://github.com/twbs/bootstrap/tree/v4-dev-xmr-hugo

@digitalcraftsman: if you have any suggestions please don't hesitate to try the branch and provide any patches to help move forward with this. :)

So far I face the following issues:

1. We need to have Hugo copy the dist dir without us moving it into the static folder; I made an issue in Hugo discourse and I think I'll ask for such a feature in the GitHub repo
2. I can't seem to be able to make shortcodes or highlight to work in index.html; https://discourse.gohugo.io/t/migrating-bootstrap-docs-to-hugo/8898/12?u=xhmikosr
3. I'll need to convert the Jekyll plugins to Hugo shortcodes, but that's after I have something working locally first :P

[digitalcraftsman]

Hello together,

first of all it's great to see projects of this size using Hugo. I'll try to help you as good as I can to speed up your transition.

We need to have Hugo copy the dist dir without us moving it into the static folder; I made an issue in Hugo discourse and I think I'll ask for such a feature in the GitHub repo

The best solution, that was already mentioned in the forum thread, is the ability to specify multiple static dirs. But this is currently not possible. Alternatively, we could make dist the new static directory via the staticDir config option.

The dist directory would remain untouched while other assets from the static folder could be moved into dist. Are the any naming conflicts or other issues that aren't on my radar?

I can't seem to be able to make shortcodes or highlight to work in index.html;

Have a look at my answer in the forum thread. The shown approach works for smaller code examples but it doesn't really scale.

Hopefully I or some other community members will come up with a better solution.

I'll need to convert the Jekyll plugins to Hugo shortcodes, but that's after I have something working locally first :P

I have not much experience with Jekyll so please help me to clarify the different terminologies. Are Jekyll's plugins == Hugo's short codes?

[XhmikosR]

Hi, @digitalcraftsman, thanks for the reply!

The best solution, that was already mentioned in the forum thread, is the ability to specify multiple static dirs. But this is currently not possible. Alternatively, we could make dist the new static directory via the staticDir config option.

The dist directory would remain untouched while other assets from the static folder could be moved into dist. Are the any naming conflicts or other issues that aren't on my radar?

That is indeed a valid workaround. It's just that traditionally the dist folder just contained only our dist files and nothing else. I think I will just combine nodemon which we already have.

Have a look at my answer in the forum thread. The shown approach works for smaller code examples but it doesn't really scale.

Hopefully I or some other community members will come up with a better solution.

Wouldn't it work if I had a partial with that section and in the partial use highlight? Or one can't use shortcodes in index.html at all?

I have not much experience with Jekyll so please help me to clarify the different terminologies. Are Jekyll's plugins == Hugo's short codes?

Not exactly, but Hugo's way of doing things is shortcodes :) Judging by our existent Jekyll plugins, it seems we need the following:

1. bugify.rb: just sets the bug URL based on a first parameter. I shouldn't have trouble converting this
2. callout.rb: just wraps the inner "shortcode" text into <div class="bd-callout bd-callout-#{@type}">#{output}</div>" so it shouldn't be hard either
3. example.rb: similar to callout, it's a wrapper for highlight, so I shouldn't have a problem either
4. markdown-block.rb: this allows us to write markdown code inside HTML code.

So as you can see, it's doable, I just need to have a working directory structure first and I'll get to it.

[XhmikosR]

After spending a few more hours on this, I believe it's a deal breaker the fact that we can't use variables in Markdown. And shortcodes is just not the answer to everything. For example:

Looking to quickly add Bootstrap to your project? Use the Bootstrap CDN, provided for free by the folks at MaxCDN.
Using a package manager or need to download the source files?
[Head to the downloads page.]({{ site.baseurl }}/docs/{{ site.docs_version }}/getting-started/download/)

or

{% highlight html %}
<script src="{{ site.cdn.jquery }}" integrity="{{ site.cdn.jquery_hash }}" crossorigin="anonymous"></script>
<script src="{{ site.cdn.popper }}" integrity="{{ site.cdn.popper_hash }}" crossorigin="anonymous"></script>
<script src="{{ site.cdn.js }}" integrity="{{ site.cdn.js_hash }}" crossorigin="anonymous"></script>
{% endhighlight %}

I was willing to spend any time needed to make the switch happen, but this is just not flexible enough.

So, at this point I consider the Hugo switch as a no go unfortunately, regardless of how much I wanted this to happen. :/

[bep]

Why aren't shortcodes a viable option for both cases above?

[XhmikosR]

Because the above are 2 examples; the docs have hundreds of such cases. And Hugo doesn't throw or anything just silently ignores variables in markdown making things harder to spot...

[bep]

And Hugo doesn't throw or anything just silently ignores variables in markdown making things harder to spot...

If you had asked me, which you did not, I would have said that ...

Why don't you create a param shortcode (or something) which fetches a given variable by name and, if not found, exits with -1 and prints "ERROR: Variable "foo" in page "/path/to/page.md" does not exist". Which would be pretty great error handling. Oh, and not only can shortcodes have great error handling, they can also be both inline and used as blocks, they can be nested as you please ...

But you didn't, and if you want to have that "half empty" attitude looking for blockers, that is fine. I thought having Bootstrap run its documentation site on Hugo would be cool for Hugo, and wanted to chime in with some support. But that support voucher requires a slightly different approach to have any longevity. I have edited enough big Hugo docs sites to be pretty certain that it would have been worth it. I'd even be willing to throw away some functionality to get there.

But up to you.

[XhmikosR]

@bep: Hey, no need to be defensive here (or even aggressive for that matter).

If you look at my messages above you will see that I created the issue and I expressed how much I want to make the switch. Personally, I don't have anything against either Jekyll nor Hugo; I use both of them, and I even prefer to use Hugo in new projects.

That put aside, the issues I face here are not issues that I only face. Having to edit so many markdown files is one thing, having to replace all instances of any custom variables in these files is just insane, at least to me. Searching for {{ in my branch, I get (1147 hits in 90 files).

So, you see why I say that this problem is a deal-breaker, unfortunately.

Oh, and yes, you can expect your end-users to do all these things you mention above, that is fine, I have done it in other personal projects too. But unfortunately, this isn't user-friendly, at all.

Now, I don't know the technical details behind this choice; I'm pretty sure there is some reasoning apart from the usual "content is content and is not supposed to have this". That being said, I'd be glad to have a look again. Either if we get some help (my branch is here) or if/when things become easier for the end-user.

[moorereason]

@XhmikosR, I agree. That's a ton of boring work to fix all of those files. However, I suspect a few sed/perl commands could do most of the work for you. It looks like you're in Windows, though, so I'd recommend using grepwin. RegEx FTW! 😄

[bep]

Yes, @moorereason is correct about the "search and replace" being a good general tool for these migrations.

I had a talk with @XhmikosR Off Broadway (aka Off GitHub) and concluded that he should provide me with some examples of the core pain points in this migration, and I would create a suggested "Hugo way" of doing it.

"If you only have a hammer, you tend to see every problem as a nail" is a saying that could be applied to this. I don't like that saying. In the enterprise world you may be restricted by politics, but in the open source world, almost any problem can be solved given a solid combination of time+brain. So until the fat lady starts singing the "This is a Road Blocker!" song, I suggest we keep a positive look on this.

If any of you want to discuss the meta question "does Hugo support variables in markdown?", I suggest you open a thread here: http://discuss.gohugo.io/

My answer to the above is: "Yes it does for most practical applications." Hugo doesn't currently support scripting directly in markdown (indirectly via shortcodes). If that is really needed, maybe we could consider (bold and italic) adding Lua shortcode? As I said, look for solutions, not deal breakers.

[digitalcraftsman]

I've tried to optimize the way code snippets can be handled in templates. For the insertion of variables into the code snippets we could add a template function that takes a string and the current context. The string gets parsed as a template in which the variables ({{ .Params.... }}) are replaced with the values from the passed context.

This could look like this:

{{ render "code snippet with variables" . }}

However, I wasn't able to create multiline strings in Go templates that preserve line breaks, not even with backticks. Hence "code snippet with variables" can't passed as follows:

{{ render `code 
snippet 
with variables` . }}

[XhmikosR]

All right, here's what I have so far:

1. I have removed the highlight shortcode from index.html to make Hugo build, but I need to find a solution for this and bring it back later
2. I have some trouble with our current data files. For example I converted docs-sidebar.html but I end up with no error but empty sidebar here c0199de. This has also another issue with split and last, see the inline comment
3. I ended up using browsersync and I manually copy the dist folder to the publishdir. I tried making the root dir the static dir, but then I couldn't make the ignores to work. So, I think this solution should suffice for now and it actually makes things a little more organized
4. I'm not sure I did it right, probably not, but I had to move the homepage's code in index.html, so I think the layouts I chose are wrong for index.html
5. This doesn't work either https://github.com/twbs/bootstrap/blob/v4-dev-xmr-hugo/docs/content/docs/4.0/examples/index.md. Probably something I'm missing here too since simple.html, has layout: default so it's basically extending the default.html.
6. And the known issue with variables in markdown files like Get involved with Bootstrap development by [opening an issue]({{ .Site.Params.repo }}/issues/new) or submitting a pull request. Read our [contributing guidelines]({{ .Site.Params.repo }}/blob/v{{ .Site.Params.current_version }}/.github/CONTRIBUTING.md) for information on how we develop.

I also rebased the branch to keep things clean, but I won't do it from now on so that if anyone wants to contribute something, we don't get any conflicts.

Thanks in advance for any help!

[bep]

@XhmikosR the issue gohugoio/hugo#4011 should describe a solution to the most critical issues you're facing. For the dist folder problem you could first try to create a symbolic link from static/dist to dist and see if that could be a workable solution.

And if you thought that my earlier comment on this was defensive, that was not my intention, and I also don't read that in my remarks. I just saw this negative approach to this whole thing that I believe would never have moved you anywhere close to the goal. My intention was to help.

So, even after Issue 4011 has been fixed (should not take too long, but I'm in Spain right now, so my focus is elsewhere), you still will have to fix those "hundreds of files". It will still, not surprisingly, not be Ruby syntax. This should be fairly straightforward with some regexp replacements, but I see how it could be a problem if you need to do it by hand.

So, for me to spend a considerable amount of time on 4011 I need some kind of commitment from both you, @XhmikosR and also the people who eventually would be in a position to merge this, that you intend to try to get this working and merged. I understand that there still may be blockers that we don't see, but the "hundreds of files" that needs an edit is not in that category.

[XhmikosR]

@bep: about the dist folder, I just added a step in our scripts which should do the job for now. That being said, I do see something inconsistent between Windows and the Travis builds:

Built site for language en:
0 draft content
0 future content
0 expired content
0 regular pages created
2 other pages created
0 non-page files copied
0 paginator pages created
total in 6 ms

On my Windows dev VM I get:

Built site for language en:
0 draft content
0 future content
0 expired content
87 regular pages created
4 other pages created
38 non-page files copied
0 paginator pages created
total in 486 ms

So, I think it's gohugoio/hugo#3568 to blame. After that is fixed I will try moving the config files in root and make use of the ignoreFiles section, which when I last tried it I couldn't make the regexes to work at all.

Now, I don't mind replacing many stuff, that is why regex exists. :) What I mind, is if we can use directly variables like we do now. Of course we'll need some shortcodes, I already added a couple to replace the Jekyll plugins.

As for going through with this, I don't see any reason not to, as long as we can do things with the same ease like we did with Jekyll regarding the variables and content. Anything else we will face, I'm pretty sure we will be able to find a solution.

@mdo: are you OK with the changes so far? https://github.com/twbs/bootstrap/compare/v4-dev-xmr-hugo

[bep]

The reason I have not rushed into fixing gohugoio/hugo#3568 is that I'm not really sure the current behaviour is broken. We could probably improve the situation: But the gist of it is that you provide the value for staticDir. staticDir=c:\my\project will never make sense on Linux. If you provide a relative file path, we could fix the slashes -- but you can do that yourself. Or, currently, just use forward slashes, which I think will work on both.

That said. I will schedule a "Hugo 0.31: The Bootstrap Edition" with the relevant improvements on the first Monday after @mdo gives his blessing to this project (i.e. "if this is doable, let's do it."). The changes will be in the Hugo master before that, but I like to let it linger for some days before pushing a binary release.

[XhmikosR]

@bep: forward slashes don't fix the issue, see my last comment on that issue.

But the problem I mention above might be something else since it happens on *nix and not on Windows.

[GoNode5]

@mdo the only non depency logical lingua franca for Bootstrap would be JavaScript/NodeJs. The rest is syntatic sugur, personal preference and taste du jour.

[bep]

A note from me, the main Hugo developer; I have said that I'm willing to address the issues raised above (most notably by adding support for inline shortcodes for all the ad-hoc scripting here), but for me to put down time on that I need some kind of commitment from you that this is something you (i.e. @mdo) really want to do.

I would recommend doing it. Hugo is very nice to work with for larger documentation sites, and this will be even more obvious if you plan to do translations in the future.

[XhmikosR]

I'm gonna take another stab at this, but due to the amount of changes my Hugo branch is basically useless and I'll need to create it from scratch. I'll see when I'll get around to do it.

[ghost]

I would like to see the docs migrated to Hugo :-)
I don't know either Jekyll or Hugo but I'll have a look in the following months .. provided that you didn't finish the migration by then ;-)

[XhmikosR]

@bep: I started making some progress here: https://github.com/twbs/bootstrap/tree/v4-dev-xmr-hugo-2

Some notes so far:

1. not being able to use inline code like before becomes easily a problem. I mean, I could add a few shortcodes for the most used stuff, but still it seems there will be cases we'd need this

2. the staticDir addition (since the other time I tried this) is nice for sure. But setting it to:

   staticDir:
     - "static"
     - "../dist"

   while it copies the dist folder subfolders and files to the publishDir, it doesn't seem to be a way to output the folder to a custom location in the publishDir. Not a big deal if it's not possible, we already have to do this separately. It's just this would get rid of the need to fire up two shells when developing

3. I really like the ref and relref shortcodes, and I believe it's the proper way to reference pages and files. But I had trouble using a variable even with print so that we don't hardcode the version. Could be I'm missing something here. Maybe we could extend this or create a new shortcode wrapper to the existent shortcode

4. There are still many things left to fix but the branch at least builds :)

Thanks for any help!

[bep]

not being able to use inline code like before becomes easily a problem.

1. I have said that I would be willing to add "inline shortcodes" to Hugo if Bootstrap was moving its docs to Hugo (it would be good PR for us). I understand that you need it, and I guess it would be generally usable -- but for me to put a priority on this and the 20 hours or so it would take me to develop, I would need some kind of commitment from this project (and @mdo) that this is something you will try to get done. I understand if there are technical roadblocks that could stop this (doubt it), but the intention should be that this is going to be merged.

2. ref/relref shortcodes are only very shallow wrappers around .Page.GetPage. You can easily create your own. See Use shortcode inside a one-line shortcode gohugoio/hugo#5429 (comment)

[XhmikosR]

1. It's already been decided internally to move to Hugo by all members. So, it's a matter of how easily we can solve the issues and the switch will happen ASAP :)
2. Thanks! I'd love to be able to use such thing without the extra shortcode, but that'll do, at least for one var in the link.

[bep]

Thanks! I'd love to be able to use such thing without the extra shortcode,

Note that with what I call "inline shortcodes" above, you can pretty much do whatever you want -- but I would still consider it to be cleaner if you created shortcodes (i.e. files) for your "common stuff".

[XhmikosR]

I agree, but I also need to visualize the transition and make changes along the way :)

That being said, do let me know if there are any improvements so that I keep the branch rebased and ready, not like my old one.

I have a short TODO

[bep]

Also see gohugoio/hugo#5463

[bep]

@XhmikosR a quick note: I have now merged what I think is the 2 main "showstoppers" for you into Hugo master.

gohugoio/hugo@bc337e6
gohugoio/hugo@f37c5a2

I will cut a new Hugo release shortly, but it is fairly trivial to build from source (esp. if you use Homebrew on macOS).

[XhmikosR]

@bep: thanks! I got notified already and rebased my hugo-2 branch and I will give this a go when 0.52 is out and see what's left :)

[XhmikosR]

@bep: I gave it a go in my branch with my hugo-bin fork which is using v0.52. param works pretty good, but I couldn't get baseURL to work with it.

That being said, maybe there's a simpler way to reference a page that could accept variables?

After this is sorted, I need to figure out how to fix the examples layout, and finally the remaining shortcodes and highlighting styles/ToC.

[bep]

That being said, maybe there's a simpler way to reference a page that could accept variables?

As I said earlier, you would need to create your own "relref" to do that. There is gohugoio/hugo#5463 -- but that needs to wait. You should have no problem creating a clean and fast "Hugo variant" of this site.

Note that the ToC structure is currently a little odd (it is on my short term plan to fix that) -- but I have done similar "Bootstrap CSS" ToC styling as you have on your site, so it should be possible. I think you have to be a little pragmatic, and revisit the "not perfect" areas later.

[bep]

A tip here: Note that you can provide your own implementation of shortcodes with the same name as the built-ins in Hugo, so if you create a relref (or ref) that takes an optional second argument, so you can do this to get the "latest version":

{{< relref "components/foo" >}}

And

{{< relref "components/foo" "v3.0.0" >}}

Or something to get another version. Note that I do not know your particular use case, so the above is just some random examples.

[bep]

... Or use named params ... or create your own vref shortcode ... Main point being: Don't underestimate the power and simplicity of shortcodes.

[ghost]

After this is sorted, I need to figure out how to fix the examples layout [...].

If you're topic branch includes your current working tree LMK and I'll take a stab at it my day tomorrow.

[XhmikosR]

Sorry for the late reply :/

@bep: thanks a lot for these, I'll try to find a solution. I'm all for refactoring, it's just that I'm also learning along the way more.

@JHabdas: yeah, but keep in mind I rebase it.

Any help will be greatly appreciated :)

[ghost]

@JHabdas: yeah, but keep in mind I rebase it.

Sorry I didn't see your note until now (though I did check before the 6th). I'm preparing for travel and have to renege on my offer at least for the time being. This is something I could help with in the future so please feel free to reach out if you get stuck: [email protected].

[XhmikosR]

@JHabdas you can check #28014 and make PRs against that branch.

[XhmikosR]

I just merged the PR 🕺

I also made an issue about stuff that we need to look into, any help appreciated :)

Huge thanks to @bep for adding functionality that helped us make this happen!

[bep]

This is really, really great!