Use Go Modules for Theme Component dependency management

[bep]

This has been mentioned before (I will track down and close that issue), but I'll start fresh.

So, Hugo Themes and Theme Components are very powerful constructs, but the current manual installation/dependency management leaves a lot to be desired. I have tinkered with the idea of using some of the ideas from Go to create a hugo get type of command that could do this stuff, but I imagined lots of work, reinventing the wheel, so to speak.

Today I did some quick tests with Go Modules, and it works surprisingly well with non-Go type of modules. We would need to add some small layer of logic in Hugo to support it, but we would then get really, really good dependency management in Hugo.

To add a new dependency to a theme or a project would then look like this:

1. go get -u github.com/bep/my-shortcodes
2. Add github.com/bep/my-shortcodes to the theme slice in config.toml (and yes, @kaushalmodi, we should add an alias for that config key). Note that we of course still will support the /themes/my-shortcodes variant.

When you want to update your dependencies, you do go get -u to get the latest minor or patch release etc.

Note that there can only be one version of my-shortcodes in a Hugo project at a given time, which is how Minimal Version Selection comes into play. We need to investigate github.com/bep/my-shortcodes/v2 vs namespace vs names etc., but that can probably wait.

Note that the above will require that you have Go (>= v1.11) installed, but we should also probably add a hugo mod vendor command that writes all the dependencies to a folder in the project which could be used without needing Go.

Shout if you think this is a terrible idea, @kaushalmodi @onedrawingperday @digitalcraftsman @budparr @regisphilibert @RickCogley, and gang.

[budparr]

I Love this idea for getting us away from submodules.

If I understand this correctly, everything would be packaged up before our site hits a deployment service/script. It's not clear in my mind exactly: Would we be sending a binary to deploy, so instead of building with Hugo, Netlify—for instance—is building the site with Hugo_site_somethingspecific or whatever?

[kaushalmodi]

So this would need the users of themes with "theme components" to have go >= v1.11 installed, right?

Update: Or may be, for the folks with go >= v1.11 installed, they can use the go get approach, others can fall back to using git sub modules?

I am just trying to understand how this will pan out on Netlify.

[regisphilibert]

I find it very appealing.
If there's a Go fallback for users using < v1.11, it's great, but the current solution could also suffice as a fallback.

I am just trying to understand how this will pan out on Netlify.

Same here, worst case scenario we have special build command to run for when a dep needs to be updated/added and it's totally fine with me.
Best case: Netlify gets/updates the dep as it builds.

[bep]

@budparr there is no "binary"; the dependencies (shortcodes, themes etc.) are purely files and directories.

So, Netlify (at least their latest build image) has Go 1.12 installed, so that should not be a problem. And this is kind of analogous to JS projects needing NPM.

We can possibly improve this in the future if Go exports these commands as an API somewhat.

So for Netlify (given that it has Go 1.12 installed) you would not do anything special on the build side.

Running:

hugo

Will pull down any (new) dependency to the Go Module cache (which I assume is or could be easily cached between builds) and build.

For the situations where you want to create "standalone Hugo" distribution, you would run:

hugo mod vendor

The above would write the dependencies to /vendor (or something) below your Hugo project, and it will build fine even on servers without Go installed.

[budparr]

Yes, of course. I wasn't thinking straight. Makes perfect sense; I was just trying to work out deployment in my head. I like the stand alone solution as well.

[regisphilibert]

Very exciting!

[bep]

Or may be, for the folks with go >= v1.11 installed, they can use the go get approach, others can fall back to using git sub modules?

Yes, submodules etc. will still work. Also see the hugo mod vendor thing.

[onedrawingperday]

This looks great. Particularly the planned hugo mod vendor command, since it will make this feature host agnostic.

[larzza]

If I understand this correctly, it means that this could be used to download any git repository to use as a dependency, for example Bootstrap (which I today add as a git submodule in the assets directory for sass customizing)?

[bep]

@larzza this is, in general, true, but there is a caveat: It will be added as a Hugo Theme Component (which has a fixed folder structure: layouts, static, assets ...), so anything outside those will currently have no effect. We could improve on that, but that is outside of the scope of this particular issue.

[bep]

That said, there are some exploration left to do here -- thinking about it, I'm not totally sure how a submodule inside a dependency would work, etc.

[bep]

Google just announced that they are hosting a proxy server for these modules which should speed things up, and it is supposed to make multi-modules easier; yet to be tested.

[Jos512]

I'm concerned that this makes it harder for new people to use Hugo. It introduces an additional dependency (Go), which needs to be installed and updated. Plus that can generate its own error messages.

Every so often on the forum we already get people who get stuck with the well-documented quick start. I imagine that having to install Go and run go commands leads to much more confusion and people giving up on Hugo.

[bep]

@Jos512 what alternative would you suggest?

[regisphilibert]

I thought you had to install Go to use Hugo in the first place...

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

[bep]

If not, couldn't we wrap a default go get into the Hugo CLI? hugo get -u github.com/bep/my-shortcodes

Yes, and we will probably do that (to be able to wrap it with the Hugo stuff we need), but you will even then currently need to have Go installed. I have not looked closely into this, but it looks like the Go Modules stuff isn't available as methods/functions from the outside, so you need to interact with the binary. I do, however, suspect that this will change in the future. I think they have kept this API close to their chest until it is carved in stone.

And no, you don't need Go installed to use Hugo. But I would say that it would feel more natural to me to install Go to get package management than having to install Node/NPM which feels more foreign in this context.

[regisphilibert]

But I would say that it would feel more natural to me to install Go to get package management than having to install Node/NPM which feels more foreign in this context.

I agree. It seemed no one ever complained about having to install node in order to build a website.I don't know of any framework which will install node + npm for you.

[larzza]

It seems kind of strange (but pretty cool) to package (a part of) Go in an executable built by Go. It’s also kind of strange to have a dependency on Go from a stand-alone executable, i.e. Hugo. (I think docker would be of good use to bundle things up, then of course the user has to install docker, but not git and hugo and go and....)

Is it possible to elaborate on the usecase? Is a theme supposed to be a ”Go module”? Or just components in a theme... shortcodes, partials etc? Or is the scenario that it will be possible to mix.

If I develop a theme I guess I would like to have it as a git submodule (or just an ordinary git repo) in the themes directory in my hugo site and then add module dependencies to public available shortcodes and partials I find useful.

[bep]

Is it possible to elaborate on the usecase?

When I start a new site, I want to do add something like this:

imports:
- github.com/bep/mynicetheme
- github.com/bep/common-shortcodes
- github.com/budparr/hugo-components/useful-partials

And then, when I run:

hugo

All of the dependencies (including any transitive dependencies) is set up for me and cached as effectively as possible. Then I want easy ways to update dependencies on change and many of the other benefits that come with a proper package manager.

Most of the above may be possible using Git submodules, but it isn't practical (your head will get dizzy if you start to think about transitive dependencies) -- which is why Hugo Theme Components are so criminally underused.

[larzza]

Thanks, got it!

[gcushen]

Can we statically link Go/Go Modules so that there's no need for users to install Go?

[bep]

@gcushen what/how?

[bep]

I have been doing a quick implementation today. I have some challenges left, but it fits surprisingly well into the existing theme code.

See https://github.com/bep/my-modular-site

For a very lean site ...

A positive side to this that I didn't know before is that you can have submodules in subfolders that can have its own version, which is super useful, see

https://github.com/bep/hugotestmods

[moorereason]

@bep, this is an interesting idea.

I'm not super-excited about requiring Go for theme development. I'm inclined to add this as a "preview feature" similar to who Go handled modules initially (check for HUGO056MODULE=on in env).

I agree that we're all hoping for the day when the Go team exports the modules packages. Can we keep the module feature in preview until they do? I wonder how long that will be. Perhaps we should ask Steve what the team is thinking.

[larzza]

Cool!

I'm not super-excited about requiring Go for theme development.

Isn't the idea that the way we do it today will still exist?

A positive side to this that I didn't know before is that you can have submodules in subfolders that can have its own version, which is super useful, see

https://github.com/bep/hugotestmods

About multi module repos:
https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories

I note that the recommendation is not to use multi module repos, but maybe this is a case of a perfect fit. The possibility to tag a subdirectory in git with its own separate tag was new to me.

[bep]

Isn't the idea that the way we do it today will still exist?

Yes.

[bep]

@larzza @regisphilibert @budparr @RickCogley @digitalcraftsman @onedrawingperday and gang; this branch is now back to a working state and I will wrap it up in a few days. This is a classic example where the first 90% was easy, but the final 10 ... With the introduction of the very flexible disk mounts (now including content) required me to take a step back and redesign parts of the filesystem handling.

The config below is from https://github.com/bep/my-modular-site and shows the syntax.

In short, the goal is a module definition replaces most of the "dir config" in Hugo (configDir, staticDir etc.). I have created adapters to handle the old setup, but this should be much easier to grasp, esp. for bigger projects.

Summary points:

• Every Hugo Component (main project, shortcode collection etc.) can have a module configuration in their config.toml (or similar).
• All of these can define a set of mounts (there are defaults if not sets).
• A mount is:
  • source: Directory path to the source. For the main project, this can be absolute.
  • target: The mount point in the project. This must start with the component folder, e.g. content/blog.
  • lang: Language code, only relevant for content mounts and static mounts when in multihost mode.
• You can also define mounts when you import a component; if so, any mounts definition in the imported component will be ignored. This can be especially useful when importing non-Hugo-projects (e.g. Bootstrap SCSS) or if you only need a small part of a theme.

All of the above can be used with components inside the old themes folder or as components managed (version, dependency management) by the new hugo mod commands -- which is a whole another cool story.

The above may look technical, but is in my head super duper cool.

[module]
[[module.mounts]]
source="content"
target="content"
[[module.imports]]
path="github.com/bep/hugotestmods/mymounts"
[[module.imports.mounts]]
source="myassets/subfolder"
target="assets/images"
[[module.imports.mounts]]
source="mydata/subfolder"
target="data/datakey"
[[module.imports]]
path="github.com/bep/hugotestmods/mypartials"
[[module.imports]]
path="github.com/bep/hugo-fresh"

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.