Page resources duplication for translations (even in 0.123)

[TiGR]

There is an issue #11453 already dealing with somewhat similar problem. It is marked as solved with 0.123.0, but that solution only works with md files. In our case, all translations are in html format, and all the resources are duplicated to all the languages and there is no way to prevent this from happening. In our case it takes dozens of gigabytes of extra space, creating all sorts of problems. We even tried avoiding this by loading the resources only from the base language, but it doesn't matter as hugo still duplicates all the resources to all the languages.

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.123.0-DEV+extended linux/amd64 BuildDate=unknown

Does this issue reproduce with the latest release?

Yes.

[bep]

I'm not sure I follow.

• Don't duplicate page resource output per language #11453 talks about bundled non-content files (e.g. images)
• Content files used as resources (in leaf bundles), has never (as I remember) been duplicated. They're assigned to a language.

I may be misunderstanding this, but then you need to provide me with a full and running example project.

[TiGR]

We have something like this:

content/
└── news/
    └── 2023/
        └── 01/
            └── 1/
                ├── image1.jpg
                ├── image2.jpg
                ├── index.md
                ├── index.es.html
                └── index.fr.html

We get this built:

news/
└── 2023/
    └── 01/
        ├── image1.jpg
        ├── image2.jpg
        └── 1.html
es/
└── news/
    └── 2023/
        └── 01/
            ├── image1.jpg
            ├── image2.jpg
            └── 1.html
fr/
└── news/
    └── 2023/
        └── 01/
            ├── image1.jpg
            ├── image2.jpg
            └── 1.html

All images are duplicated in all languages.

[bep]

Is this a multihost setup? As in: Do you have multiple baseURLs?

[TiGR]

No, we have only one baseURL

[TiGR]

BTW, you tagged it as regression, but it is not - resources were always duplicated for us. We hoped that 0.123 would fix this, but it does not.

[bep]

We have something like this:

Again, we have certainly tested this. This behaviour is disabled if you have either

• Multihost setup
• Or markup.goldmark.duplicateResourceFiles config is set to true.

If you can give me a failing and running test case, I can look at it. But I cannot guess from the above.

[TiGR]

@bep this issue is still there with the latest commits.

I've created reproduction case for this issue.

So, the problem is, that we use a lot of images through shortcodes, since we need complicated code to be generated (carousels, complicated images, etc). And all these images are being duplicated into all languages. The code reproduces it - file featured.png from news/1 is duplicated both in public/news/1/featured.png and in public/es/news/1/featured.png.

[bep]

Yea, sorry, I closed the wrong issue. Thanks for the repro, I will look at it later today.

[bep]

OK, I have found the culprit. I had a head-scratching moment with this one. @jmooring if I could borrow your brain for a minute.

We added a

markup.goldmark.duplicateResourceFiles

Config, which is default false for Markdow/Goldmark, but

• Is true in Multihost sites
• It is always true for other Markups (e.g. HTML)

The last bullet is this issue. I guess @TiGR 's challenge is that there's no way for him to configure it his way. We could always push this problem to another day, it behaves like before for HTML and friends.

[artch]

I'm watching this thread and my 2 cents is that it's also an issue for me. Having the same localized page in different formats (md/html) makes perfect sense for me. Page resources management should not be related to page markup format, it feels like an unexpected behavior.

[bep]

Page resources management should not be related to page markup format, it feels like an unexpected behavior.

It behaves the same as before, which I guess is not unexpected. The reasoning behind the current behaviour is that for Markdown/Goldmark we have render hooks which provides "portable links". You have fixed it with a shortcode, but this isn't the case for all HTML users.

[artch]

It behaves the same as before, which I guess is not unexpected.

Before this change it worked the same way for Markdown and HTML. A non-optimized way but at least the same way. After this change it will start to behave differently depending on file format which will cause confusion. The idea that it's tied to Goldmark configuration is a little bit hidden and will require efforts to realize.

In my opinion if there is no way to make it work for both Markdown and HTML pages identically, this option should be disabled by default. However, for me personally proper full support of this feature would be a game changer in regard to resources consumption optimization in CI/CD pipelines.

[bep]

@artch yea, I don't think we disagree about this; it's just that we (mostly I and @jmooring) have spent a fair amount of thinking about ways to do this (and some other changes) with the least amount of noise from people shouting that we somehow broke their site. We can still fix this, though.

We could possibly add a duplicateResourceFiles (default false) to the top level, so the defaults would be:

markup.duplicateResourceFiles=true
markup.goldmark.duplicateResourceFiles=false

[TiGR]

That would be perfect!

[jmooring]

We could possibly add a duplicateResourceFiles (default false) to the top level, so the defaults would be:

markup.duplicateResourceFiles=true
markup.goldmark.duplicateResourceFiles=false

The proposed default value of markup.duplicateResourceFiles is not clear to me.

[bep]

The proposed default value of markup.duplicateResourceFiles is not clear to me.

Thinking about it, it's not clear to me either. I'm closing this for now, as this certainly works as designed and this isn't breaking behaviour vs how it behaved.

We need to finish this release and get it out the door without creating new problems.

Please create a new proposal for the above use case outlining how this should work.

[artch]

@bep I understand that you have put some efforts into designing this. You surely have reasons to implement things as you see fit. I am just providing feedback from an outside person that linking page resource management features to formatting configuration options looks highly counterintuitive to a Hugo site maintainer, despite the fact it seems logical to you. If there is a chance that it will have to be redesigned later, it's better to spend additional time now and make it right from the beginning.

[jmooring]

linking page resource management features to formatting configuration options looks highly counterintuitive

Well, we could put config options anywhere we want. But it would not change the underlying relationship with content format. Markdown is the only content format that provides render hooks, the critical components that allow the de-duplicated, multilingual, single-host approach to work without shortcodes. And the chances of creating render hooks for html, adoc, pdc, rst, and org are zero.

What we have now is backwards compatible.

You're certainly welcome to suggest changes, but you'll need to create a new issue (proposal) for that.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.