Umlauts in static filenames fail to sync/serve with "hugo server" on macOS

[danst0]

Hi,

I have two files in my static files directory 00a.png and 00ä.png. For the 00ä.png file I get a 404. The 00a.png file is found fine.

Daniel

[anthonyfok]

Hi @danst0,

Which OS are you using? Is it macOS?

[anthonyfok]

If you are indeed running macOS, please check whether your filesystem is HFS+ or the newer APFS, and share with us here.

Also, please tell us whether you are testing locally with hugo server or with a local or remote web server, e.g. you uploaded the files to a Linux server running Apache?

The reason I am asking is that macOS uses Unicode Normalization Form D (NFD) whereas the rest of the world uses Normalization Form C (NFC) for filenames.

Or maybe it was just macOS HFS+; the new APFS does not normalize to NFD any more. In that case, things become slightly more complicated, and I wonder if Hugo and/or https://github.com/spf13/afero needs to do anything special...

Anyhow, here is a relevant somewhat reading list:

• Static sync to memory fails with Norwegian "å" (fixed): Static sync to memory fails with Norwegian "å" #2203
• MacOS: Umlauts in filenames: cat: ö: No such file or directory: MacOS: Umlauts in filenames: cat: ö: No such file or directory vgough/encfs#316
• APFS’s “Bag of Bytes” Filenames: https://mjtsai.com/blog/2017/03/24/apfss-bag-of-bytes-filenames/ (read the comments too)
  • https://lists.apple.com/archives/cocoa-dev/2017/Mar/msg00154.html

    “iOS HFS Normalized UNICODE names, APFS now treats all files as a bag of bytes on iOS. We are requesting that Applications developers call the correct Normalization routines to make sure the file name contains the correct representation.”

• Unicode Normalization Forms: http://unicode.org/reports/tr15/
• https://en.wikipedia.org/wiki/Unicode_equivalence
• https://blog.zhiheng.io/ch4-text-versus-bytes/#normalizing-unicode-for-saner-comparisons (Chinese)
• "Unicode on Mac is insane. Mac OS X uses NFD while everything else uses NFC. This fixes that." (gist for using convmv and its --nfc / --nfd flags to convert between the two forms): https://gist.github.com/JamesChevalier/8448512

Update by Michael Tsai on 2017-06-27:

• APFS Native Normalization: https://mjtsai.com/blog/2017/06/27/apfs-native-normalization/ (read the comments too, which goes to January 2018)

[danst0]

Thanks for the quick reply!

I am using MacOS and APFS; Hugo runs as "hugo server".
I am not sure if this really is a problem with the normalization.
I try to access the url:

http://localhost:1313/pubthumb/00ä.png

, which gives me a 404
With the same keyboard on the same computer I have before created and named the file 00ä.png.
--> so URL and filename should be the same Unicode?

Furthermore.
If I do

pubthumb ❯❯❯ python -m SimpleHTTPServer in the relevant directory, I can access my file just fine

under:

http://localhost:8000/00ä.png

[anthonyfok]

Hi @danst0,

Please try hugo server --renderToDisk and see if you could access http://localhost:1313/pubthumb/00ä.png like you could with python -m SimpleHTTPServer.
Thanks!

[danst0]

Thanks! renderToDisk did the trick! Now it supports the umlauts properly. Still weird behavior.

Daniel

[anthonyfok]

Thank you for your testing and feedback!

It looks like Issue #2203 all over again, which was supposedly fixed back in 2016.
Maybe it wasn't entirely fixed? Or circumstances changed? Need to investigate.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[pfhawkins]

I attempted to reproduce this error on a mac running Mojave and APFS, but I was unable to. I think I might be using bad test data, as I named the file ä.jpg right on the command line.

@danst0 could you please point me to an already-named file online I could test with? I have a hypothesis for what the problem might be, but I want to make sure I can reproduce the issue before exploring it further.

[pfhawkins]

As it turns out, today I was able to reproduce by downloading https://upload.wikimedia.org/wikipedia/commons/thumb/6/6a/Per_Krafft_d_%C3%A4%3B_Portr%C3%A4tt_av_Carl_Christoffer_Gj%C3%B6rwell_d_%C3%A4.jpg/525px-Per_Krafft_d_%C3%A4%3B_Portr%C3%A4tt_av_Carl_Christoffer_Gj%C3%B6rwell_d_%C3%A4.jpg directly from Chrome into my static dir, as opposed to using wget.

[davidsneighbour]

This is still an issue, this time in Windows.

https://discourse.gohugo.io/t/gitinfo-fails-if-there-is-an-umlaut-in-the-folder-path/32746

[jmooring]

Cannot reproduce on macOS 13.3.1 (APFS). See https://discourse.gohugo.io/t/44336

This is still an issue, this time in Windows.
https://discourse.gohugo.io/t/32746

That is a separate issue, #9810.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.