internal fmtutil calling on windows does not work

[cderv]

> tinytex:::fmtutil()
...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:941: ...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:864: no appropriate script or program found: fmtuti~1

This is not an issue with tlmgr_update which call this function because we set stdout = FALSE

> tinytex:::fmtutil(stdout = FALSE)

which hide the error

So something is not right.

> system2("fmtutil-sys", "--all")
...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:941: ...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:864: no appropriate script or program found: fmtuti~1
> system("fmtutil-sys.exe --all")
...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:941: ...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:864: no appropriate script or program found: fmtuti~1
[1] 1

However, processx::run("fmtutil-sys", "--all") works ok, and sys::exec_wait("fmtutil-sys", "--all") also.

Found while looking at quarto-dev/quarto-cli#7252 (which means that solution in TinyTeX was not working on windows 🤔 )

This is probably a shell thing. Other commands that works

• xfun:::powershell("fmtutil-sys --all")
• shell("fmtutil-sys --all")

Not sure how to tweak system2() to solve this problem 🤔

[yihui]

That's interesting. Not sure why system2() preferred using the shortened path. Does system('fmtutil-sys --all') work?

[cderv]

Does system('fmtutil-sys --all') work?

No it doesn't

> system('fmtutil-sys --all')
...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:941: ...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:864: no appropriate script or program found: fmtuti~1
[1] 1

[cderv]

Not sure why system2() preferred using the shortened path

Maybe because R Sys.which() does use it (because of the - ? ) 🤔

> Sys.which("fmtutil-sys")
                                                              fmtutil-sys 
"C:\\Users\\chris\\AppData\\Roaming\\TinyTeX\\bin\\windows\\FMTUTI~1.EXE" 
> xfun::normalize_path(Sys.which("fmtutil-sys"))
[1] "C:/Users/chris/AppData/Roaming/TinyTeX/bin/windows/fmtutil-sys.exe"
> system2(xfun::normalize_path(Sys.which("fmtutil-sys")), "--all")
...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:941: ...\chris\AppData\Roaming\TinyTeX\bin\windows\runscript.tlu:864: no appropriate script or program found: fmtuti~1

but as you see using full path does not help...

It seems really related to how system() and system2() works by not calling a shell really, as explicitly asking to run command through CMD works

> system('cmd /c fmtutil-sys --all')

Which is probably the equivalent of base::shell()

[yihui]

I think this is because system() and system2() will use short paths when executing commands: https://github.com/wch/r-source/blob/635a672151a18a0e475986af592fab59e7479a9b/src/gnuwin32/run.c#L141 But TeX Live can't deal with short paths.

Perhaps we need to switch to shell() when a command's path would be shortened. Let me first ask on r-devel why paths need to be shortened (I know one reason is to avoid spaces in paths).

[yihui]

BTW, it will be nice if R could provide an option to disable shortening paths, but this might also be solved on TeX Live's end if TeX Live could recognize short paths. Cc @norbusan: you don't need to read the full thread here---the problem that we ran into was that calling fmutil-sys.exe on Windows using its short path FMTUTI~1.EXE doesn't work. I wonder if there's a chance to fix it on TeX Live's side. This sounds like a recent breakage, since we have never received a report in the past few years (now we have received multiple). Thanks!

[norbusan]

@yihui I am not sure how we should fix that. Do you suggest that instead of calling system("fmtutil.exe ...") we should call system("FMTUTI~1.EXE ,,,")?

BTW, what happens if there are multiple fmtutil? How do we know which one is the correct one?

[yihui]

No, I'm not suggesting that. From my limited understanding of runscript.tlu, the error came from the line 864: https://tug.org/svn/texlive/trunk/Master/bin/windows/runscript.tlu?revision=68579&view=markup#l864 When fmutil-sys.exe was invoked by the name FMTUTI~1.EXE (which should be valid on Windows), find_texmfscript() was unable to find the correct executable. In R, I could use normalizePath() to resolve the short path to the long path. I don't know how to do that in TeX Live or Lua.

As I said, this seems to be a recent change, because no one has reported this problem for the past few years. I could be wrong and this may be a certain coincidence (in other words, it might have existed for a long time but didn't surface until another problem occurred).

BTW, what happens if there are multiple fmtutil? How do we know which one is the correct one?

I don't know the answer in general, but in our case, we are calling a specific bin/windows/fmtutil-sys.exe in the root directory of TinyTeX.

[yihui]

When fmutil-sys.exe was invoked by the name FMTUTI~1.EXE (which should be valid on Windows), find_texmfscript() was unable to find the correct executable.

@norbusan This was also verified by an R core team member: https://stat.ethz.ch/pipermail/r-devel/2023-October/082979.html They said they wouldn't "fix" it in R, so our only hope is that TeX Live would accept short paths of commands.

[norbusan]

@yihui sorry for the long silence, I lost track of that.

What about if you call fmtutil.exe --sys ...other arg... ? That should result in an un-shortened fmtutil.exe call and all should be fine?

[yihui]

@norbusan This is definitely a workaround but only because the name fmtutil has less than 8 characters. I'm not sure which executables would first go through runscript.tlu. There might be other ones that have more than 8 characters such as kpsewhich.exe. So it will be nice if runscript.tlu can accept short paths.

[norbusan]

kpsewhich is an actual .exe, but fmtutil, updmap are scripts.
Anyway, I have asked on the luatex-dev mailing list whether there is a way to convert "short" names to long names.

I just fear that passing in only the short name like FMTUTI~1.EXE will not suffice to resolve it back to the long name. But honestly, I have no idea about Windows ...

[yihui]

Thank you very much! I have changed fmtutil-sys to fmtutil --sys (also updmap-sys to updmap --sys) as you suggested. I think we are good for now.

[norbusan]

@yihui Thanks to Akira and Luigi, we have updated runscript DLL to get the long name by itself.
So with a current/uptodate TeX Live it should now work even with fmtutil-sys.
Could you try and report back? Thanks

[yihui]

@norbusan I can confirm that the new runscript.dll works perfectly now. Thank you and Akira/Luigi so much!

However, a deeper problem is that TeX Live doesn't work when installed to a directory that contains non-ASCII characters in its path, as Akira mentioned in the mailing list. It will be great if this limitation can be removed. In the meantime, I'll also try to come up with a installation path that doesn't contain non-ASCII characters.

[norbusan]

Hi @yihui
Thanks for confirmation of the fix, much appreciated!

Concerning the non-ASCII characters in its path, I really don't know whether this is easy to fix or not. I will discuss with the Japanese Development Team who often helps with fixing these kinds of things.