Long paths on Windows: Cannot even use 260 chars

[kinke]

I'm seeing issues with long paths on a recent Windows 10, with LDC and Visual Studio 2019 in this case.

Errors with the MS linker

E.g.:

LINK : fatal error LNK1181: cannot open input file '..\..\..\..\AppData\Local\dub\packages\unit-threaded-1.0.15\unit-threaded\subpackages\integration\.dub\build\library-debug-windows-x86_64-ldc_v1.27.0-beta1-sym1-55CBD351797D62157A88F9C8E869018F\unit-threaded_integration.lib'

Interesting because the relative path length is only 223, and the absolute one (C:\Users\packer\AppData\…) is 227 chars, nowhere near the 260 limit. Edit: Ah, C:\Users\packer\dev\SIL\lang\test_plugin1\..\..\..\..\AppData\… exceeds 260.

One workaround is using -link-internally with LDC, which makes it use the LLD linker integrated in LDC.

Another workaround is converting the relative path to \\?\C:\Users\packer\AppData\… (in the LDC cmdline issued by dub; these are simply forwarded to the linker). This could be done by LDC as well, but see below. Edit: Nope, link.exe doesn't support the \\?\ prefix for absolute paths > 260.

Superfluous rebuilds

I was about to tackle this in LDC, but then came across another very annoying bit: such dependencies are rebuilt again and again, even if nothing has changed. Running dub with -v shows:

File ..\..\..\..\AppData\Local\dub\packages\unit-threaded-1.0.15\unit-threaded\subpackages\integration\.dub\build\library-debug-windows-x86_64-ldc_v1.27.0-beta1-sym1-4B9CC688E7DF7F085E7CCAC75893DE7E\unit-threaded_integration.lib doesn't exist, triggering rebuild.

(The file obviously exists.) So handling this in the compiler alone isn't sufficient.

[kinke]

So any path-shortening via relative paths starting with ..\ is counter-productive on Windows, as the expanded path is longer than the (normalized) absolute one. I thought the culprit was

dub/source/dub/generators/build.d

Line 223 in 4c1ce74

cbuildsettings.targetPath = shrinkPath(target_path, cwd);

but it's not.

[PetarKirov]

@kinke @Geod24 @s-ludwig Wouldn't the correct solution be to use long Win32 paths? Specifically:

1. Obtain the normalized absolute path
2. Prepend prepend \\?\
3. Call the Win32 File I/O functions with W suffix

( I haven't looked at the code in detail and It's been a while since I did Win32 programming or used Windows on a daily basis, so please correct if I'm wrong.)

Edit: Some links regarding the long path support transition for .NET:

"Long Paths in .NET" by Kim Hamilton (blog post from 2007):

• part 1: https://docs.microsoft.com/en-us/archive/blogs/bclteam/long-paths-in-net-part-1-of-3-kim-hamilton
• part 2: https://docs.microsoft.com/en-us/archive/blogs/bclteam/long-paths-in-net-part-2-of-3-long-path-workarounds-kim-hamilton
• part 3: https://docs.microsoft.com/en-us/archive/blogs/bclteam/long-paths-in-net-part-3-of-3-redux-kim-hamilton

CoreFX (issue):

• Allow extended long path directories dotnet/corefx#2700
• Unblock paths > MAX_PATH without extended syntax. dotnet/corefx#3001
• Clean naming and disable stack pathhelper dotnet/corefx#3010

CoreCLR (issue):

• Support LongFile in the VM dotnet/coreclr#2998
• Enabling LongFile Support in PAL dotnet/coreclr#3441
• Support long paths in CoreCLR runtime on Windows dotnet/coreclr#3433

[s-ludwig]

True, UNC paths would be the proper fix, although we'd have to try with all involved linkers and compilers whether they are able to handle those. It would also be nice though if Phobos would automatically perform that conversion, if needed.

[kinke]

As stated above, Microsoft's linker doesn't support long paths as of VS 2019. From the dub side, there are things that can be done, like using the wide I/O functions consistently. This issue was about dub further shrinking the max path length from 260 to ~220 though, depending on where your src repos are cloned to.

[WebFreak001]

@kinke you said Microsoft's linker doesn't support long paths anymore - do you mean it doesn't support UNC paths?

[kinke]

It has never supported them - IIRC, the \\?\ prefix works, but doesn't make absolute paths > 260 chars work.

[kinke]

WTF - just tested it again to be sure, and it (the MS linker) does seem to work now with an absolute path > 300 chars, not even requiring any prefix. This is on my native Win10 (21H1) box, the tests ~2 weeks ago were in a Win10 VM (20H2 or 20H1); I'm not sure whether the VS 2019 installation in the VM is totally up-to-date - here on my native box, it's v16.10.1.

[kinke]

Seems independent from Windows, as it's not working with VS 2017 v15.9.25. VS 2019 v16.10 was a pretty recent upgrade IIRC, so I think it may have made the difference.

[PetarKirov]

Just FYI, there's an option on the Windows side to enable support for larger paths. I'm pretty sure that not much software is taking it into an account (especially if they assumed that MAX_PATH can't change at runtime), but perhaps this new VS update is forward compatible with it?
https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd#enable-long-paths-in-windows-10-version-1607-and-later

[kinke]

I've checked the VS 2019 link.exe manifest, and it still doesn't specify longPathAware, so the registry/GPO hack shouldn't have any effect. (But hey, it's working anyway.) On a side note, I've also had to create the file in the console, as Windows Explorer apparently refuses to generate files with absolute paths > 260 chars...

---

I've also quickly looked into the dub file-exists check, which boils down to std.file.exists, which uses GetFileAttributesW IIRC; so that's apparently a Phobos issue and would probably need to add the \\?\ abomination. Edit: I.e., something similar to DMD's https://github.com/dlang/dmd/blob/ef39b070819216a8d9835e2a66d3e365420763ec/src/dmd/root/filename.d#L1146.

[kinke]

Filed https://issues.dlang.org/show_bug.cgi?id=22059.