path/filepath: EvalSymlinks of a directory which is mapped from volumeID (instead of drive letter) does not work

[jingxu97]

What version of Go are you using (go version)?

go 1.13

$ go version

Does this issue reproduce with the latest release?

Not tested

What operating system and processor architecture are you using (go env)?

set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows

go env Output

$ go env

What did you do?

Create a symlink from volume id
mklink /D \my\path \?\Volume{1b3b1146-4076-11e1-84aa-806e6f6e6963}\

Running filepath.EvalSymlinks against a symlink (\my\path) which is created from a volumeid returns the following error:

too many links!

What did you expect to see?

What did you see instead?

[cagedmantis]

/cc @robpike @rsc

[jingxu97]

I found a way to use Powershell command to workaround this issue in kubernetes code.
kubernetes/kubernetes#92328

[networkimprov]

EDIT: Related: #39785

cc @alexbrainman @bcmills

@gopherbot add OS-Windows

[gopherbot]

Change https://golang.org/cl/241278 mentions this issue: path/filepath: try GetFinalPathNameByHandle with VOLUME_NAME_GUID

[networkimprov]

@ericwj can you look at fixing #39785 in your fix for this?

Also, if you're interested in fixing other GOOS=windows issues, there's a handful I could point you to :-)

[ericwj]

Man I am so annoyed about this for years now that I learned Go just to fix this - I've written about 100 lines total now...

[ericwj]

Consider weighing in on #40104 to see issues like this finally be resolved once and forever.

[alexbrainman]

@jingxu97

I tried reproducing your problem on current tip (it will be part of go1.15 soon), and I cannot. In particular, I tried an existing Go library test, that does exactly what you are describing in this issue.

c:\>go version
go version devel +3a4322627e Fri Jul 10 11:32:36 2020 +0000 windows/amd64

c:\>go test -v -run=NTName path/filepath
=== RUN   TestNTNamespaceSymlink
--- PASS: TestNTNamespaceSymlink (0.06s)
PASS
ok      path/filepath   0.087s

c:\>

Can you, please, try to run this test yourself (you will have to be the Administrator) and tell us, if it fails or passes? If you think your situation is different, then, please, provide complete reproduction steps, and not just vague description.

Thank you.

Alex

[networkimprov]

This is the code Alex refers to: https://github.com/golang/go/blob/master/src/path/filepath/path_windows_test.go#L538

@alexbrainman according to the kubernetes issue linked above, there's no drive letter assigned to this volume.

[alexbrainman]

@alexbrainman according to the kubernetes issue linked above, there's no drive letter assigned to this volume.

How do I reproduce this? The issue above asks me to use mklink command. So I did.

Alex

[networkimprov]

I believe you create a VHDX and don't assign it a drive letter. Then mklink /D with that volume as target.

[ericwj]

Yes, @networkimprov is correct. This is a very common configuration on Windows that go should not pre-empt with bugs.

The PowerShell:

$vhd = New-VHD -Path Test.vhdx -SizeBytes 3MB -Fixed
$vhd | Mount-VHD
$vhd = Get-VHD Test.vhdx # refresh IsAttached etc
$vhd | Get-Disk | Initialize-Disk -PartitionStyle GPT
$part = $vhd | Get-Disk | New-Partition -UseMaximumSize -AssignDriveLetter:$false
$vol = $part | Format-Volume -FileSystem NTFS

cmd /c mklink /j Test $vol.Path
[System.IO.File]::WriteAllText("$PWD\Test\Dockerfile", "FROM scratch")
docker build Test

# Cleanup
Dismount-Vhd Test.vhdx
rmdir Test
del Test.vhdx

The output from docker is this time:

unable to prepare context: unable to evaluate symlinks in context path: CreateFile C:\tools\Volume{9d7e0c41-a29a-4cea-b0f8-f16693425a69}: The system cannot find the file specified.

[ericwj]

This fails, I believe about exactly the same, configuring Storage Spaces and wanting to put docker on a VirtualDisk as opposed to a Hyper-V VHD. So now docker doesn't run on any kind of serious Windows Server farm.

[ericwj]

I don't know if my go is actually patched with #40095... Your output will be different in that case. This might be bug no. 2 on this code path, seeing that go somehow needed to add an absolute path the the current working directory which it should have replaced resolving the junction at "$PWD\Test".

[ericwj]

Sorry @jingxu97, I didn't get any notifications from this somehow.

Anyway yes, the test passes, but obviously they are very incomplete:

go test -v -run=NTName path/filepath
=== RUN   TestNTNamespaceSymlink
--- PASS: TestNTNamespaceSymlink (0.10s)
PASS
ok      path/filepath   0.396s

[ericwj]

I don't know if my go is actually patched with #40095...

It is and I tested my hello.go to correctly resolve symlinks, but doing that today I get in an endless loop in walkSymlinks which after 255 tries will still claim too many links in symlink.go#L99.

For one it by now has path: "C:\Docker\Mount\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\...+213 more" on the stack where it started with C:\Docker\Mount.

So this fix is partial. Obviously I also did not add any tests, changing only one file.

[ericwj]

This is the culprit

PS C:\Docker> dir


    Directory: C:\Docker

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           6/13/2020  1:50 AM                DockerWeb
l----           6/13/2020  4:20 AM                Mount -> Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
d----           6/13/2020  1:27 AM                Other
-a---           7/11/2020  6:12 PM       37748736 Disk.vhdx

PS C:\Docker> (Get-VHD .\Disk.vhdx) | Get-Disk | Get-Partition | select -ExpandProperty AccessPaths
C:\Docker\Mount\
\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\

I'm not sure how C:\Docker\Mount became an access path for the volume, but this means that in this setup the #40095 fix isn't applicable, because GetFinalPathNameByHandle(... VOLUME_NAME_DOS) will return "\\?\C:\Docker\Mount\" with a handle created from C:\Docker\Mount and hence GetFinalPathNameByHandle(... VOLUME_NAME_GUID) will not be tried.

After running

(Get-VHD .\\Disk.vhdx) | Get-Disk | Get-Partition | Remove-PartitionAccessPath -AccessPath C:\Docker\Mount\

I run in yet another problem, although I like the first part of it - symlink.go#L88 will say C:\Docker\Mount is not a symlink, followed by symlink.go#L83 Lstat yielding (null, 2) and the result hence becomes:

API server listening at: 127.0.0.1:47251
Abs [<nil>] C:\Docker\Mount\Subdir => C:\Docker\Mount\Subdir
Dir [nul] C:\Docker\Mount\Subdir => C:\Docker\Mount
EvalSymlinks [CreateFile C:\Docker\Mount\Subdir: The system cannot find the file specified.] C:\Docker\Mount\Subdir => (empty string)
```

And still my fix isn't applicable. It is such a mess.

[ericwj]

Removing the C:\Docker\Mount\ access path in fact made Mount a regular directory, so at least fi.Mode()&os.ModeSymlink == 0 being true was correct about that, but shouldn't EvalSymLinks in that case just return the original path?

Running

PS> rmdir Mount
PS> cmd /c mklink /j Mount \\?\Volume...

my fix becomes applicable.

API server listening at: 127.0.0.1:12289
Abs [<nil>] C:\Docker\Mount\Subdir => C:\Docker\Mount\Subdir
Abs [nul] C:\Docker\Mount\Subdir => C:\Docker\Mount
EvalSymlinks [<nil>] C:\Docker\Mount\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
IsAbs [<nil>] C:\Docker\Mount\Subdir => true
VolumeName [<nil>] C:\Docker\Mount\Subdir => C:
ResolveAndValidateContextPath [<nil>] C:\Docker\Mount\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
ResolveAndValidateContextPath [<nil>] C:\Docker\Mount\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
GetContextFromLocalDir [<nil>] C:\Docker\Mount\Subdir => (\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir, Dockerfile)
Process exiting with code: 0

[ericwj]

Finally, to reproduce against the original issue text above:

PS C:\Docker> cmd /c mklink /D DirectorySymbolicLink "\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\"
symbolic link created for DirectorySymbolicLink <<===>> \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
PS C:\Docker> hello
API server listening at: 127.0.0.1:16921
Abs [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => C:\Docker\DirectorySymbolicLink\Subdir
Abs [nul] C:\Docker\DirectorySymbolicLink\Subdir => C:\Docker\DirectorySymbolicLink
EvalSymlinks [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
IsAbs [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => true
VolumeName [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => C:
ResolveAndValidateContextPath [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\
ResolveAndValidateContextPath [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => \\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir
GetContextFromLocalDir [<nil>] C:\Docker\DirectorySymbolicLink\Subdir => (\\?\Volume{93183f0f-24a1-4a9e-84ff-bd8ecebf5139}\Subdir, Dockerfile)
Process exiting with code: 0

[ericwj]

So this fix is partial.

This fix is complete for the situation that is described.

[alexbrainman]

@ericwj

You seems to be talking to yourself. I am still confused where are the repro steps. Could you point me to the steps, please?

Thank you.

Alex

[networkimprov]

@alexbrainman use this powershell script to make a vhdx and symlink, then pass the symlink to filepath.EvalSymlinks.

$vhd = New-VHD -Path Test.vhdx -SizeBytes 3MB -Fixed
$vhd | Mount-VHD
$vhd = Get-VHD Test.vhdx # refresh IsAttached etc
$vhd | Get-Disk | Initialize-Disk -PartitionStyle GPT
$part = $vhd | Get-Disk | New-Partition -UseMaximumSize -AssignDriveLetter:$false
$vol = $part | Format-Volume -FileSystem NTFS

cmd /c mklink /j Test $vol.Path

package main

import ("path/filepath"; "fmt")

func main() {
   p, err := filepath.EvalSymlinks("Test")
   if err != nil { panic(err) }
   fmt.Println(p)
}

Related bug: #40176

[ericwj]

I find that the feature names differ for desktop and server. Also on server, the command I gave previously isn't enough, since the PowerShell functionality is not actually a subcomponent of Hyper-V itself.

For Desktops:

PS> Get-WindowsOptionalFeature -Online -FeatureName *Hyper-V* | ft
PS> Enable-WindowsOptionalFeature Microsoft-Hyper-V-All -Online # will install Microsoft-Hyper-V-Hypervisor

It could be possible that New-VHD starts working with just Microsoft-Hyper-V-Management-PowerShell and Microsoft-Hyper-V-Services such that the hypervisor isn't needed, but I haven't tried de-installing anything:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Management-PowerShell, Microsoft-Hyper-V-Services

On server, the feature installation is unfortunately less detailed such that the provider services and Hyper-V cannot be selected separately, as mentioned.

PS> Install-WindowsFeature Hyper-V -Restart
PS> Install-WindowsFeature Hyper-V-PowerShell
PS> Get-WindowsFeature -Name *Hyper-V*

Display Name                                            Name                       Install State
------------                                            ----                       -------------
[X] Hyper-V                                             Hyper-V                        Installed
        [X] Hyper-V Management Tools                    RSAT-Hyper-V-Tools             Installed
            [X] Hyper-V Module for Windows PowerShell   Hyper-V-PowerShell             Installed

[alexbrainman]

Install Hyper-V in a machine running in VirtualBox. That can work if virtualbox can be made to passthrough the virtualization.

I don't have operating system that runs inside of VirtualBox that supports Hyper-V. I only have Linux and Windows 7 VMs.

Alex

[networkimprov]

Alex, I think Eric can make a reproducer CL that runs on a Windows builder instance.

We just need to know how a Go test program (run by the builder) can call a local script. Where in the tree should the test program & script live, and what path should exec.Command() use for the script?

The test program looks something like:

// +build windows

package main

import (
   "os/exec"
   "fmt"
   "os"
   "testing"
)

const kPath = "dir"
var kArgs = []string{"."} // args in separate strings
   // re quoting, see https://golang.org/pkg/os/exec/#Command

func TestScript(i *testing.T) {
   aCmd := exec.Command(kPath, kArgs...)
   aCmd.Stdin, aCmd.Stdout, aCmd.Stderr = os.Stdin, os.Stdout, os.Stderr
   err := aCmd.Run()
   if err != nil {
      fmt.Fprintf(os.Stderr, "%v\n", err)
      i.Fail()
   }
}

[ericwj]

To get Windows is really easy.

• I suspect you are not a Windows Insider, but it is really easy to register - assuming you already have a Microsoft Account
• Then you can download an ISO file of the Release Preview Channel from Windows Insider Preview Downloads

There is also always the option to download an evaluation version, for example Windows 10 Enterprise.

The evaluation version will be good for half a year. The insider previews also come with a version-specific expiration date, but I believe those are valid a little longer.

[ericwj]

Is this still true? The post is from 2018.

How to enable nested virtualization on Google Cloud

Nested virtualization does not currently support Windows instances.

I'm not sure if that would be a blocker, since I have no clue whether that also means VM's inside such a machine can run Windows. That would at least be very unhandy, having to run some *nix with a Windows VM inside it with nested virtualization enabled twice just to create some VHD's.

But it would be very easy to just run go tests on Microsoft Azure.

• How to enable nested virtualization in an Azure VM.

The script this program calls can be run with pwsh -File but

• It is easiest to have PowerShell 7 already installed or it would have to invoke a PowerShell 5 script just to download and install that - I get all sorts of problems running my script on PowerShell 5 which I don't feel like fixing and the version of .NET it runs on has blocking problems which might never be fixed over compatibility concerns.
• It should probably not call Install-WindowsFeature. To be able to depend on Hyper-V PowerShell being available (or crash over it), the machine should be setup with Hyper-V before testing go. For one in my experience it requires a reboot.

[ericwj]

That said any number of disks attached to a regular VM would be sufficient to demonstrate the issues - as long as the machine uses GPT any and all partitions will have partition access paths and can be configured to use any drive letters or not, access paths and links and junctions.

The repro steps include this line

$part = $vhd | Get-Disk | New-Partition -UseMaximumSize -AssignDriveLetter:$false

But all that is needed is to initialize the $part variable with a partition object that is OK to be modified, e.g.:

$part = Get-Disk -Number 0 | Get-Partition | where Type -eq Basic | sort Size | select -First 1
$part = Get-Partition -DiskNumber 0 -PartitionNumber 3
$part = Get-Partition -DriveLetter C

Then these commands allow reproducing the issue against any partition on the machine:

# list
$part.AccessPaths
# directly configured access path
$part | Add-PartitionAccessPath -AccessPath foofullpath
$part | Remove-PartitionAccessPath -AccessPath foofullpath
# directory junction
mklink /j foo ($part | Get-Volume).Path
rmdir foo
# hardlink
mkdir /d foo ($part | Get-Volume).Path
rmdir foo

although beware that the issues do not reproduce if the partition already has a drive letter or any other DOS device path with a drive letter.

Now to get a partition that is OK to be modified (or several), run:

$boot = Get-Partition -DriveLetter C
$size = $part.Size - 128MB
$boot | Resize-Partition -Size $size
$disk = $boot | Get-Disk
for ($i = 0; $i -lt 4; $i++) {
    $part = $disk | New-Partition -Size 8MB -AssignDriveLetter:$false
    $part | Format-Volume -Label "TestPart$i" -FileSystem NTFS
}

for example. I am not sure about minimum sizes and hence how much space is needed. But with these examples you can make it any way you like, on any disk you like. At least if the disk is directly attached - external disks over USB and SD cards and the like are weird, might not be formatted with GPT or look to Windows like that, even if they are and might not be able to have more than one partition.

[networkimprov]

Eric & Alex, let's focus on creating a VHDX on a Go builder for Windows, as any Go project member could run a builder.

Creating a CL with the repro script should be easy, we just need the info I listed above.

[alexbrainman]

We just need to know how a Go test program (run by the builder) can call a local script. Where in the tree should the test program & script live, and what path should exec.Command() use for the script?

Normally I create temp directory, and put whatever I need there. And delete after test is finished.

• I suspect you are not a Windows Insider, but it is really easy to register - assuming you already have a Microsoft Account

I don't have Microsoft Account. I don't want to create an account just to run this repo. If nothing else, I can create Windows VM on Google Cloud or something. But that would take me long time, because I need to install all appropriate tools. And I don't install Windows every day, so, I expect, it will take me awhile to figure out.

Is this still true?

I have no idea.

That said any number of disks attached to a regular VM would be sufficient to demonstrate the issues - as long as the machine uses GPT any and all partitions will have partition access paths and can be configured to use any drive letters or not, access paths and links and junctions.

I don't know what "machine uses GPT" means. If you give me steps to repro, I will try them. Thank you.

Alex

[ericwj]

I don't know what "machine uses GPT" means.

GPT as opposed to MBR.

See also Hard Drives and Partitions | Microsoft Docs.

On Windows it is just massively much easier to do things like resizing partitions and add a few than on *nix. All the steps you need have been described here and here. You can also use pointy clicky Disk Management or command line tools that won't have you read massive amounts of man pages and then still have you try and hopefullly not die and hack the kernel: diskpart.

[gillg]

I have exactly the same issue and this problem is anoying for a better automation on Windows.
To decouple infra/software provisioning I try to apply a "unix style" mounts in C:. The idea is the use the feature of NTFS AccessPath (which are basicaly a simple Junction). Thanks to that the software provisioning automation can use more predictible path wich will work all the time, nevermind if the infra provisioning decided or not to add a dedicated volume in a subpath.
On windows Junctions and SymLinks are not the same thing and golang should not threat them as same !

A Symlink can be done on a file or folder and point to another filer or folder or even Junction. But it's virtual.
A Junction is really a "mount point", and it's what happen when you use the powershell command Add-PartitionAccessPath mantioned above.

The function EvalSymlink should not try to extrapolate a Junction, it's like if in unix you try to convert "/mnt/myvolume" to "/dev/sda1"
A windows junction object has a Type "Junction", but the target should be an identifier to a VolumeID :

Exemple :

PS C:\var> Get-Item .\lib\ | Format-List

    Directory: C:\var

Name           : lib
CreationTime   : 7/12/2022 2:33:56 PM
LastWriteTime  : 7/12/2022 2:33:56 PM
LastAccessTime : 7/12/2022 2:33:56 PM
Mode           : d----l
LinkType       : Junction
Target         : {Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\}

PS C:\var> Get-Volume | Format-List


ObjectId             : {1}\\xxxxxxxx\root/Microsoft/Windows/Storage/Providers_v2\WSP_Volume.ObjectId="{ce6930b6-
                       fed1-11ec-838a-806e6f6e6963}:VO:\\?\Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\"
PassThroughClass     :
PassThroughIds       :
PassThroughNamespace :
PassThroughServer    :
UniqueId             : \\?\Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\
AllocationUnitSize   : 4096
DedupMode            : NotAvailable
DriveLetter          : 
DriveType            : Fixed
FileSystem           : NTFS
FileSystemLabel      : applications
FileSystemType       : NTFS
HealthStatus         : Healthy
OperationalStatus    : OK
Path                 : \\?\Volume{1566c3e6-85da-48d5-aba4-98a3a99f32b0}\
Size                 : 32210153472
SizeRemaining        : 32137633792
PSComputerName       :

As mentioned above the volume ID is probably related to GPT format, but the overall idea should be the same with MBR.
The target should just contain another kind of identifier.

For now as workaround I added a letter to my volume to let docker service start but it also mess up the automations because in the case of docker the "configured root path" (in deamon.json) is different than the output in docker info --format "{{ .DockerRootDir }}" which is the result after EvalSymlinks, pointing on D:

[gillg]

Hello,

Any chance to move forward on this problem ?
It's really a conceptual bug in golang for windows and it cause issues in multiple golang softwares.

As exemples I have :

• docker if the data files are configured inside a junction (eg: C:/var/lib is a junction and docker configured to strore data in C:/var/lib/docker/)

time="2023-03-29T14:32:23.347218600Z" level=info msg="Starting up"
Unable to get the full path to root (C:/var/lib/docker/): failed to canonicalise path for C:/var/lib/docker/: EvalSymlinks: too many links

• opentelemetry collector when parsing log files in a junction (eg: configured to read C:/var/log/iis/*/*.log where C:/var/log is a junction. Other test case, read C:/var/lib/docker-sites/*/Logs/IIS/W3SVC1/*.log where C:/var/lib/docker-sites/*/Logs is a symlink to a folder in a junction => C:\var\log\docker-sites\xxx, C:/var/log is the junction)

2023-03-29T14:54:53.977Z        error   fileconsumer/reader_factory.go:117      resolve attributes: %!w(*errors.errorString=&{EvalSymlinks: too many links})    {"kind": "receiver", "name": "filelog/iis", "pipeline": "logs", "component": "fileconsumer"}

The solutions seems around :
https://github.com/golang/go/blob/master/src/path/filepath/symlink_windows.go evalSymlinks() function
and

go/src/path/filepath/path_windows.go

Line 61 in e09bbae

func volumeNameLen(path string) int {

and

go/src/cmd/vendor/golang.org/x/sys/windows/syscall_windows.go

Line 1457 in e09bbae

func Readlink(path string, buf []byte) (n int, err error) {

But the logic in this functions seems kind of blur to me.

[ericwj]

Or Docker should stop using EvalSymlinks. And crappy workarounds like isUNC. And support Windows properly in general. https://github.com/search?q=repo%3Adocker%2Fcli+evalsymlinks&type=code

This PR has been open for ages for one docker/cli#2649

[gopherbot]

Change https://go.dev/cl/516695 mentions this issue: filepath: EvalSymlinks to not follow mount points on Windows

[gopherbot]

Change https://go.dev/cl/536655 mentions this issue: os,path/filepath: treat all non-symlink reparse points as irregular files

[gopherbot]

Change https://go.dev/cl/565136 mentions this issue: os: don't treat mount points as symbolic links