Installing some plugins broken on windows

[ahmetb]

(had to use https://storage.googleapis.com/kubernetes-release/release/v1.17.0-alpha.0/bin/windows/amd64/kubectl.exe to be able to execute plugins)

It looks like plugins can't be installed on windows. It seems like while moving the files, there's an open file handle.

c:\> kubectl krew install get-all
Updated the local copy of plugin index.
Installing plugin: get-all
W0819 16:19:15.140637    5160 install.go:132] failed to install plugin "get-all": failed to dowload and move during installation: failed to move files: failed moving files: could not rename file from "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all\\ketall-windows-amd64" to "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move160330531\\ketall-windows-amd64.exe": rename C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all\ketall-windows-amd64 C:\Users\alp\AppData\Local\Temp\krew-temp-move160330531\ketall-windows-amd64.exe: The process cannot access the file because it is being used by another process.
F0819 16:19:15.141640    5160 root.go:52] failed to install some plugins: [get-all]
exit status 255

full output

I0819 16:35:19.490880    6512 scanner.go:74] Reading plugin "get-all"
I0819 16:35:19.491899    6512 install.go:119] Will install plugin: get-all
Installing plugin: get-all
I0819 16:35:19.491899    6512 install.go:74] Looking for installed versions
I0819 16:35:19.491899    6512 util.go:79] Searching for installed versions of get-all in "C:\\Users\\alp\\.krew\\bin"
I0819 16:35:19.492898    6512 install.go:83] Finding download target for plugin get-all
I0819 16:35:19.492898    6512 util.go:38] Using os=windows arch=amd64
I0819 16:35:19.492898    6512 util.go:61] Matching platform for labels(arch=amd64,os=windows)
I0819 16:35:19.492898    6512 util.go:68] Found matching platform with index (2)
I0819 16:35:19.492898    6512 util.go:131] Matching plugin version is e0313c356d280ad12a858b8de539aa1e5cb19908589483bd153896ace08e679f
I0819 16:35:19.492898    6512 install.go:46] Creating download dir "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all"
I0819 16:35:19.492898    6512 install.go:61] Getting sha256 (e0313c356d280ad12a858b8de539aa1e5cb19908589483bd153896ace08e679f) signed version
I0819 16:35:19.492898    6512 downloader.go:36] Fetching "https://github.com/corneliusweig/ketall/releases/download/v1.2.0/bundle.tar.gz"
I0819 16:35:20.227011    6512 downloader.go:43] Reading download data into memory
I0819 16:35:28.668521    6512 downloader.go:48] Read 32492427 bytes of download data into memory
I0819 16:35:28.668521    6512 downloader.go:177] detected .tar.gz file
I0819 16:35:28.668521    6512 downloader.go:93] tar: extracting to "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all"
I0819 16:35:28.669537    6512 downloader.go:110] tar: processing "ketall-linux-amd64" (type=48, mode=-rwxr-xr-x)
I0819 16:35:28.669537    6512 downloader.go:125] tar: ensuring parent dirs exist for regular file, dir=C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all
I0819 16:35:28.954431    6512 downloader.go:139] tar: processed "ketall-linux-amd64"
I0819 16:35:28.954828    6512 downloader.go:110] tar: processing "ketall-windows-amd64" (type=48, mode=-rwxr-xr-x)
I0819 16:35:28.955833    6512 downloader.go:125] tar: ensuring parent dirs exist for regular file, dir=C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all
I0819 16:35:29.241547    6512 downloader.go:139] tar: processed "ketall-windows-amd64"
I0819 16:35:29.241547    6512 downloader.go:110] tar: processing "ketall-darwin-amd64" (type=48, mode=-rwxr-xr-x)
I0819 16:35:29.243532    6512 downloader.go:125] tar: ensuring parent dirs exist for regular file, dir=C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all
I0819 16:35:29.545013    6512 downloader.go:139] tar: processed "ketall-darwin-amd64"
I0819 16:35:29.545013    6512 downloader.go:141] tar extraction to C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all complete
I0819 16:35:29.546163    6512 move.go:155] Creating plugin dir "C:\\Users\\alp\\.krew\\store\\get-all"
I0819 16:35:29.548205    6512 move.go:161] Creating temp plugin move operations dir "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move050602455"
I0819 16:35:29.548205    6512 move.go:125] Finding move targets from "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all" to "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move050602455" with file operation=index.FileOperation{From:"./ketall-windows-amd64", To:"ketall-windows-amd64.exe"}
I0819 16:35:29.549210    6512 move.go:44] Trying to move single file directly from="C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all" to="C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move050602455" with file operation=index.FileOperation{From:"./ketall-windows-amd64", To:"ketall-windows-amd64.exe"}
I0819 16:35:29.572998    6512 move.go:48] Detected single move from file operation=index.FileOperation{From:"./ketall-windows-amd64", To:"ketall-windows-amd64.exe"}
I0819 16:35:29.572998    6512 move.go:132] Move file from "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all\\ketall-windows-amd64" to "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move050602455\\ketall-windows-amd64.exe"
W0819 16:35:29.588991    6512 install.go:132] failed to install plugin "get-all": failed to dowload and move during installation: failed to move files: failed moving files: could not rename file from "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all\\ketall-windows-amd64" to "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move050602455\\ketall-windows-amd64.exe": rename C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all\ketall-windows-amd64 C:\Users\alp\AppData\Local\Temp\krew-temp-move050602455\ketall-windows-amd64.exe: The process cannot access the file because it is being used by another process.
F0819 16:35:29.589993    6512 root.go:50] failed to install some plugins: [get-all]
github.com/GoogleContainerTools/krew/cmd/krew/cmd.init.1.func1
        /go/src/github.com/GoogleContainerTools/krew/cmd/krew/cmd/install.go:142
github.com/GoogleContainerTools/krew/vendor/github.com/spf13/cobra.(*Command).execute
        /go/src/github.com/GoogleContainerTools/krew/vendor/github.com/spf13/cobra/command.go:762
github.com/GoogleContainerTools/krew/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /go/src/github.com/GoogleContainerTools/krew/vendor/github.com/spf13/cobra/command.go:852
github.com/GoogleContainerTools/krew/vendor/github.com/spf13/cobra.(*Command).Execute
        /go/src/github.com/GoogleContainerTools/krew/vendor/github.com/spf13/cobra/command.go:800
github.com/GoogleContainerTools/krew/cmd/krew/cmd.Execute
        /go/src/github.com/GoogleContainerTools/krew/cmd/krew/cmd/root.go:48
main.main
        /go/src/github.com/GoogleContainerTools/krew/cmd/krew/main.go:23
runtime.main
        /usr/local/go/src/runtime/proc.go:201
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1333
exit status 255

however this is not always reproducible, on another plugin, it fails because I am running on a cmd.exe without Administrator privileges:

c:\>kubectl krew install who-can
Updated the local copy of plugin index.
Installing plugin: who-can
W0819 16:23:05.164281    5724 install.go:132] failed to install plugin "who-can": failed to create a symlink form "C:\\Users\\alp\\.krew\\bin" to "C:\\Users\\alp\\.krew\\bin\\kubectl-who_can.exe": symlink C:\Users\alp\.krew\store\who-can\29d8c8c85db54a8a748d7ca1f3d523c5dac123cadeb80907b7f66a9cf55040d6\kubectl-who-can.exe C:\Users\alp\.krew\bin\kubectl-who_can.exe: A required privilege is not held by the client.
F0819 16:23:05.165294    5724 root.go:52] failed to install some plugins: [who-can]
exit status 255

However if I re-run cmd.exe as Administrator, this who-can installs, yet get-all fails with the same error as before indicating there's an open descriptor/handle on the file.

version info:
GitTag v0.2.1
GitCommit cd2c6e1

/area platform/windows
/priority P1
/kind bug

[ahmetb]

I observed this error on the same plugin while doing receipts migration (on the exact same plugin) on krew built from 739c312 (untagged).

c:\>kubectl krew system receipts-upgrade
Updated the local copy of plugin index.
I0819 16:50:40.079822    5476 migration.go:83] These plugins will be reinstalled:  [get-all who-can]
I0819 16:50:40.085819    5476 migration.go:158] Uninstalling get-all
I0819 16:50:40.086820    5476 migration.go:173] Re-installing get-all
I0819 16:50:46.704672    5476 migration.go:177] Updated the local copy of plugin index.
WARNING: Detected stdin, but discarding it because of --manifest or args
Installing plugin: get-all
W0819 16:50:46.681515    2524 install.go:126] failed to install plugin "get-all": install failed: failed while moving files to the installation directory: failed to move files: failed moving files: could not rename file from "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-downloads\\get-all\\ketall-windows-amd64" to "C:\\Users\\alp\\AppData\\Local\\Temp\\krew-temp-move083425595\\ketall-windows-amd64.exe": rename C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all\ketall-windows-amd64 C:\Users\alp\AppData\Local\Temp\krew-temp-move083425595\ketall-windows-amd64.exe: The process cannot access the file because it is being used by another process.
F0819 16:50:46.681515    2524 root.go:65] failed to install some plugins: [get-all]
exit status 255
I0819 16:50:46.704672    5476 migration.go:98] Reinstalling get-all failed
I0819 16:50:46.706598    5476 migration.go:158] Uninstalling who-can
I0819 16:50:46.711598    5476 migration.go:173] Re-installing who-can

[ahmetb]

So I upgraded to 739c312 (supposedly v0.3) and it reproduces there, too.

I am suspecting of two things:

• our rename (move) functionality opens a file handle (file descriptor) and doesn't close it before attempting to moving it
• our download path is deterministic and there's no random components (e.g. C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all\ketall-windows-amd64) and something like the virus scanner (Windows Defender idk) is trying to read it at the same, just after we downloaded it.

One thing to note is that the downloaded/unarchived stuff at C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all doesn't get deleted when install command fails (missing defer?)

[ahmetb]

More investigation...

• our download path is deterministic and there's no random components (e.g. C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all\ketall-windows-amd64) and something like the virus scanner (Windows Defender idk) is trying to read it at the same, just after we downloaded it.

This is not the issue. I removed DownloadPath() entirely from the system and started generating a download dir on each install... Same issue.

One thing to note is that the downloaded/unarchived stuff at C:\Users\alp\AppData\Local\Temp\krew-downloads\get-all doesn't get deleted when install command fails (missing defer?)

We actually have the defer _ = os.RemoveAll(...) but it's failing with the same error.

It looks like whatever it is, it's also opening something in this dir for read, so the dir cannot be deleted. I've tried procexp.exe, still can't find what is holding onto this dir/file. Best bet is: krew itself, but where.

[ahmetb]

More debugging. Installed procmon to see what's happening on ketall-windows-amd64 file.

It appears like, indeed, Windows Defender (MsMpEng.exe) is accessing this plugin binary concurrently, though I'm not sure if it's surely restricting the access by krew because of that.

[corneliusweig]

Wow, that's already a great analysis. Could you verify your Windows Defender hypothesis by turning it off for an upgrade/install? I plan to get a windows VM to also help with debugging, but I can only start next week.

[ahmetb]

I spent 20 mins trying to figure out how to disable windows defender and gave up.
The instructions aren’t matching my win10home version sadly.

[ahmetb]

I'm nearly confident that this issue can not be easily fixed, and is because of Windows Defender (why am I confident: I "assumed" it was something like windows defender, I monitored it, and there it was).

If you look at the FILE LOCKED WITH WRITERS line, I think what’s happening here is that Windows Defender is locking the .exe files for any future writes (because, well, people aren't supposed to edit these files).

I see two options:

1. we should modify our tar/zip extraction code to accept []FileOperation and figure out the files that should be extracted while we're traversing the files inside the archive, then put them at their final destination. This would eliminate (1) all move.go, and (2) a staging directory, which is sort of nice
2. investigate what os.Rename (MoveFileW in win32) does and figure out if we can make it work with this windows defender restriction somehow.

[ahmetb]

I've managed to disable Windows Defender and it appears like it's not entirely Windows Defender. Now the only process accessing this ketall-windows-amd64 is krew's PID, and it's still happening.

The procmon output on this file is almost the same. So it seems like there's an actual issue here with how we do things.

[ahmetb]

Found the issue.
It turns out we actually weren't closing the files as we extract them from the tarball.