[Bug]: Windows CI builds failing to find docker (Update to Run Windows inside docker containers)

[mch2]

Describe the bug

Windows CI builds are failing, example: https://build.ci.opensearch.org/job/gradle-check/14914/console

+ docker logout
C:/Users/Administrator/jenkins/workspace/gradle-check@tmp/durable-392d4e2e/script.sh: line 1: docker: command not found
[Pipeline] }
[Pipeline] // script
Error when executing always post condition:
hudson.AbortException: script returned exit code 127
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:664)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:610)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:554)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

To reproduce

N/A

Expected behavior

Builds should pass and docker tests should run.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[jordarlu]

Hi, @mch2 , could you let me know how do yor trigger the gradle_check in this case? if you had a PR that triggered it, can you send me the PR link?
thanks,

CC @peterzhuamazon

[peterzhuamazon]

I will take care of this as I have talked to @mch2 offline.
Thanks.

[peterzhuamazon]

Able to get docker running on Windows with hyperv.

Administrator@<> MINGW64 ~
$ docker version
Client:
 Version:           23.0.6
 API version:       1.42
 Go version:        go1.19.9
 Git commit:        ef23cbc
 Built:             Fri May  5 21:18:35 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.6
  API version:      1.42 (minimum version 1.24)
  Go version:       go1.19.9
  Git commit:       9dbdbd4
  Built:            Fri May  5 21:17:32 2023
  OS/Arch:          windows/amd64
  Experimental:     false



Administrator@<> MINGW64 ~
$  docker pull mcr.microsoft.com/windows/nanoserver:ltsc2019
ltsc2019: Pulling from windows/nanoserver
aaaa081173ae: Pulling fs layer
aaaa081173ae: Verifying Checksum
aaaa081173ae: Download complete
aaaa081173ae: Pull complete
Digest: sha256:fb78bd84ac937f6b1453e19015ccce41636bbeca5fe5bc6dc5c7d55adb4a2bc5
Status: Downloaded newer image for mcr.microsoft.com/windows/nanoserver:ltsc2019
mcr.microsoft.com/windows/nanoserver:ltsc2019

Needs @mch2 to confirm what are the exact images that windows docker is running with.

On windows, if you use hyperv then windows host can only run windows container.
If we need windows host to run linux container, we need to enable wsl2 later on and might have issues.

Please let me know about this.
Thanks.

[peterzhuamazon]

Also, this can be a good start into these two issues to bring windows integTest with docker host and containers, even building the artifacts on windows docker containers.

• [META] RPM, Deb and Windows integration testing lacks jenkins automation opensearch-build#2994
• [Bug]: Fix OS integration tests for Windows platform opensearch-build#3196

Here is a chart showcasing the comparison between different offers of containers on Windows:

Here's a chart comparing some of the key differences between Windows Server with Server Core installation and Windows Nano Server:

Feature	Windows Server with Server Core	Windows Nano Server
Installation size	Larger (several GBs)	Smaller (a few hundred MBs)
Attack surface	Larger	Smaller
Support for GUI	Yes (minimal)	No
Support for 32-bit applications	Yes	No
Support for Windows Services	Yes	Limited
Support for .NET Framework	Yes	Limited
Support for Containers	Yes	Yes
Licensing	Standard, Datacenter	Standard, Datacenter
Available editions	All Windows Server editions	Standard and Datacenter only

Will try to see if we can bring nanoserver in place to make Windows light wight in build, test, and check.

Thanks.

[peterzhuamazon]

I eventually get the docker container running the nanoserver on Windows:

PS C:\Users\Administrator> docker images
REPOSITORY                             TAG        IMAGE ID       CREATED       SIZE
mcr.microsoft.com/windows/nanoserver   ltsc2019   82ef3885248c   2 weeks ago   252MB

PS C:\Users\Administrator> docker run 82ef3885248c
Microsoft Windows [Version 10.0.17763.4645]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>

PS C:\Users\Administrator> docker ps -a
CONTAINER ID   IMAGE          COMMAND                    CREATED              STATUS                          PORTS     NAMES
4aced3bb72dd   82ef3885248c   "c:\\windows\\system32…"   About a minute ago   Exited (0) About a minute ago             blissful_liskov

PS C:\Users\Administrator> docker rm 4aced3bb72dd
4aced3bb72dd

PRs:

• [Windows Container] Add Windows 2019 ServerCore Docker Images opensearch-build#3859
• [Windows Container] Add initial windows docker builder and host AMI #334
• [Windows Container] Setup docker for windows docker images to build docker images opensearch-build#3903
• [Windows Container] Windows docker setups jenkins docker build opensearch-build#3905
• [Windows Container] Move old perf-test and ccr-perf-test to legacy and remove old gradle check runner as they are deprecated now opensearch-build#3910
• [Windows Container] Replace gzip with pigz for Windows docker extraction #337
• [Windows Container] Add unzip and scoop clear cache in Windows #338
• [Windows Container] Add #337 #338 AMI and add new customDeviceMapping config with new benchmark agent #339
• [Windows Container] Speed up windows servercore docker images opensearch-build#3939
• [Windows Container] Replace temurin with openjdk in Windows container and tweak docker build better support Windows opensearch-build#3941
• [Windows Container] Update openjdk8 to openjdk8-redhat as the former is not supported on scoop opensearch-build#3942
• [Windows Container] Update hash in python39 reg files in Windows docker opensearch-build#3943
• [Windows Container] Remove windows docker speedup command as it is only available in non-container opensearch-build#3944
• [Windows Container] Switch windows docker jdk back to temurin/openjdk mix opensearch-build#3946
• [Windows Container] Add unzip and lock pipenv/awscli versions in Windows docker images opensearch-build#3947
• [Windows Container] Use windows home dir as temp dir location opensearch-build#3951
• [Windows Container] Update AMI ID from PR Number #342 #343
• [Windows Container] Remove interactive shell so Windows AMI can start properly #344
• [Windows Container] Switch Windows Runner from Agents to Docker Container on Jenkins Distribution Build opensearch-build#3976
• [NodeJS18 Upgrade] [Windows Container] Adjust spare instance number for AL2023 and Windows Docker Host #345
• [Windows Container] Add Windows IntegTest support on Jenkins Libs opensearch-build-libraries#308
• [Windows Container] Refactor integTest java version handling logic for all distributions opensearch-build-libraries#314
• [Windows Container] Make sure WindowsZip/LinuxTar remove correct dir when uninstall opensearch-build#4023
• [Windows Container] Add Windows IntegTest support to Jenkins with Windows Docker Container opensearch-build#4027

• Updating

[peterzhuamazon]

We will be better of with the servercore option rather than the nanoserver, as the latter lack of several core components, while the servercore is just a headless version of the normal server base of Windows.

https://techcommunity.microsoft.com/t5/containers/nano-server-x-server-core-x-server-which-base-image-is-the-right/ba-p/2835785

[peterzhuamazon]

Issues in the windows docker that is currently not able to solve to make it the same as AMI:
Move-Item : Access to the path is denied.

moby/moby#38256
microsoft/Windows-Containers#147

[peterzhuamazon]

Just able to confirm that I am using --isolation=process not --isolation=hyperv.

[peterzhuamazon]

Able to resolve the move issue by just using mingw and force the mv happens by bash.exe.

bash.exe -c "mv -v 'C:\\Windows\\System32\\find.exe' 'C:\\Windows\\System32\\find_windows.exe'"

renamed 'C:\Windows\System32\find.exe' -> 'C:\Windows\System32\find_windows.exe'

[peterzhuamazon]

Seems like issue with volta on 1.1.1: volta-cli/volta#1435

Will revert to either the older 1.0.8 or 1.1.0 now.

Thanks.

[peterzhuamazon]

Able to invoke bash.exe directly in the windows container and able to run test workflow:

ContainerAdministrator@44082dfc4844 MINGW64 /c
$ whoami
ContainerAdministrator

[peterzhuamazon]

git clone now on the windows host is instant on build repo.

[peterzhuamazon]

There is a bug right now that every time when we pull the image from fresh it will always fail once on the sh stage.
I suspect we need to pre-load the image on the runner beforehand.
It will goes to success soon after in the second rerun:

ERROR: script returned exit code 127

[peterzhuamazon]

Add a docker image initialization step on Windows Docker Host to resolve above issues.

[peterzhuamazon]

Add new integTest support with Windows container now.

[peterzhuamazon]

Per opensearch-project/opensearch-build#3816 we have fixed the docker commands issues on Windows, but it only supports hyperv running windows on windows through docker.

Per discussion with @mch2 the core team needs to disable the linux container related test on Windows.

Thanks.