Support container operations on Windows runners

[tprk77]

Describe the enhancement

I would like to run a Windows Docker container on a Windows runner. But was surprised to find out that this is not supported:

Error: Container operations are only supported on Linux runners

Specifically, I am asking that:

• jobs.<job_id>.container work on Windows runners
• It allows for use of Windows containers (as opposed to Linux containers)

Code Snippet

Here's an example workflow of what I'm asking for:

jobs:
  windows-build:
    name: Windows Build
    runs-on: windows-2019
    container:
      image: mcr.microsoft.com/windows/servercore:ltsc2019
    steps:
      - run: ...

Additional information

N/A

[TingluoHuang]

@tprk77
we are not supporting windows container for the following reasons:

• Extra windows version requirements, ex regular windows 10 desktop may not work with certain network/volume features
• We can't install docker-ce on GitHub windows hosted runner VM due to license issues.
• We have to install docker-ee which is usually 6-12 months behind docker-ce on feature and bug fixes.
• Container on Linux and Windows are not feature parity.
• Windows is just not good at supporting containers.

[tprk77]

That's too bad, but it seems there is good justification for not supporting Windows container.

However, I don't see why this isn't mentioned in the documentation of jobs.<job_id>.container. It would be really easy to include a line saying that it only works on Linux runners, and would prevent people form wasting their time trying to use it like I did.

Should I open another ticket for the documentation?

[jelmansouri-legion]

Is Microsoft, improving/planning on maintaining container support or is the plan to deprecate it at some point? If the latter I understand it's not worth the time of devs and GitHub. If not, maybe even with some restrictions, it might be worth allowing some workflow to used them, as long as the non supported cases are documented.
Since the documentation wasn't explicit about the ability to use the container operations on windows, I gave it a try. It yielded the same reported error above. As a workaround, I ended up doing this:

      - name: Build
        run: >-
          docker run
          --name cargo_test
          --rm
          --workdir ${{ matrix.workdir }}
          -v ${{ github.workspace }}:${{ matrix.workdir }}
          ${{ matrix.container }}
          cargo test --no-run

[Kleptine]

Is this ever going to be on the roadmap? At the moment this bans a fairly large number of actions from ever working on Mac or Windows, which splits the ecosystem pretty dramatically for anyone trying to build cross platform software.

I understand the above licensing issues, but as Docker becomes more stable on Windows, the issues with using docker-ee seem like they would diminish. Presumably we'll get to a point where docker-ee being 6 months behind on bug fixes isn't an issue?

[neersighted]

Hi, I'm an engineer who works on the product formerly known as Docker EE (and now known as Mirantis Container Runtime). We work in the upstream Moby project on a daily basis to ensure our product is as close to upstream Moby as possible, and count several maintainers among our team.

I'd love to understand why GitHub needs MCR (Docker EE) over a from-source build of Moby. I'd also love to determine if we can do something as Mirantis to help GitHub support this for all open-source projects. If you're from GitHub, please reach out -- I'd be very happy to discuss upstream Moby and its development process/licensing, MCR's lifecycle (we usually deliver patches in a matter of weeks, but historically version numbers have not been aligned, and this may be the source of confusion), and any other questions/concerns you may have.

[thecloudtaylor]

@tprk77 we are not supporting windows container for the following reasons:

• Extra windows version requirements, ex regular windows 10 desktop may not work with certain network/volume features
• We can't install docker-ce on GitHub windows hosted runner VM due to license issues.
• We have to install docker-ee which is usually 6-12 months behind docker-ce on feature and bug fixes.
• Container on Linux and Windows are not feature parity.
• Windows is just not good at supporting containers.

@TingluoHuang - I am the group product manager for Windows containers. I disagree with a set of these assertions and very much disagree with the over decision as clearly from this discussion and many others we have had there is customer need for this support.

To the points:
#1 (Win 10 etc...) - Firstly, I'm not sure why this is even relevant to the discussion. While of course developers leverage Windows client while building applications the build pipeline and production usage of them is on Windows Server.

#2 (Docker CE) - You can use Moby (to @neersighted's point).

#3 (MCR delays) - I would be very interested to understand what features fixes, specific to Windows containers, you where delayed on? The Mirantis team has typically been very active at incorporating blocking issues into EE builds.

#4 (feature parity) - The first priority/goal of Windows containers is enabling customers a part forward for there Windows Server applications. There are definitely features on our roadmap that will enhance that further but we believe and have many customers in production that have been very successful moving apps forward. There are a whole host of Linux features/functionality that just simply don't make since in a Windows world. Just as there are Windows features that don't make since in a linux world.

#5 (windows is not good at hosting containers) - I'm just going to leave this at argumentative and unproductive.

[knocte]

we are not supporting windows container for the following reasons:

Why was this closed as completed (instead of not planned) then?

[davidkhala]

@TingluoHuang Would you consider reopen the issue based on above discussions?

[MJohnson20]

Bumping this again:

The fact that windows runners don't have feature parity with Linux runners from a company owned by Microsoft is baffling.

[nebuk89]

Hey! (I am one of the PMs on Actions 😄 )

I am happy to get this re-opened as something we track, I will be up front and say it's not something we are going to get to in the next 3 months (we have a plan, and it's not part of it right now). But I am very happy for this to be an open item on the backlog that we review. This would be cool to support, we just need to focus on what we ship 😁 (right now getting ARM ready and for Windows as well as Linux, is a big focus for example).

Windows containers have come a long way. I mean Docker ee is not a Docker thing anymore 👀 so clearly the comments on here don't represent current state.

I will sort getting this re-opened and we can keep the enhancement label on this as is.

[shawn-hurley]

I would like to add that if you support docker on windows, supporting other container tools would be good as well Podman being the one I personally care about :)

[seantleonard]

@nebuk89 were you still planning on reopening this issue?

[davidmwhynot]

@nebuk89 any plans to re-open this issue? it's been 3 months since your last update. Is this going to be prioritized?

[Danielku15]

@nebuk89 Would it be an option to add an experimental option to disable the currently existing checks?

We understand that its not that straight forward to support things officially and ensure the GitHub.com infrastructure can officially support this. With such an option, some early adopters could get an early hands on via self-hosted runners and share feedback. We'd love to have this available for our GHES environment.

[NickCraver]

@nebuk89 Can we reopen this please?

We've been wanting to move to GitHub Actions for many years now, but we can't for example run a Redis (Linux) container on a Windows build agent, which we have to be able to do to test the StackExchange.Redis .NET client on both .NET 6/8 and .NET Framework. As long as we can't run Linux test servers and hit them from a Windows build, there's a whole set of client use cases we can't reasonably test. This applies to Dapper, MiniProfiler, and some other projects we maintain as well.

[nebuk89]

Yes sorry, let me see if I can get my entitlements changed so I can re-open things :)

[EricDales]

Hello,

an option that wouldn't break / change much current design is to make ACTIONS_RUNNER_CONTAINER_HOOKS variable mandatory for running job containers on Windows. People would have to bring their hook so that wouldn't put the load on runner maintainers team, but at least, that gives the capability to use containers on Windows.

[Danielku15]

@EricDales Why would this be needed? As far I can see the Default IDockerCommandManager implementation should be working fine on Windows. There is a #if OS_WINDOWS on the network part to use a nat driver but beside that I think there is nothing to special when it comes to talking to the docker CLI.

[EricDales]

Because, obviously, GitHub is not confident with container implementation on Windows and all the discussion above has not changed their mind since job containers are still not available on Windows. So, you can continue to ask blindly the same thing or try to find a trade-off that would allow a first implementation.

[Danielku15]

@EricDales You are referring to a years old (2021) opinion of primarily one individual. There wasn't much involvement from GitHub Employees until lately. If you look into the source code you will notice that GitHub has already done the most of the heavy lifting to support Windows containers (search for OS_WINDOWS):

• https://github.com/actions/runner/blob/main/src/Runner.Worker/Container/DockerCommandManager.cs
• https://github.com/actions/runner/blob/main/src/Runner.Worker/ContainerOperationProvider.cs

Requiring ACTIONS_RUNNER_CONTAINER_HOOKS would not really bring us a good step forward to get a first experimental version. IMO: Replacing the current checks with a feature flag is the better path forward evaluating the compatibility and investment to get things production ready.

enable-win-container.diff

diff --git a/src/Runner.Common/Constants.cs b/src/Runner.Common/Constants.cs
index 383ec7a..5de0e0d 100644
--- a/src/Runner.Common/Constants.cs
+++ b/src/Runner.Common/Constants.cs
@@ -163,6 +163,7 @@ namespace GitHub.Runner.Common
                 public static readonly string LogTemplateErrorsAsDebugMessages = "DistributedTask.LogTemplateErrorsAsDebugMessages";
                 public static readonly string UseContainerPathForTemplate = "DistributedTask.UseContainerPathForTemplate";
                 public static readonly string AllowRunnerContainerHooks = "DistributedTask.AllowRunnerContainerHooks";
+                public static readonly string AllowContainerOperationsOnWindows = "DistributedTask.AllowContainerOperationsOnWindows";
             }
 
             public static readonly string InternalTelemetryIssueDataKey = "_internal_telemetry";
diff --git a/src/Runner.Worker/Container/DockerCommandManager.cs b/src/Runner.Worker/Container/DockerCommandManager.cs
index 41b914a..de7293d 100644
--- a/src/Runner.Worker/Container/DockerCommandManager.cs
+++ b/src/Runner.Worker/Container/DockerCommandManager.cs
@@ -9,6 +9,7 @@ using System.Threading.Channels;
 using System.Threading.Tasks;
 using GitHub.Runner.Common;
 using GitHub.Runner.Sdk;
+using GitHub.Runner.Worker;
 
 namespace GitHub.Runner.Worker.Container
 {
@@ -336,10 +337,7 @@ namespace GitHub.Runner.Worker.Container
                 }
             };
 
-            if (!Constants.Runner.Platform.Equals(Constants.OSPlatform.Linux))
-            {
-                throw new NotSupportedException("Container operations are only supported on Linux runners");
-            }
+            context.Variables.EnsureContainerOperationsFeature();
             return await processInvoker.ExecuteAsync(
                             workingDirectory: HostContext.GetDirectory(WellKnownDirectory.Work),
                             fileName: DockerPath,
diff --git a/src/Runner.Worker/ContainerOperationProvider.cs b/src/Runner.Worker/ContainerOperationProvider.cs
index c5cccb7..a2b51d9 100644
--- a/src/Runner.Worker/ContainerOperationProvider.cs
+++ b/src/Runner.Worker/ContainerOperationProvider.cs
@@ -46,10 +46,7 @@ namespace GitHub.Runner.Worker
         public async Task StartContainersAsync(IExecutionContext executionContext, object data)
         {
             Trace.Entering();
-            if (!Constants.Runner.Platform.Equals(Constants.OSPlatform.Linux))
-            {
-                throw new NotSupportedException("Container operations are only supported on Linux runners");
-            }
+            context.Variables.EnsureContainerOperationsFeature();
             ArgUtil.NotNull(executionContext, nameof(executionContext));
             List<ContainerInfo> containers = data as List<ContainerInfo>;
             ArgUtil.NotNull(containers, nameof(containers));
diff --git a/src/Runner.Worker/FeatureManager.cs b/src/Runner.Worker/FeatureManager.cs
index 98f49e8..713186e 100644
--- a/src/Runner.Worker/FeatureManager.cs
+++ b/src/Runner.Worker/FeatureManager.cs
@@ -11,5 +11,25 @@ namespace GitHub.Runner.Worker
             var isContainerHooksPathSet = !string.IsNullOrEmpty(Environment.GetEnvironmentVariable(Constants.Hooks.ContainerHooksPath));
             return isContainerHookFeatureFlagSet && isContainerHooksPathSet;
         }
+
+        public static void EnsureContainerOperationsFeature(Variables variables)
+        {
+            if (Constants.Runner.Platform.Equals(Constants.OSPlatform.Linux))
+            {
+                return;
+            }
+            else if(Constants.Runner.Platform.Equals(Constants.OSPlatform.Windows))
+            {
+                var isContainerOnWindowsFeatureFlagSet = variables?.GetBoolean(Constants.Runner.Features.AllowContainerOperationsOnWindows) ?? false;
+                if (!isContainerHookFeatureFlagSet)
+                {
+                    throw new NotSupportedException($"Container operations are not supported on Windows runners (experimental support can be enabled via '{Constants.Runner.Features.AllowContainerOnWindows}')");
+                }
+            }
+            else
+            {
+                throw new NotSupportedException($"Container operations are not supported on macOS runners");
+            }
+        }
     }
 }

[Danielku15]

Update: To get some more progress on the topic I started to look into getting a local runner build working. And after a few code changes things are running:

build.yml

name: Hello World

on:
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: self-hosted
    container: mcr.microsoft.com/windows/servercore:ltsc2022
    steps:
      - run: echo "Hello, World!"
      - run: systeminfo

What I can share so far is:

• These are the code changes which are fairly simple: main...Danielku15:actions-runner:feature/docker-windows
  • Adding a feature flag
  • Disabling the check for Windows Server versions (maybe temporarily, it allows me to use docker desktop on my workstation).
  • Only mounting folders if the source exists (in my case the _actions folder was missing on an empty workflow)
  • Platform specific path for the github runner files. (C:\github\** and /github/**)
• There is a need to discuss "Windows Containers on Windows" (easy) vs "Linux Containers on Windows via WSL" (not so easy) and how to support it.
  • Various #if OS_WINDOWS will need to become runtime settings. The platform should be pulled from the docker daemon to decide what it supports.
  • For GitHub.com runners I'd say a new workflow syntax is needed to specify the desired platform. And the runner has to reconfigure itself.
    • WSLv2 has to be there (windows-latest runners only have WSLv1 by default)
    • A Linux VM with Docker Engine has to be created in WSL
    • The Docker CLI commands have to be forwarded to the daemon in the Linux VM (DOCKER_HOST)
    • Handling volume mounts and other things have to be tunneled through correctly.
    • Maybe also GitHub decides not to support this on the cloud runners. But IMO if there is a working Docker with Linux container configured, some basics should work.

So in short: I think we should differenciate clearly between Windows containers (easy) on Windows runners, and Linux Containers on Windows Runners (harder) and manage the expectations of the people accordingly.

[bebuch]

We need Windows Containers on a Windows Runner on Windows 11. As we want to switch to GitHub soon, we hope this will be available soon. The progress here is very encouraging!

For Linux Containers we use a separate computer with Linux.

[danpetitt]

I just want to be able to run a Linux container (SQL Server) so I can run my integration tests as part of PR build/test check, but none of this is easy or possible which is so annoying having spent 2 weeks setting up integration tests using testcontainers.

[Danielku15]

My personal option:
I think it should be technically achievable that the runner software itself can:

• Run Windows Containers on Windows assuming the container runtime is already configured to work with Windows Containers.
• Run Linux Containers on Windows assuming the container runtime is already configured to work with Linux Containers.

That means for self hosted runners you should be able to get things working in the way you need your environment.

But for GitHub.com Runners I don't see it (in mid-term) realistic that they will be able to provide Linux Containers on Windows due to the Container Runtime Limitations.

Linux containers on windows are not officially supported by "Windows Containers" feature in the Windows Server OS. They are a feature of Docker Desktop taking care of the WSL VM Administration and taking care of any aspects like volume mounts, port forwarding etc. I don't think with a OS built-in foundation foundation for this setup will be officially supported.

Would be great to get the opionion of some GitHub Folks and project maintainers on that topic. I'd be happy to invest some time to contribute maybe some parts to the runner application to get this rolling.

[ChristopherHX]

Technically yes, however this is not trivial to do.

As someone who did implement this back in 2021/2022 some technical aspects from my still alive runner fork for running actions locally that I still remember.

assuming the container runtime is already configured to work with Windows Containers.

I do the auto detection here: https://github.com/ChristopherHX/runner.server/blob/28253d8f22363cdf8edecfab73cb5868af91d855/src/Runner.Worker/ContainerOperationProvider.cs#L254-L260 and adjust a lot of things based on this at runtime.

Like choosing the nodejs runtime (I have a on demand linux node downloader), path separator and much more.

Run Linux Containers on Windows assuming the container runtime is already configured to work with Linux Containers.

This runner without the no support bomb performs badly if the runner is a native windows app that uses docker desktop VM or wsl, because of excessive use of bind mounts. (You didn't notice because you couldn't run setup-go?, I can and you as well with my fork)

Downloading this to the runner tool cache like golang can take a lot of more time, those mounts used to have very small iops.

Additionally you would need to workaround that windows doesn't support env variables keys with different capitalization.

For windows containers are weird behaviors out there, my fork of this runner has a years old windows container backend that is tested on top of GitHub Hosted Runners .

e.g. -e PATH=some/path has been ignored by windows runners (test $GITHUB_PATH).

I have worked around this by using the cmd shell to execute the choosen shell https://github.com/ChristopherHX/runner.server/blob/16d82cee21e9d634fa7cb62597dbdf7318da76fa/src/Runner.Worker/Handlers/StepHost.cs#L249-L259

...,...

[SS1823]

I have tested windows containers and the basics worked in #1801

[chadmyers]

I think #1402 is related to this, if not a dupe

[joshelson]

@SS1823 but are you testing Windows containers on self hosted runners or on the Github hosted runners? Do we have any more options here at this point other than self-hosting to build Windows containers?

[ChristopherHX]

Do we have any more options here at this point other than self-hosting to build Windows containers?

You can host your self-hosted runner fork in a windows-latest job, the GitHub Hosted Runner has a preconfigured docker.exe for windows container.

Here an public action doing this: https://github.com/ChristopherHX/runner.server/actions/runs/11262424850/job/31318159792#step:9:42

I'm doing this for years and don't have problems other than random long pull times of some windows images

One step further would be to register the self-hosted runner inside the GitHub Hosted Runner via a PAT secret or similar and then queue one to the self-hosted runner label and one to run that runner in parallel on a windows-latest machine. This enshures the job looks better in the ui.