Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images #17700

csabatuz-chess · 2023-11-30T14:40:42Z

What Happened?

I was planning using Minikube with docker driver, on an Apple M1 chip.

My Docker Engine, when freshly started supports the following emulators:

% docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}
% minikube start
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver. Other choices: qemu2, virtualbox, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
% docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.10"
  ]
}

Minikube messes up the binfmt config, so from that point on, it's only possible to run images of the same architecture as minikube itself (in my case linux/arm64).

This invasive behaviour is a showstopper for me to use Minikube, despite it being lovely and I'd really like to switch my K3S based stack to Minikube.

My attempts to find a workaround, or try to restore the Binfmt settings manually failed, so reaching out with an issue.

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Docker

The text was updated successfully, but these errors were encountered:

csabatuz-chess · 2023-11-30T14:46:07Z

Correction
My issue might be misplaced here

The problem is happening with OrbStack, but cannot reproduce with Docker Desktop. So the issue might be specific to Minikube on Orbstack with docker driver.

phbelitz · 2023-11-30T16:30:59Z

If I understand this correctly, then this also messes up the WSL interoperability -> anyone how uses WSL can normally run windows executables inside their WSL (e.g. vscode), but once you try to start minikube, this is no longer possible. Then this fix needs to be applied manually.

afbjorklund · 2023-11-30T16:37:26Z

I don't think the minikube container drivers do anything with binfmt_misc, do they? Should only be the VM that sets it up, and then not for the host...

cotej · 2023-12-05T17:47:30Z

Running into a similar issue here. My setup is on a Mac M2 using lima + vz + rosetta for the Docker VM.

With minikube 1.32.0 (with docker driver), once I start minikube it somehow breaks rosetta in the lima VM.
Same configuration works fine on minikube 1.31.2.

afbjorklund · 2023-12-05T22:19:37Z

Seems like it is our friend systemd, that is at it again:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1999275

So likely this started to happen after upgrading Ubuntu...

Python 3.10 now has a /usr/lib/binfmt.d/python3.10.conf

-ARG UBUNTU_JAMMY_IMAGE="ubuntu:jammy-20230624"
+ARG UBUNTU_JAMMY_IMAGE="ubuntu:jammy-20231004"

python3.10 (3.10.8-2) "Please add support for systemd-binfmt"

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1012377

RUN rm -f /usr/lib/binfmt.d/python3.10.conf

afbjorklund · 2023-12-05T22:41:36Z

cc @spowelljr @medyagh

fbuetler · 2023-12-06T11:35:13Z

I can confirm that the same configuration works with minikube v1.31.2 and fails with v1.32.0 (we are also using orbstack):

v1.31.2:

~
❯ minikube version
minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

~
❮ minikube start
😄  minikube v1.31.2 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
🐳  Preparing Kubernetes v1.27.4 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

v1.32.0

~
❯ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

~
❯ minikube start
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.10"
  ]
}

afbjorklund · 2023-12-06T13:04:50Z

Unfortunately the base image version is not shown by default.

🚜 Pulling base image ...
🚜 Pulling base image ...

But the troublesome file has been added in the later version of it.

 const (
        // Version is the current version of kic
-       Version = "v0.0.40"
+       Version = "v0.0.42"

spowelljr · 2023-12-06T16:01:52Z

Curious, is it fixed in #17719 @fbuetler?

Without a patch release it seems that that current options are to use v1.31.2 or you should be able to specify to use v1.31.2's version of the base image during start with v1.32.0, I don't think there were any changes that would make the two incompatible.

minikube start --base-image gcr.io/k8s-minikube/kicbase:v0.0.40

fbuetler · 2023-12-13T11:50:58Z

Curious, is it fixed in #17719

@spowelljr yes, the problem still persists.

❯ ./minikube-darwin-arm64 version
minikube version: v1.32.0
commit: e08a2828f2be3e524baaf41342316dad88935561

❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

~
❯ ./minikube-darwin-arm64 start
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.28.4 preload ...
    > preloaded-images-k8s-v18-v1...:  341.36 MiB / 341.36 MiB  100.00% 21.02 M
    > gcr.io/k8s-minikube/kicbase...:  410.64 MiB / 410.64 MiB  100.00% 15.74 M
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
🐳  Preparing Kubernetes v1.28.4 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.10"
  ]
}

fbuetler · 2023-12-13T12:02:14Z

minikube start --base-image gcr.io/k8s-minikube/kicbase:v0.0.40

@spowelljr the suggested workaround works (with a warning tough)

~
❯ /opt/homebrew/Cellar/minikube/1.32.0/bin/minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

~
❯ /opt/homebrew/Cellar/minikube/1.32.0/bin/minikube start --base-image gcr.io/k8s-minikube/kicbase:v0.0.40
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
❗  Image was not built for the current minikube version. To resolve this you can delete and recreate your minikube cluster using the latest images. Expected minikube version: v1.31.0 -> Actual minikube version: v1.32.0
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

sugymt · 2023-12-17T08:35:53Z

Fix by wsl-binfmt. microsoft/WSL#8843 (comment)

minikube start; sudo systemctl start wsl-binfmt

Log:

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d
$ curl.exe -s -I https://www.google.com/ | head -n 1
HTTP/1.1 200 OK
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/WSLInterop:enabled
/proc/sys/fs/binfmt_misc/WSLInterop:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop:flags: PF
/proc/sys/fs/binfmt_misc/WSLInterop:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop:magic 4d5a
/proc/sys/fs/binfmt_misc/WSLInterop-late:enabled
/proc/sys/fs/binfmt_misc/WSLInterop-late:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop-late:flags: P
/proc/sys/fs/binfmt_misc/WSLInterop-late:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop-late:magic 4d5a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled
$ minikube start >/dev/null
    > gcr.io/k8s-minikube/kicbase...:  453.89 MiB / 453.90 MiB  100.00% 19.09 M
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/python3.10:enabled
/proc/sys/fs/binfmt_misc/python3.10:interpreter /usr/bin/python3.10
/proc/sys/fs/binfmt_misc/python3.10:flags:
/proc/sys/fs/binfmt_misc/python3.10:offset 0
/proc/sys/fs/binfmt_misc/python3.10:magic 6f0d0d0a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled
$ curl.exe -s -I https://www.google.com/ | head -n 1
-bash: /mnt/c/Windows/system32/curl.exe: cannot execute binary file: Exec format error
$ sudo systemctl start wsl-binfmt
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/WSLInterop-late:enabled
/proc/sys/fs/binfmt_misc/WSLInterop-late:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop-late:flags: P
/proc/sys/fs/binfmt_misc/WSLInterop-late:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop-late:magic 4d5a
/proc/sys/fs/binfmt_misc/python3.10:enabled
/proc/sys/fs/binfmt_misc/python3.10:interpreter /usr/bin/python3.10
/proc/sys/fs/binfmt_misc/python3.10:flags:
/proc/sys/fs/binfmt_misc/python3.10:offset 0
/proc/sys/fs/binfmt_misc/python3.10:magic 6f0d0d0a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled
$ curl.exe -s -I https://www.google.com/ | head -n 1
HTTP/1.1 200 OK

In my opinion, the problem is that minikube mounts binfmt_misc of the host WSL Ubuntu and runs systemd-binfmt to rewrite it when the kicbase container starts.
WSL itself seems to have taken care of this by installing /etc/systemd/system/systemd-binfmt.service.d/00-wsl.conf. (Probably in this same way: microsoft/WSL#9029 (comment))

$ cat /etc/systemd/system/systemd-binfmt.service.d/00-wsl.conf
[Unit]
ConditionVirtualization=!container

Is it possible to do the same for minikube?
I thought about mounting it from the host, but it doesn't seem to be supported yet. #12733

afbjorklund · 2023-12-17T10:11:16Z

There is no need to install python binfmt for minikube, so we should just remove this "feature" from ubuntu

Then we don't to restore any pre-existing config from the docker host that was wiped, including WSL (or Rosetta)

spowelljr · 2023-12-19T17:49:13Z

@fbuetler @sugymt Could you try with --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1702948046-17830 and see if my fix resolves it

fbuetler · 2023-12-20T09:34:21Z

@spowelljr looks good 👍🏻

❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

~
❯ /opt/homebrew/Cellar/minikube/1.32.0/bin/minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1702948046-17830
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
    > gcr.io/k8s-minikube/kicbase...:  413.18 MiB / 413.18 MiB  100.00% 23.09 M
🏃  Updating the running docker "minikube" container ...
❗  Image was not built for the current minikube version. To resolve this you can delete and recreate your minikube cluster using the latest images. Expected minikube version: v1.31.0 -> Actual minikube version: v1.32.0
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.4 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~                                                                                                                                                                                                                                                                                                                          
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

sugymt · 2023-12-20T14:44:54Z

@spowelljr
Unfortunately it doesn't seem to work on WSL.

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d
$ curl.exe -s -I https://www.google.com/ | head -n 1
HTTP/1.1 200 OK
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/WSLInterop:enabled
/proc/sys/fs/binfmt_misc/WSLInterop:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop:flags: PF
/proc/sys/fs/binfmt_misc/WSLInterop:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop:magic 4d5a
/proc/sys/fs/binfmt_misc/WSLInterop-late:enabled
/proc/sys/fs/binfmt_misc/WSLInterop-late:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop-late:flags: P
/proc/sys/fs/binfmt_misc/WSLInterop-late:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop-late:magic 4d5a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled
$ minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1702948046-17830
😄  minikube v1.32.0 on Ubuntu 22.04 (amd64)
✨  Automatically selected the docker driver. Other choices: ssh, none
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.28.3 preload ...
    > preloaded-images-k8s-v18-v1...:  403.35 MiB / 403.35 MiB  100.00% 15.97 M
    > gcr.io/k8s-minikube/kicbase...:  456.54 MiB / 456.54 MiB  100.00% 14.26 M
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ curl.exe -s -I https://www.google.com/ | head -n 1
-bash: /mnt/c/Windows/system32/curl.exe: cannot execute binary file: Exec format error
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/python3.10:enabled
/proc/sys/fs/binfmt_misc/python3.10:interpreter /usr/bin/python3.10
/proc/sys/fs/binfmt_misc/python3.10:flags:
/proc/sys/fs/binfmt_misc/python3.10:offset 0
/proc/sys/fs/binfmt_misc/python3.10:magic 6f0d0d0a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled

Maybe it's because systemd is installed after that.

minikube/deploy/kicbase/Dockerfile

Line 96 in ad5e692

systemd \

spowelljr · 2023-12-20T18:04:38Z

@fbuetler @sugymt I moved removing the file until after installing systemd, if you would mind retrying with --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830

sugymt · 2023-12-20T19:34:02Z

@spowelljr it works!

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d
$ curl.exe -s -I https://www.google.com/ | head -n 1
HTTP/1.1 200 OK
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/WSLInterop:enabled
/proc/sys/fs/binfmt_misc/WSLInterop:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop:flags: PF
/proc/sys/fs/binfmt_misc/WSLInterop:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop:magic 4d5a
/proc/sys/fs/binfmt_misc/WSLInterop-late:enabled
/proc/sys/fs/binfmt_misc/WSLInterop-late:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop-late:flags: P
/proc/sys/fs/binfmt_misc/WSLInterop-late:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop-late:magic 4d5a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled
$ minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830
😄  minikube v1.32.0 on Ubuntu 22.04 (amd64)
✨  Automatically selected the docker driver. Other choices: none, ssh
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.28.3 preload ...
    > preloaded-images-k8s-v18-v1...:  403.35 MiB / 403.35 MiB  100.00% 28.39 M
    > gcr.io/k8s-minikube/kicbase...:  456.56 MiB / 456.56 MiB  100.00% 29.36 M
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ curl.exe -s -I https://www.google.com/ | head -n 1
HTTP/1.1 200 OK
$ sudo grep . /proc/sys/fs/binfmt_misc/*
/proc/sys/fs/binfmt_misc/WSLInterop:enabled
/proc/sys/fs/binfmt_misc/WSLInterop:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop:flags: PF
/proc/sys/fs/binfmt_misc/WSLInterop:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop:magic 4d5a
/proc/sys/fs/binfmt_misc/WSLInterop-late:enabled
/proc/sys/fs/binfmt_misc/WSLInterop-late:interpreter /init
/proc/sys/fs/binfmt_misc/WSLInterop-late:flags: P
/proc/sys/fs/binfmt_misc/WSLInterop-late:offset 0
/proc/sys/fs/binfmt_misc/WSLInterop-late:magic 4d5a
grep: /proc/sys/fs/binfmt_misc/register: Invalid argument
/proc/sys/fs/binfmt_misc/status:enabled

fbuetler · 2024-01-03T09:21:11Z

@spowelljr also works on my machine 👍

~                                                                                                                                                                                                                                                                                                                        
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

~
❯ /opt/homebrew/Cellar/minikube/1.32.0/bin/minikube start --base-image gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703092832-17830
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~                                                                                                                                                                                                                                                                                                                          
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

spowelljr · 2024-01-03T18:03:05Z

Thanks for both trying it, just waiting for people to get back from vacation so they can approve my PR.

BenTheElder · 2024-02-08T23:52:17Z

FWIW: systemd is not supposed to be run in a privileged container per their docs, so we have to keep watching out for changes like this.

I think something like this is probably a better fix:
https://github.com/kubernetes-sigs/kind/pull/3511/files#diff-55b511fe27bbfb1625d543b65b6b81e030b8ca93b9437b8b48666344ae9d162dR94

csabatuz-chess mentioned this issue Nov 30, 2023

Minikube breaks x86 emulation orbstack/orbstack#829

Closed

afbjorklund added kind/bug Categorizes issue or PR as related to a bug. co/docker-driver Issues related to kubernetes in container priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Dec 5, 2023

afbjorklund mentioned this issue Dec 6, 2023

Add the kic base image version to the output #17739

Merged

afbjorklund mentioned this issue Dec 11, 2023

on wsl2, minikube deleted the folder and files in /proc/sys/fs/binfmt_misc/WSLInterop, caused the cannot run windows APP in wsl2 #17756

Closed

spowelljr mentioned this issue Dec 18, 2023

Kicbase: Fix modifying Docker binfmt config #17830

Merged

spowelljr closed this as completed in #17830 Jan 9, 2024

f84anton mentioned this issue Feb 8, 2024

v0.21.0: kind container makes changes to binfmt configutation kubernetes-sigs/kind#3510

Closed

jglogan mentioned this issue Feb 9, 2024

docker run error 'exec format error' post upgrade on Apple silicon M2 docker/for-mac#7168

Closed

afbjorklund mentioned this issue Mar 20, 2024

kicbase runs systemd-binfmt.service and wipes out host binfmt registrations #18455

Closed

Sefi4 mentioned this issue Mar 21, 2024

Minikube on WSL causes Exec Format Error #18484

Closed

ConnorNelson mentioned this issue Sep 5, 2024

Infrastructure: Fix MacOS support pwncollege/dojo#558

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images #17700

Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images #17700

csabatuz-chess commented Nov 30, 2023

csabatuz-chess commented Nov 30, 2023

phbelitz commented Nov 30, 2023

afbjorklund commented Nov 30, 2023

cotej commented Dec 5, 2023

afbjorklund commented Dec 5, 2023 •

edited

Loading

afbjorklund commented Dec 5, 2023

fbuetler commented Dec 6, 2023 •

edited

Loading

afbjorklund commented Dec 6, 2023 •

edited

Loading

spowelljr commented Dec 6, 2023 •

edited

Loading

fbuetler commented Dec 13, 2023 •

edited

Loading

fbuetler commented Dec 13, 2023

sugymt commented Dec 17, 2023

afbjorklund commented Dec 17, 2023 •

edited

Loading

spowelljr commented Dec 19, 2023

fbuetler commented Dec 20, 2023

sugymt commented Dec 20, 2023

spowelljr commented Dec 20, 2023

sugymt commented Dec 20, 2023

fbuetler commented Jan 3, 2024 •

edited

Loading

spowelljr commented Jan 3, 2024

BenTheElder commented Feb 8, 2024

Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images #17700

Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images #17700

Comments

csabatuz-chess commented Nov 30, 2023

What Happened?

Attach the log file

Operating System

Driver

csabatuz-chess commented Nov 30, 2023

phbelitz commented Nov 30, 2023

afbjorklund commented Nov 30, 2023

cotej commented Dec 5, 2023

afbjorklund commented Dec 5, 2023 • edited Loading

afbjorklund commented Dec 5, 2023

fbuetler commented Dec 6, 2023 • edited Loading

afbjorklund commented Dec 6, 2023 • edited Loading

spowelljr commented Dec 6, 2023 • edited Loading

fbuetler commented Dec 13, 2023 • edited Loading

fbuetler commented Dec 13, 2023

sugymt commented Dec 17, 2023

afbjorklund commented Dec 17, 2023 • edited Loading

spowelljr commented Dec 19, 2023

fbuetler commented Dec 20, 2023

sugymt commented Dec 20, 2023

spowelljr commented Dec 20, 2023

sugymt commented Dec 20, 2023

fbuetler commented Jan 3, 2024 • edited Loading

spowelljr commented Jan 3, 2024

BenTheElder commented Feb 8, 2024

afbjorklund commented Dec 5, 2023 •

edited

Loading

fbuetler commented Dec 6, 2023 •

edited

Loading

afbjorklund commented Dec 6, 2023 •

edited

Loading

spowelljr commented Dec 6, 2023 •

edited

Loading

fbuetler commented Dec 13, 2023 •

edited

Loading

afbjorklund commented Dec 17, 2023 •

edited

Loading

fbuetler commented Jan 3, 2024 •

edited

Loading