Unmarshalling error when using `podman exec` #20821

jloewe · 2023-11-28T19:09:00Z

Issue Description

When I use podman exec ... for any container, I get the following error:

Error: unmarshalling error into &errorhandling.ErrorModel{Because:"", Message:"", ResponseCode:0}, data "Not Found\n": invalid character 'N' looking for beginning of value

Steps to reproduce the issue

Run a detached container: podman run --name httpd -p 8080:80 -d -i -t docker.io/library/httpd
Try to run podman exec: podman exec httpd pwd
Error message shows up

Describe the results you received

An unexpected error message.

Describe the results you expected

No error message, if the command is run correctly.

podman info output

host:
  arch: arm64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 99.33
    systemPercent: 0.34
    userPercent: 0.33
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2039
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.5.11-300.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 3151986688
  memTotal: 6189867008
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.1-1.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.1
      commit: 1084f9527c143699b593b44c23555fb3cc4ff2f3
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc39.aarch64
    version: |
      pasta 0^20231004.gf851084-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 12h 26m 50.00s (Approximately 0.50 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 6775140352
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 33
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762633
  BuiltTime: Tue Oct 31 15:30:33 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/arm64
  Version: 4.7.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

ZSH on MacOS

Additional information

When downgrading to v4.7.2 the issue disappeared.

The text was updated successfully, but these errors were encountered:

Luap99 · 2023-11-29T11:07:04Z

This is because your 4.8 client talks to a 4.7 server and tries to call a new endpoint that does not exists there. When the server is updated it should work again.

Regardless this must be fixed on client, the new client should never call endpoints that do not exits on older versions.

Commit f48a706 added a new API endpoint to remove exec session correctly. And the bindings try to call that endpoint for exec every time. Now since client and server must not be the same version this causes a problem if a new 4.8 client calls an older 4.7 server as it has no idea about such endpoint and throws an ugly error. This is a common scenario for podman machine setups. The client does know the server version so it should make sure to not call such endpoint if the server is older than 4.8. I added a exec test to the machine tests as this can be reproduced with podman machine as at the moment at least the VM image does not contain podman 4.8. And it should at least make sure podman exec keeps working for podman machine without regressions. Fixes containers#20821 Signed-off-by: Paul Holzinger <[email protected]>

jloewe · 2023-11-29T12:26:27Z

I guess I have to wait until the fedora package is out of testing and I can update the machine (or it does it automatically).
Thank you for the insights!

Luap99 · 2023-11-29T12:35:32Z

It will take a bit more time as you need to wait for it to land in coreos and that takes another two weeks, see https://docs.podman.io/en/latest/markdown/podman-machine-init.1.html on how you can update.

Thus it is likely much faster for us to merge my client side fix and release 4.8.1 with it.

baude · 2023-11-30T21:31:20Z

@jloewe you could also use podman machine os apply and fix it straight away!

jimlindeman · 2023-12-01T00:41:30Z

@baude unfortunately, that workaround doesn't work for the existing minikube/kind kubernetes frameworks that just expect exec to work against the default machine image. Those frameworks are expecting "N-1" compatibility for podman client/server APIs (as that's standard for kubernetes & its client kubectl).

Commit f48a706 added a new API endpoint to remove exec session correctly. And the bindings try to call that endpoint for exec every time. Now since client and server must not be the same version this causes a problem if a new 4.8 client calls an older 4.7 server as it has no idea about such endpoint and throws an ugly error. This is a common scenario for podman machine setups. The client does know the server version so it should make sure to not call such endpoint if the server is older than 4.8. I added a exec test to the machine tests as this can be reproduced with podman machine as at the moment at least the VM image does not contain podman 4.8. And it should at least make sure podman exec keeps working for podman machine without regressions. Fixes containers#20821 Signed-off-by: Paul Holzinger <[email protected]>

BenTheElder · 2023-12-01T19:37:41Z

@baude unfortunately, that workaround doesn't work for the existing minikube/kind kubernetes frameworks that just expect exec to work against the default machine image. Those frameworks are expecting "N-1" compatibility for podman client/server APIs (as that's standard for kubernetes & its client kubectl).

I the case of kind we're currently literally just execing podman exec using the host's podman binary, so I expect podman commands like this to work w/o knowing the details of the VM etc.

I wouldn't necessarily comment on if podman should have skew support, that's out of scope for KIND.

I could imagine another approach would be packaging podman and the VM implementation together and ensuring they don't skew but regardless it's not really our concern, speaking for KIND.

jloewe added the kind/bug Categorizes issue or PR as related to a bug. label Nov 28, 2023

github-actions bot added the remote Problem is in podman-remote label Nov 28, 2023

Luap99 added the regression label Nov 29, 2023

Luap99 self-assigned this Nov 29, 2023

Luap99 mentioned this issue Nov 29, 2023

fix podman-remote exec regression with v4.8 #20831

Merged

afbjorklund mentioned this issue Nov 29, 2023

minikube fails to start on Mac with podman 4.8.0 (works with podman 4.7.2) kubernetes/minikube#17684

Closed

Luap99 mentioned this issue Nov 30, 2023

kind create cluster fail after podman latest update #20854

Closed

This was referenced Nov 30, 2023

Not able to start kind cluster in Mac(M2) using podman [podman exec broken upstream due to mismatched client/server version] kubernetes-sigs/kind#3439

Closed

Failed to create cluster kubernetes-sigs/kind#3440

Closed

openshift-merge-bot bot closed this as completed in #20831 Dec 1, 2023

jlebon mentioned this issue Dec 4, 2023

build-arch jobs failing with "Error: unmarshalling error into &errorhandling.ErrorModel" coreos/coreos-assembler#3678

Closed

c4rt0 mentioned this issue Dec 4, 2023

stable: new release on 2023-12-04 (39.20231119.3.0) coreos/fedora-coreos-streams#826

Closed

43 tasks

dustymabe mentioned this issue Dec 4, 2023

various issues with podman 4.8 coreos/fedora-coreos-tracker#1625

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Mar 1, 2024

github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unmarshalling error when using `podman exec` #20821

Unmarshalling error when using `podman exec` #20821

jloewe commented Nov 28, 2023

Luap99 commented Nov 29, 2023

jloewe commented Nov 29, 2023

Luap99 commented Nov 29, 2023

baude commented Nov 30, 2023

jimlindeman commented Dec 1, 2023

BenTheElder commented Dec 1, 2023

Unmarshalling error when using podman exec #20821

Unmarshalling error when using podman exec #20821

Comments

jloewe commented Nov 28, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

Luap99 commented Nov 29, 2023

jloewe commented Nov 29, 2023

Luap99 commented Nov 29, 2023

baude commented Nov 30, 2023

jimlindeman commented Dec 1, 2023

BenTheElder commented Dec 1, 2023

Unmarshalling error when using `podman exec` #20821

Unmarshalling error when using `podman exec` #20821