podman behaves differently than docker regarding /etc/hosts management

[reda-alaoui]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Unlike Docker, podman adds the /etc/hosts host entries to the /etc/hosts container

Steps to reproduce the issue:

1. On the host, create an /etc/hosts file with the following content:

127.0.0.1	localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

2. podman run -it --rm busybox cat /etc/hosts
3. docker run -it --rm busybox cat /etc/hosts

Describe the results you received:

rhousni@rhousni:~$ docker run -it --rm busybox cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	10ad1fcdad69

rhousni@rhousni:~$ podman run -it --rm busybox cat /etc/hosts
127.0.0.1	localhost
127.0.0.1	foo

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# used by slirp4netns
10.0.2.100	85e6090e533b admiring_lumiere
10.0.2.2 host.containers.internal

Describe the results you expected:

I expect podman to behave like docker. Entry 127.0.0.1 foo should not be in the container run by podman.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.6
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 16
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: rhousni
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.0-39-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 12841545728
  memTotal: 67288952832
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 1022357504
  swapTotal: 1023406080
  uptime: 30h 11m 39.52s (Approximately 1.25 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/rhousni/.config/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 0
    stopped: 12
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rhousni/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 49
  runRoot: /run/user/1000/containers
  volumePath: /home/rhousni/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 100:3.4.2-1 amd64 [installed]
podman/unknown 100:3.4.2-1 arm64
podman/unknown 100:3.4.2-1 armhf
podman/unknown 100:3.4.2-1 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

We had this discussion before - I need to find the issue. Regardless, the conclusion was that our way seemed to be more correct, and that there were efforts to patch containerd to match it, hence our decision to continue using our existing strategy.

We could potentially add an option to use the container's /etc/hosts instead, but at this point I don't want to change the default.

[mheon]

#11835 is the previous issue.

[reda-alaoui]

So looks like podman 4 has the feature.

[reda-alaoui]

#11835 (comment) says the feature is in version 4, but can't find it in https://docs.podman.io/en/latest/markdown/podman-run.1.html

[mheon]

I don't remember any PRs to add it, so @rhatdan might have been wrong.

[reda-alaoui]

In my case, this difference in behaviour is causing a functional issue.

We have been using docker-compose on our dev environment for a long time to boot our webapp platform.
The docker-compose looks like this:

version: '3.7'

services:
  reverse-proxy:
    image: traefik
    ports:
      - "80:80"
      - "443:443"
    networks:
      default:
        aliases:
          - app.example.org
          - app-identity.example.org
    // ...
  identity-provider:
   // ...
  app:
    // ...

On each developer machine (the host), /etc/hosts contains:

127.0.0.1 app.example.org app-identity.example.org

The identity-provider and app are accessed via the reverse-proxy service.

With Docker, this configuration allows the host and each container to access https://app.example.org and https://app-identity.example.org. I guess that when a container asks for https://app.example.org, app.example.org is resolved as network alias by Docker which makes the thing work.

But when using podman, 127.0.0.1 app.example.org app-identity.example.org ends up in each container /etc/hosts file. The entry then has priority over network aliases. The consequence is that a container cannot access https://app.example.org or https://app-identity.example.org because it is looking for a service on 127.0.0.1 instead of the host.

[Luap99]

We definitely did not merge such changes. I am still looking into rewrting the /etc/hosts stuff and this is on the list as well. But as @mheon said the default will likely stay. I am think of a containers.conf option for that.

[mheon]

Option in containers.conf SGTM

[reda-alaoui]

It would be a good option for me if the new flag is considered by podman system service because the client is not podman (the client is IntelliJ in my case), but a client I have no control on that only knows how to speak with the Docker daemon.

If you give me some guidance (Golang noob here), I could open a pull request (hopefully better than #13725). Unless someone else want/can implement this quickier 😊

[Luap99]

We do not add specific flags like that to podman system service. You can change the containers.conf settings yourself so docker clients would not have to set it. I want to get the /etc/hosts stuff done for podman 4.1 in a few weeks.

[reda-alaoui]

We do not add specific flags like that to podman system service. You can change the containers.conf settings yourself so docker clients would not have to set it. I want to get the /etc/hosts stuff done for podman 4.1 in a few weeks.

I think I haven't been clear enough. I don't want a flag added to podman system service command, I just want this new option in containers.conf to be considered by podman system service. I say that because it seems it is not always the case. For example, containers.no_hosts (in containers.conf) is honored by podman client but not podman system service (#13719) .

[reda-alaoui]

I want to get the /etc/hosts stuff done for podman 4.1 in a few weeks.

If you think I can add an option to containers.conf relatively easily without a big refactoring, I can start working on that earlier (this weekend probably).

[rhatdan]

Yes I think this should be added.

[white-gecko]

I have the same issue as @reda-alaoui with a similar setup, just different host names.

host: /etc/hosts

127.0.0.1 docker.localhost

docker-compose

services:
  apache2:
    …
    networks:
          default:
            aliases:
              - docker.localhost
    …
  app_service1:
    …
  app_service2:
    …

I would look forward to a possibility to be compatible with docker in this regard.

[reda-alaoui]

@white-gecko I have been using podman system service as rootless and $HOME/.config/containers/containers.conf . I haven't had issue making IntelliJ talk to the rootless socket.

[white-gecko]

Since it is always possible to add hosts when starting a pod/container using --add-host, while it is not that easy to remove entries I think it is the best idea to have the pods having not the host's /etc/hosts as default would be a better choice.

Does any workaround currently exist to get a container started without the host's /etc/hosts when going though the socket? (This is somehow also related to or blocked by #13719)

[Luap99]

@mheon WDYT making containers created via the docker compat api default to no base host file to better match docker?

[mheon]

Ehhh. Is it worth the extra complexity? We've heard very few complaints about this...

[rhatdan]

I agree with Matt, I am not sure this is worth the complexity.

[Luap99]

I don't think it is complex, it is a simple change once a cli option for base_hosts_file config option is added. #13277

But if you think we should not do this then close the issue as wontfix. The workaround is to set base_hosts_file in containers.conf (added in v4.1)

[rhatdan]

Lets talk at standup.

[white-gecko]

@Luap99 @rhatdan are you able to share some information on your plans? What was the outcome of the standup?

[rhatdan]

We agreed to add an option for this behaviour and to make it default for the Docker API.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 any update?

[white-gecko]

For me this issue is fixed with the 4.1.0 release and the included base_hosts_file feature (

podman/RELEASE_NOTES.md

Line 37 in cab9779

- Podman's handling of the `/etc/hosts` file has been rewritten to improve its consistency and handling of edge cases ([#12003](https://github.com/containers/podman/issues/12003) and [#13224](https://github.com/containers/podman/issues/13224)). As part of this, two new options are available in `containers.conf`: `base_hosts_file` (to specify a nonstandard location to source the base contents of the container's `/etc/hosts`) and `host_containers_internal_ip` (to specify a specific IP address for containers' `host.containers.internal` entry to point to).

).