Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker Engine does not start on Ubuntu 22.04 due to iptables issues #1437

Open
kellervater opened this issue Oct 14, 2022 · 10 comments
Open

Docker Engine does not start on Ubuntu 22.04 due to iptables issues #1437

kellervater opened this issue Oct 14, 2022 · 10 comments

Comments

@kellervater
Copy link

kellervater commented Oct 14, 2022 •
edited
Loading

Got a server with Ubuntu 22.04
and tried to execute these steps (like described in official docs):

#first remove everything
apt-get remove docker docker-engine docker.io containerd runc -y
apt-get purge docker-ce docker-ce-cli containerd.io docker-compose-plugin -y
apt clean 
apt autoremove -y
rm -rf /var/lib/docker
rm -rf /var/lib/containerd
rm -rf /var/run/docker

apt-get update
apt-get install ca-certificates curl gnupg lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

apt-get update
apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

I always end up having this output:

Setting up docker-ce (5:20.10.19~3-0~ubuntu-jammy) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Fri 2022-10-14 08:24:26 UTC; 5ms ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 23891 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 23891 (code=exited, status=1/FAILURE)
        CPU: 139ms
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1

I tried all Docker versions down to 5:20.10.13~3-0~ubuntu-jammy.

When checking the logs with dockerd -D I found 2 interesting lines:

...
WARN[2022-10-14T08:24:30.801222966Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.7 (nf_tables): table `nat' is incompatible, use 'nft' tool.`, error: exit status 1
...
failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to Setup IP tables: Unable to allow intercontainer communication:  (iptables failed: iptables --wait -I FORWARD -i docker0 -o docker0 -j ACCEPT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (Operation not supported): chain FORWARD
(exit status 4))

So I also reinstalled iptables (v1.8.7, there's no other version to try when looking in apt-cache madison ).

Nothing helped so far.
I also tried installing it via Ansible roles (geerlingguy) in the first place. this works perfectly fine on my Ubuntu 20.04 servers.

AND one addition:
It once worked on Ubuntu 22.04 when installing docker during the installation of the OS itself.
I recently reinstalled the OS without docker and did the installation as described above, which led to this issue.

Does someone have a clue what's going on or what am I doing wrong?
@mijofa
Copy link

mijofa commented Oct 20, 2022

I seem to have also run into this issue, it was working fine for weeks, but I rebooted early this week and now it's all stopped working.

Looks like maybe a change to the kernel/iptables versions had broken iptables entirely, which docker requires.
I don't think it's a problem with docker itself

@fabianbees
Copy link

I have also experienced this issue, but could resolve it by downgrading iptables (with running the following comands as root):

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

@mijofa
Copy link

mijofa commented Oct 28, 2022

Oh I forgot I'd checked in on this one. The issue I ran into was actually this one with Ubuntu upstream: https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1958260

I don't know if you are having the same problem, but just in case, I was able to get it working again with::

sudo umount --lazy /usr/lib/modules
# Make sure this only reinstalls things, no newly installed packages.
# You should be safe to purge any packages it wants to newly install before rerunning it.
sudo apt install --reinstall $(sudo dpkg-query -S /lib/modules | sed 's/,//g;s/:.*$//')  
sudo reboot

I have not yet confirmed that it won't come back again, nor have I determined why/how it became a problem in the first place since I'm not running anything "cloud" on this system.

@eagleusb
Copy link

I have also experienced this issue, but could resolve it by downgrading iptables (with running the following comands as root):

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

I confirm that the latest default version of iptables is against nftables and using the legacy ones solve it. 😸

@kellervater
Copy link
Author

Since I was under time-pressure, I went back to Ubunut 20.04 to workaround this.
But if you guys were able to workaround this by using legacy modules, I think we can let this issue go stale or even close it.

@Rositza
Copy link

Rositza commented Mar 21, 2023

Hi, I found the issue while searching for decision. Actually I have the same problem on Ubuntu 20.04.3 LTS.

sayboras added a commit to sayboras/packer-ci-build that referenced this issue Mar 25, 2023
@sayboras
docker: Set alternatives for iptables
540c1ba 
This is to avoid the below issue while starting up docker.

```
Mar 24 22:59:10 ubuntu-jammy dockerd[2820]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument
```

Relates: docker/for-linux#1437
sayboras added a commit to sayboras/packer-ci-build that referenced this issue Mar 25, 2023
@sayboras
docker: Set alternatives for iptables
9bcab51 
This is to avoid the below issue while starting up docker.

```
Mar 24 22:59:10 ubuntu-jammy dockerd[2820]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument
```

Relates: docker/for-linux#1437

Signed-off-by: Tam Mach <[email protected]>
@davidmiguel02
Copy link

Help me.
Thank you

@Idosegevv
Copy link

Worked for me ass well on Petalinux dist of PYNQ-RFSoC

@mdaffad mdaffad mentioned this issue Jul 11, 2023
Closed
leblowl pushed a commit to leblowl/packer-ci-build that referenced this issue Aug 7, 2023
@sayboras @leblowl
docker: Set alternatives for iptables
7241ba5 
This is to avoid the below issue while starting up docker.

```
Mar 24 22:59:10 ubuntu-jammy dockerd[2820]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument
```

Relates: docker/for-linux#1437

Signed-off-by: Tam Mach <[email protected]>
aanm pushed a commit to cilium/packer-ci-build that referenced this issue Aug 8, 2023
@sayboras @aanm
docker: Set alternatives for iptables
a5933aa 
This is to avoid the below issue while starting up docker.

```
Mar 24 22:59:10 ubuntu-jammy dockerd[2820]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument
```

Relates: docker/for-linux#1437

Signed-off-by: Tam Mach <[email protected]>
@spirillen
Copy link

Well this is still an issue that have to be fixed, as iptables have been deprecated for year for nftables. So why are the commands not updated to match the real world realities...

DEBU[2024-04-09T13:26:36.186488670Z] Network Control Plane MTU: 1500              
DEBU[2024-04-09T13:26:36.188829616Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION] 
DEBU[2024-04-09T13:26:36.189759746Z] /usr/sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2024-04-09T13:26:36.190840751Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER] 
DEBU[2024-04-09T13:26:36.191795232Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER] 
DEBU[2024-04-09T13:26:36.192827253Z] /usr/sbin/iptables, [--wait -t nat -D PREROUTING] 
DEBU[2024-04-09T13:26:36.193805364Z] /usr/sbin/iptables, [--wait -t nat -D OUTPUT] 
DEBU[2024-04-09T13:26:36.194672033Z] /usr/sbin/iptables, [--wait -t nat -F DOCKER] 
DEBU[2024-04-09T13:26:36.195517505Z] /usr/sbin/iptables, [--wait -t nat -X DOCKER] 
DEBU[2024-04-09T13:26:36.262160016Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER] 
DEBU[2024-04-09T13:26:36.265093372Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER] 
DEBU[2024-04-09T13:26:36.267894264Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-1] 
DEBU[2024-04-09T13:26:36.322261864Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-1] 
DEBU[2024-04-09T13:26:36.325635441Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-2] 
DEBU[2024-04-09T13:26:36.390282204Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-2]
DEBU[2024-04-09T13:26:36.393465502Z] /usr/sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[2024-04-09T13:26:36.396226517Z] /usr/sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[2024-04-09T13:26:36.399023334Z] /usr/sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[2024-04-09T13:26:36.401828580Z] /usr/sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[2024-04-09T13:26:36.404467507Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[2024-04-09T13:26:36.407222513Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER]
DEBU[2024-04-09T13:26:36.409686321Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-1]
DEBU[2024-04-09T13:26:36.412109893Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-1]
DEBU[2024-04-09T13:26:36.414436184Z] /usr/sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-2]
DEBU[2024-04-09T13:26:36.416687171Z] /usr/sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-2]
DEBU[2024-04-09T13:26:36.418934837Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2024-04-09T13:26:36.421334686Z] /usr/sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2024-04-09T13:26:36.423607874Z] /usr/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN]
DEBU[2024-04-09T13:26:36.425957943Z] /usr/sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-2 -j RETURN]
DEBU[2024-04-09T13:26:36.435780805Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.438306631Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[2024-04-09T13:26:36.440625841Z] /usr/sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN]
DEBU[2024-04-09T13:26:36.442995562Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.445555108Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j DROP]
DEBU[2024-04-09T13:26:36.447780342Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2024-04-09T13:26:36.450094555Z] /usr/sbin/iptables, [--wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT]
WARN[2024-04-09T13:26:36.486139751Z] could not create bridge network for id ca495b73f0d5180761e35cfab4e58571671bf1bb1eb1c66e681b7fbac426d96f bridge name docker0 while booting up from persistent state: Failed to Setup IP tables: Unable to allow intercontainer communication:  (iptables failed: iptables --wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (Operation not supported): chain FORWARD
(exit status 4))
DEBU[2024-04-09T13:26:36.486184603Z] Network (ca495b7) restored
INFO[2024-04-09T13:26:36.492367578Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
DEBU[2024-04-09T13:26:36.492435409Z] Allocating IPv4 pools for network bridge (a6381b2be7e366a9597643115f5bf823a7c28ee2f6dd5fa0642d1981841ab132)
DEBU[2024-04-09T13:26:36.492454280Z] RequestPool(LocalDefault, 172.17.0.0/16, , _, false)
DEBU[2024-04-09T13:26:36.492498645Z] RequestAddress(LocalDefault/172.17.0.0/16, 172.17.0.1, map[RequestAddressType:com.docker.network.gateway])
DEBU[2024-04-09T13:26:36.492537499Z] Request address PoolID:172.17.0.0/16 Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:172.17.0.1
DEBU[2024-04-09T13:26:36.492919071Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.495827334Z] /usr/sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[2024-04-09T13:26:36.498628169Z] /usr/sbin/iptables, [--wait -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker0 -j MASQUERADE]
DEBU[2024-04-09T13:26:36.501635211Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j DROP]
DEBU[2024-04-09T13:26:36.504260105Z] /usr/sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2024-04-09T13:26:36.506858071Z] /usr/sbin/iptables, [--wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2024-04-09T13:26:36.554482717Z] releasing IPv4 pools from network bridge (a6381b2be7e366a9597643115f5bf823a7c28ee2f6dd5fa0642d1981841ab132)
DEBU[2024-04-09T13:26:36.554532727Z] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.1)
DEBU[2024-04-09T13:26:36.554573789Z] Released address Address:172.17.0.1 Sequence:Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0
DEBU[2024-04-09T13:26:36.554591415Z] ReleasePool(LocalDefault/172.17.0.0/16)
DEBU[2024-04-09T13:26:36.554620258Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2024-04-09T13:26:36.554640143Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2024-04-09T13:26:36.556095581Z] Unix socket /var/run/docker/libnetwork/e28645b2e610.sock was closed. The external key listener will stop.
INFO[2024-04-09T13:26:36.556801370Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
DEBU[2024-04-09T13:26:36.556828184Z] Cleaning up old mountid : start.
DEBU[2024-04-09T13:26:36.557417089Z] Cleaning up old mountid : done.
failed to start daemon: Error initializing network controller: error creating default "bridge" network: Failed to Setup IP tables: Unable to allow intercontainer communication:  (iptables failed: iptables --wait -t filter -I FORWARD -i docker0 -o docker0 -j ACCEPT: iptables v1.8.7 (nf_tables):  CHAIN_ADD failed (Operation not supported): chain FORWARD
(exit status 4))

Keep using deprecated software is just so bad and calls out for errors and insecurity

@nandaadi241
Copy link

thanks bro its help me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@mijofa @Rositza @eagleusb @fabianbees @kellervater @spirillen @Idosegevv @davidmiguel02 @nandaadi241