[BUG] error waiting for log line k3s is up and running

[rc452860]

What did you do

• How was the cluster created?
  using command: k3d cluster create --agents 3 demo

• What did you do afterwards?
  Just upgrade k3d version to 5.7.3, after I tried several versions, I found that version 5.6.0 works well, and version 5.6.3 does not work.

What did you expect to happen

start cluster error with: error waiting for log line k3s is up and running

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

# k3d  cluster create --agents 3 demo 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-demo'                   
INFO[0000] Created image volume k3d-demo-images         
INFO[0000] Starting new tools node...                   
INFO[0000] Starting node 'k3d-demo-tools'               
INFO[0001] Creating node 'k3d-demo-server-0'            
INFO[0001] Creating node 'k3d-demo-agent-0'             
INFO[0001] Creating node 'k3d-demo-agent-1'             
INFO[0001] Creating node 'k3d-demo-agent-2'             
INFO[0001] Creating LoadBalancer 'k3d-demo-serverlb'    
INFO[0001] Using the k3d-tools node to gather environment information 
INFO[0001] HostIP: using network gateway 192.168.64.1 address 
INFO[0001] Starting cluster 'demo'                      
INFO[0001] Starting servers...                          
INFO[0001] Starting node 'k3d-demo-server-0'            
ERRO[0002] Failed Cluster Start: Failed to start server k3d-demo-server-0: Node k3d-demo-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-demo-server-0': stopped returning log lines: node k3d-demo-server-0 is running=true in status=restarting 
ERRO[0002] Failed to create cluster >>> Rolling Back    
INFO[0002] Deleting cluster 'demo'                      
INFO[0002] Deleting cluster network 'k3d-demo'          
INFO[0002] Deleting 1 attached volumes...               
FATA[0002] Cluster creation FAILED, all changes have been rolled back! 

Which OS & Architecture

arch: x86_64
cgroupdriver: cgroupfs
cgroupversion: "1"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: VM-12-32-centos
name: docker
os: Tencent tlinux 2.6
ostype: linux
version: 20.10.22

Which version of k3d

k3d version v5.7.3
k3s version v1.30.3-k3s1 (default)

Which version of docker

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:30:24 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:28:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.27.0)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 9
  Running: 9
  Paused: 0
  Stopped: 0
 Images: 41
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.119-1-tlinux4-0010.2
 Operating System: Tencent tlinux 2.6
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.38GiB
 Name: VM-12-32-centos
 ID: WMUJ:FIQW:XUAF:O4N4:RTXL:ROBN:ICZX:PKA4:WF5V:EACC:TLEK:STOW
 Docker Root Dir: /data/docker
 Debug Mode: false
 Username: arkura
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  docker.oa.com
  dockerimage.isd.com
  registry.christtu.com:5000
  csighub.tencentyun.com
  127.0.0.0/8
 Registry Mirrors:
  https://dockerhub.woa.com/
 Live Restore Enabled: false

[rc452860]

After adding the parameter -v /tmp/asdf:/var/log, Below is the output log.

# tail /tmp/asdf/k3d-entrypoints_240806085645.log 
Bad argument `[unsupported'
Error occurred at line: 19
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
[2024-08-06T08:56:45+00:00] Running k3d entrypoints...
[2024-08-06T08:56:45+00:00] Running /bin/k3d-entrypoint-dns.sh
[2024-08-06T08:56:45+00:00] [DNS Fix] Use the detected Gateway IP 192.168.80.1 instead of Docker's embedded DNS (127.0.0.11)
[2024-08-06T08:56:45+00:00] [DNS Fix] > Changing iptables rules ...
Bad argument `[unsupported'
Error occurred at line: 19
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
(base) [root@VM-12-32-centos /data/k3d]# k3d runtime-info

[rc452860]

this may be help

# iptables -h
iptables v1.4.21

Usage: iptables -[ACD] chain rule-specification [options]
       iptables -I chain [rulenum] rule-specification [options]
       iptables -R chain rulenum rule-specification [options]
       iptables -D chain rulenum [options]
       iptables -[LS] [chain [rulenum]] [options]
       iptables -[FZ] [chain] [options]
       iptables -[NX] chain
       iptables -E old-chain-name new-chain-name
       iptables -P chain target [options]
       iptables -h (print this help information)

Commands:
Either long or short options are allowed.
  --append  -A chain            Append to chain
  --check   -C chain            Check for the existence of a rule
  --delete  -D chain            Delete matching rule from chain
  --delete  -D chain rulenum
                                Delete rule rulenum (1 = first) from chain
  --insert  -I chain [rulenum]
                                Insert in chain as rulenum (default 1=first)
  --replace -R chain rulenum
                                Replace rule rulenum (1 = first) in chain
  --list    -L [chain [rulenum]]
                                List the rules in a chain or all chains
  --list-rules -S [chain [rulenum]]
                                Print the rules in a chain or all chains
  --flush   -F [chain]          Delete all rules in  chain or all chains
  --zero    -Z [chain [rulenum]]
                                Zero counters in chain or all chains
  --new     -N chain            Create a new user-defined chain
  --delete-chain
            -X [chain]          Delete a user-defined chain
  --policy  -P chain target
                                Change policy on chain to target
  --rename-chain
            -E old-chain new-chain
                                Change chain name, (moving any references)
Options:
    --ipv4      -4              Nothing (line is ignored by ip6tables-restore)
    --ipv6      -6              Error (line is ignored by iptables-restore)
[!] --protocol  -p proto        protocol: by number or name, eg. `tcp'
[!] --source    -s address[/mask][...]
                                source specification
[!] --destination -d address[/mask][...]
                                destination specification
[!] --in-interface -i input name[+]
                                network interface name ([+] for wildcard)
 --jump -j target
                                target for rule (may load target extension)
  --goto      -g chain
                              jump to chain with no return
  --match       -m match
                                extended match (may load extension)
  --numeric     -n              numeric output of addresses and ports
[!] --out-interface -o output name[+]
                                network interface name ([+] for wildcard)
  --table       -t table        table to manipulate (default: `filter')
  --verbose     -v              verbose mode
  --wait        -w [seconds]    maximum wait to acquire xtables lock before give up
  --wait-interval -W [usecs]    wait time to try to acquire xtables lock
                                default is 1 second
  --line-numbers                print line numbers when listing
  --exact       -x              expand numbers (display exact values)
[!] --fragment  -f              match second or further fragments only
  --modprobe=<command>          try to insert modules using this command
  --set-counters PKTS BYTES     set the counter during insert/append
[!] --version   -V              print package version.
# sed --version
sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <[email protected]>.
Be sure to include the word ``sed'' somewhere in the ``Subject:'' field.
# sed 
Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...

  -n, --quiet, --silent
                 suppress automatic printing of pattern space
  -e script, --expression=script
                 add the script to the commands to be executed
  -f script-file, --file=script-file
                 add the contents of script-file to the commands to be executed
  --follow-symlinks
                 follow symlinks when processing in place
  -i[SUFFIX], --in-place[=SUFFIX]
                 edit files in place (makes backup if SUFFIX supplied)
  -c, --copy
                 use copy instead of rename when shuffling files in -i mode
  -b, --binary
                 does nothing; for compatibility with WIN32/CYGWIN/MSDOS/EMX (
                 open files in binary mode (CR+LFs are not treated specially))
  -l N, --line-length=N
                 specify the desired line-wrap length for the `l' command
  --posix
                 disable all GNU extensions.
  -r, --regexp-extended
                 use extended regular expressions in the script.
  -s, --separate
                 consider files as separate rather than as a single continuous
                 long stream.
  -u, --unbuffered
                 load minimal amounts of data from the input files and flush
                 the output buffers more often
  -z, --null-data
                 separate lines by NUL characters
  --help
                 display this help and exit
  --version
                 output version information and exit

If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret.  All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.

GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.

[conradwt]

I was able to get things working without error using the following command:

➜ k3d version                                                                                                 
k3d version v5.7.3
k3s version v1.30.3-k3s1 (default)

➜ k3d cluster create mycluster2 --api-port 6550 --servers 3 --agents 3 --image rancher/k3s:v1.30.3-k3s1 --wait

➜ k get node                                                                                                  
NAME                      STATUS   ROLES                       AGE   VERSION
k3d-mycluster2-agent-0    Ready    <none>                      39s   v1.30.3+k3s1
k3d-mycluster2-agent-1    Ready    <none>                      40s   v1.30.3+k3s1
k3d-mycluster2-agent-2    Ready    <none>                      40s   v1.30.3+k3s1
k3d-mycluster2-server-0   Ready    control-plane,etcd,master   75s   v1.30.3+k3s1
k3d-mycluster2-server-1   Ready    control-plane,etcd,master   58s   v1.30.3+k3s1
k3d-mycluster2-server-2   Ready    control-plane,etcd,master   43s   v1.30.3+k3s1

[meenakshi-koushik]

Had the same problem with a different use-case:

k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.27.15-k3s1 --wait --timeout=240s --no-lb --network test

This works with k3d 5.6.0 and 1.27.1<=k3s<=1.27.15 but fails with the image is k3s:v1.27.16-k3s1 with the same error:

INFO[0000] Prep: Network
INFO[0000] Created network 'test'
...
INFO[0001] Starting Node 'k3d-test-server-0'
ERRO[0003] Failed Cluster Start: Failed to start server k3d-test-server-0: Node k3d-test-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-test-server-0': stopped returning log lines: node k3d-test-server-0 is running=true in status=running
ERRO[0003] Failed to create cluster >>> Rolling Back
...
FATA[0004] Cluster creation FAILED, all changes have been rolled back!

With v5.6.0 k3d version outputs ... k3s version v1.27.4-k3s1 (default) so I tried to run with newer version. The same problem happens with all released k3d versions upto and including 5.7.2. Although k3d version for v5.7.2 outputs

k3d version v5.7.2
k3s version v1.29.6-k3s2 (default)

It fails to start a cluster with k3s v1.27.16+k3s1 although versions more 'recent' in semver terms work fine.

This problem is fixed in v5.7.3

The reason appears to be this change that adapts to the change in xargs version shipped in k3s. This can be misleading for users since unsupported versions can only be identified by release date. This issue is probably caused by k3s but I think this project needs to be document, for each minor version the highest patch version supported since not all are.

[rc452860]

Had the same problem with a different use-case:

k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.27.15-k3s1 --wait --timeout=240s --no-lb --network test

This works with k3d 5.6.0 and 1.27.1<=k3s<=1.27.15 but fails with the image is k3s:v1.27.16-k3s1 with the same error:

INFO[0000] Prep: Network
INFO[0000] Created network 'test'
...
INFO[0001] Starting Node 'k3d-test-server-0'
ERRO[0003] Failed Cluster Start: Failed to start server k3d-test-server-0: Node k3d-test-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-test-server-0': stopped returning log lines: node k3d-test-server-0 is running=true in status=running
ERRO[0003] Failed to create cluster >>> Rolling Back
...
FATA[0004] Cluster creation FAILED, all changes have been rolled back!

With v5.6.0 k3d version outputs ... k3s version v1.27.4-k3s1 (default) so I tried to run with newer version. The same problem happens with all released k3d versions upto and including 5.7.2. Although k3d version for v5.7.2 outputs

k3d version v5.7.2
k3s version v1.29.6-k3s2 (default)

It fails to start a cluster with k3s v1.27.16+k3s1 although versions more 'recent' in semver terms work fine.

This problem is fixed in v5.7.3

The reason appears to be this change that adapts to the change in xargs version shipped in k3s. This can be misleading for users since unsupported versions can only be identified by release date. This issue is probably caused by k3s but I think this project needs to be document, for each minor version the highest patch version supported since not all are.

yes, i was found this issue. but not work for me. I think is not same problem.

[matheus-rossi]

Had the same problem with a different use-case:

k3d cluster create test --kubeconfig-update-default=0 --image rancher/k3s:v1.27.15-k3s1 --wait --timeout=240s --no-lb --network test

This works with k3d 5.6.0 and 1.27.1<=k3s<=1.27.15 but fails with the image is k3s:v1.27.16-k3s1 with the same error:

INFO[0000] Prep: Network
INFO[0000] Created network 'test'
...
INFO[0001] Starting Node 'k3d-test-server-0'
ERRO[0003] Failed Cluster Start: Failed to start server k3d-test-server-0: Node k3d-test-server-0 failed to get ready: error waiting for log line `k3s is up and running` from node 'k3d-test-server-0': stopped returning log lines: node k3d-test-server-0 is running=true in status=running
ERRO[0003] Failed to create cluster >>> Rolling Back
...
FATA[0004] Cluster creation FAILED, all changes have been rolled back!

With v5.6.0 k3d version outputs ... k3s version v1.27.4-k3s1 (default) so I tried to run with newer version. The same problem happens with all released k3d versions upto and including 5.7.2. Although k3d version for v5.7.2 outputs

k3d version v5.7.2
k3s version v1.29.6-k3s2 (default)

It fails to start a cluster with k3s v1.27.16+k3s1 although versions more 'recent' in semver terms work fine.
This problem is fixed in v5.7.3
The reason appears to be this change that adapts to the change in xargs version shipped in k3s. This can be misleading for users since unsupported versions can only be identified by release date. This issue is probably caused by k3s but I think this project needs to be document, for each minor version the highest patch version supported since not all are.

yes, i was found this issue. but not work for me. I think is not same problem.

Getting the same error here, when trying to init a cluster with terraform
https://registry.terraform.io/providers/moio/k3d/latest

k3d --version
k3d version v5.7.3
k3s version v1.30.3-k3s1 (default)

Did you solve it?