Error: Failed to create node ...The container name is already in use by container

[arkadius]

Hi,

We've got such configuration in our project:

      - uses: AbsaOSS/[email protected]
        with:
          cluster-name: "k3s-default"
          args: >-
            --agents 1
            --no-lb
            --k3s-arg "--no-deploy=traefik,servicelb,metrics-server@server:*"

And from time to time we've got such message (https://github.com/TouK/nussknacker/runs/4712393353?check_suite_focus=true) :

Run AbsaOSS/[email protected]
  with:
    cluster-name: k3s-default
    args: --agents 1 --no-lb --k3s-arg "--no-deploy=traefik,servicelb,metrics-server@server:*"
  env:
    CROSS_BUILD: true
    CI: true
    JAVA_HOME: /home/runner/.jabba/jdk/[email protected]
Run /home/runner/work/_actions/AbsaOSS/k3d-action/v2.1.0/run.sh deploy
  /home/runner/work/_actions/AbsaOSS/k3d-action/v2.1.0/run.sh deploy
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    CLUSTER_NAME: k3s-default
    ARGS: --agents 1 --no-lb --k3s-arg "--no-deploy=traefik,servicelb,metrics-server@server:*"
    K3D_VERSION: 
Downloading [email protected] see: https://raw.githubusercontent.com/rancher/k3d/main/install.sh
Preparing to install k3d into /usr/local/bin
k3d installed into /usr/local/bin/k3d
Run 'k3d --help' to see what you can do with it.
Deploy cluster k3s-default 
INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created volume 'k3d-k3s-default-images'      
INFO[0000] Starting new tools node...                   
INFO[0000] Pulling image 'docker.io/rancher/k3d-tools:5.2.2' 
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0001] Pulling image 'docker.io/rancher/k3s:v1.21.7-k3s1' 
INFO[0004] Creating node 'k3d-k3s-default-agent-0'      
INFO[0005] Using the k3d-tools node to gather environment information 
INFO[0005] Starting new tools node...                   
INFO[0005] Starting Node 'k3d-k3s-default-tools'        
ERRO[0005] Failed to run tools container for cluster 'k3s-default' 
INFO[0005] Deleting cluster 'k3s-default'               
ERRO[0005] failed to gather environment information used for cluster creation: failed to run k3d-tools node for cluster 'k3s-default': failed to create node 'k3d-k3s-default-tools': runtime failed to create node 'k3d-k3s-default-tools': failed to create container for node 'k3d-k3s-default-tools': docker failed to create container 'k3d-k3s-default-tools': Error response from daemon: Conflict. The container name "/k3d-k3s-default-tools" is already in use by container "409a0bc5a19cca47cd1f485d9e710ac59b28b397d2cd15d3614e9dcc2fb6f015". You have to remove (or rename) that container to be able to reuse that name. 
ERRO[0005] Failed to create cluster >>> Rolling Back    
INFO[0005] Deleting cluster network 'k3d-k3s-default'   
INFO[0005] Deleting image volume 'k3d-k3s-default-images' 
FATA[0005] Cluster creation FAILED, all changes have been rolled back! 
Error: Process completed with exit code 1.

Have you got an idea why this error appear? It looks like one of option:

1. github actions doesn't clean containers between jobs
2. k3d creates doubled container

[kuritka]

Hi @arkadius ,
could you run with --verbose argument ?

[arkadius]

sure, I will try. It take a while to reproduce it but I will back with verbose results when it appear again

[kuritka]

@arkadius thanks. In addition, I highly recommend using k3d configurations, see: https://github.com/k8gb-io/k8gb/blob/master/k3d/test-gslb1.yaml

and for usage see: https://github.com/k8gb-io/k8gb/blob/master/.github/workflows/terratest.yaml#L45-L49

Also @v2 picks the latest from v2.x.y so you don't have to keep upgrading your workflow when new versions of k3d-action come out.

[arkadius]

Here https://github.com/TouK/nussknacker/runs/4737639087?check_suite_focus=true is available problem with full trace logging.
I see double logs of creating k3d-tools:
lines 105-112:

INFO[0000] Starting new tools node...                   
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0000] [Docker] DockerHost: '' ()                   
DEBU[0000] DOCKER_SOCK=/var/run/docker.sock  
TRAC[0000] Creating node from spec
&{Name:k3d-k3s-default-tools Role:noRole Image:docker.io/rancher/k3d-tools:5.2.2 Volumes:[k3d-k3s-default-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.version:v5.2.2] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[host.k3d.internal:host-gateway] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 

and lines 134-140:

INFO[0005] Using the k3d-tools node to gather environment information 
INFO[0005] Starting new tools node...                   
DEBU[0005] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0005] DOCKER_SOCK=/var/run/docker.sock             
DEBU[0005] DOCKER_SOCK=/var/run/docker.sock             
TRAC[0005] Creating node from spec
&{Name:k3d-k3s-default-tools Role:noRole Image:docker.io/rancher/k3d-tools:5.2.2 Volumes:[k3d-k3s-default-images:/k3d/images /var/run/docker.sock:/var/run/docker.sock] Env:[] Cmd:[] Args:[noop] Ports:map[] Restart:false Created: RuntimeLabels:map[app:k3d k3d.cluster:k3s-default k3d.version:v5.2.2] K3sNodeLabels:map[] Networks:[k3d-k3s-default] ExtraHosts:[host.k3d.internal:host-gateway] ServerOpts:{IsInit:false KubeAPI:<nil>} AgentOpts:{} GPURequest: Memory: State:{Running:false Status: Started:} IP:{IP:zero IP Static:false} HookActions:[]} 

Any idea why it happens?

[arkadius]

I've switched to k3d configuration:

apiVersion: k3d.io/v1alpha3
kind: Simple
image: docker.io/rancher/k3s:v1.23.1-k3s1
servers: 1
agents: 1
options:
  k3d:
    wait: true
    timeout: "60s"
    disableLoadbalancer: true
    disableRollback: true
  k3s:
    extraArgs:
      - arg: --no-deploy=traefik,servicelb,metrics-server
        nodeFilters:
          - server:*
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

But problem still appear. Also I've created an issue in k3d: k3d-io/k3d#924

I've picked @v2 but github actions still uses k3d 5.1.0 instead of 5.2.2: https://github.com/TouK/nussknacker/runs/4779179444?check_suite_focus=true
Any idea why it happens?

[kuritka]

@arkadius, We have not encountered a similar error yet although we use k3d action very intensively. You say it crashes time-to-time, which makes finding the bug more difficult.

I see that you have a rather large pipeline where you install many prerequisites (which might be an issue). k3d is initialized differently in your env than in ours after comparing raw logs: [1], [2]. Would it be possible to move the creation of the k3d-cluster ideally to the top of the pipeline? At least it would be easier in terms of error detection.

The reported error message really sounds like the docker container persist. Would it be possible to extend your pipeline with inspecting the docker network ? see usage: https://github.com/AbsaOSS/k3d-action/blob/main/.github/workflows/single-cluster.yaml#L31 (ideally make it be listed in a pre/post action and handle corner cases if any..).

Regarding usage latest release on @v2, would be fixed.

thx for inputs.

[arkadius]

It looks like workaround described by @iwilltry42 in comment: k3d-io/k3d#924 (comment) works so i guess that we can close this issue. Thank you for support