[BUG] Image imports failing

[iwilltry42]

@iwilltry42 I'm seeing this issue on ubuntu when upgrading from k3d 5.3.0 to 5.4.1.

I'm seeing output like:

INFO[0000] Importing image(s) into cluster '$CLUSTER_NAME' 
INFO[0000] Starting new tools node...                   
INFO[0000] Starting Node 'k3d-$CLUSTER_NAME-tools' 
INFO[0000] Saving 3 tarball(s) to shared image volume... 
INFO[0000] Importing images into nodes...               
INFO[0000] Importing images from tarball '/k3d/images/k3d-$CLUSTER_NAME-images-$IMAGE1.tar' into node 'k3d-$CLUSTER_NAME-server-0'... 
INFO[0000] Importing images from tarball '/k3d/images/k3d-$CLUSTER_NAME-images-$IMAGE2.tar' into node 'k3d-$CLUSTER_NAME-server-0'... 
INFO[0000] Importing images from tarball '/k3d/images/k3d-$CLUSTER_NAME-images-$IMAGE3.tar' into node 'k3d-$CLUSTER_NAME-server-0'... 
ERRO[0001] failed to import images in node 'k3d-$CLUSTER_NAME-server-0': Exec process in node 'k3d-$CLUSTER_NAME-server-0' failed with exit code '1' 
ERRO[0001] failed to import images in node 'k3d-$CLUSTER_NAME-server-0': Exec process in node 'k3d-$CLUSTER_NAME-server-0' failed with exit code '1' 
ERRO[0001] failed to import images in node 'k3d-$CLUSTER_NAME-server-0': Exec process in node 'k3d-$CLUSTER_NAME-server-0' failed with exit code '1' 
INFO[0001] Removing the tarball(s) from image volume... 
INFO[0002] Removing k3d-tools node...                   
INFO[0003] Successfully imported image(s)               
INFO[0003] Successfully imported 3 image(s) into 1 cluster(s) 

One thing that jumps out: the code in importWithToolsNode() that logs these failures when copying images swallows errors without returning them to the caller:

k3d/pkg/client/tools.go

Lines 125 to 127 in 852df77

	if err := runtime.CopyToNode(ctx, file, tarName, toolsNode); err != nil {
	l.Log().Errorf("failed to copy image tar '%s' to tools node! Error below:\n%+v", file, err)
	continue

which allows the "successfully imported images" result to get printed.

Compare that to this other k3d code to import images into clusters (rather than nodes) which fails if any individual image fails to import:

k3d/cmd/image/imageImport.go

Lines 75 to 86 in 7b1b416

	errOccurred := false
	for _, cluster := range clusters {
	l.Log().Infof("Importing image(s) into cluster '%s'", cluster.Name)
	if err := client.ImageImportIntoClusterMulti(cmd.Context(), runtimes.SelectedRuntime, images, cluster, loadImageOpts); err != nil {
	l.Log().Errorf("Failed to import image(s) into cluster '%s': %+v", cluster.Name, err)
	errOccurred = true
	}
	}
	if errOccurred {
	l.Log().Warnln("At least one error occured while trying to import the image(s) into the selected cluster(s)")
	os.Exit(1)
	}

Or even this code in the same importWithToolsNode() method that returns errors if any image fails to save:

k3d/pkg/client/tools.go

Lines 114 to 116 in 852df77

	if err := runtime.ExecInNode(ctx, toolsNode, append([]string{"./k3d-tools", "save-image", "-d", tarName}, imagesFromRuntime...)); err != nil {
	return fmt.Errorf("failed to save image(s) in tools container for cluster '%s': %w", cluster.Name, err)
	}

Should importWithToolsNode() return an error whenever any image installations encounter an error? That would at least mean that image installation errors are reported as overall install errors: that doesn't fix the install errors themselves, but it seems more appropriate than treating them as successes.

Originally posted by @CodingCanuck in #954 (comment)

[chivalryq]

Any update on this? we are building a tool based on k3d, we meet this problems too. And BTW, is there any way around to import image in arm64 ?

[yoca94]

Help

[yoca94]

I have the same problem with a mac M1 pro, any updates ?

[manno]

Not sure if related, but we sporadically get this in our Github actions workflows with [email protected].

  k3d image import rancher/fleet:dev rancher/fleet-agent:dev
  shell: /usr/bin/bash -e {0}
  env:
    GOARCH: amd64
    CGO_ENABLED: 0
    SETUP_GO_VERSION: ^1.18
INFO[0000] Importing image(s) into cluster 'k3s-default' 
INFO[0000] Starting new tools node...                   
INFO[0000] Starting Node 'k3d-k3s-default-tools'        
INFO[0001] Saving 2 image(s) from runtime...            
INFO[0001] Importing images into nodes...               
INFO[0001] Importing images from tarball '/k3d/images/k3d-k3s-default-images-20220804121758.tar' into node 'k3d-k3s-default-server-0'... 
INFO[0001] Importing images from tarball '/k3d/images/k3d-k3s-default-images-20220804121758.tar' into node 'k3d-k3s-default-agent-2'... 
INFO[0001] Importing images from tarball '/k3d/images/k3d-k3s-default-images-20220804121758.tar' into node 'k3d-k3s-default-agent-1'... 
INFO[0001] Importing images from tarball '/k3d/images/k3d-k3s-default-images-20220804121758.tar' into node 'k3d-k3s-default-agent-0'... 
ERRO[0002] failed to import images in node 'k3d-k3s-default-agent-2': Exec process in node 'k3d-k3s-default-agent-2' failed with exit code '1' 
ERRO[0002] failed to import images in node 'k3d-k3s-default-agent-1': Exec process in node 'k3d-k3s-default-agent-1' failed with exit code '1' 
ERRO[0002] failed to import images in node 'k3d-k3s-default-server-0': Exec process in node 'k3d-k3s-default-server-0' failed with exit code '1' 
ERRO[0003] failed to import images in node 'k3d-k3s-default-agent-0': Exec process in node 'k3d-k3s-default-agent-0' failed with exit code '1' 
INFO[0003] Removing the tarball(s) from image volume... 
INFO[0004] Removing k3d-tools node...                   
INFO[0005] Successfully imported image(s)               
INFO[0005] Successfully imported 2 image(s) into 1 cluster(s) 

I'd prefer a os.Exit in that case, but it doesn't happen too often.

[atljoseph]

I'm seeing this as well.
Failure, but success?

ERRO[0016] failed to import images in node 'k3d-my-cluster-server-0': Exec process in node 'k3d-my-cluster-server-0' failed with exit code '1' 

And then when I try to run the app, it sees the images is there, but then a kubelet error:

Error: failed to create containerd container: error unpacking image: failed to resolve rootfs: content digest sha256:XYZ: not found

The image does not have that sha in the registry. k3d must have changed it???

[jjngx]

Same here, running on M1. Please consider improving UX by possibly returning error explicitly and stopping execution. Successful messages after failing conditions are misleading. Cheers!

[atljoseph]

It seems that the image gets imported, but with a different hash. So there may have been an initial error, but it seems to have imported something in the end. Perhaps there is a tiered set of instructions to load, and the top tier may have failed. But changing the hash of the image is not acceptable.

…

Sent from my iPhone

On Nov 24, 2022, at 10:04 AM, Jakub Jarosz ***@***.***> wrote:  Same here, running on M1. Please consider improving UX by possibly returning error explicitly and stopping execution. Successful messages after failing conditions are misleading. Cheers! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[a-candiotti-pvotal]

Still happening:

k3d version v5.4.7
k3s version v1.25.6-k3s1 (default)

What's strange too is that it happens when I pull the image from a cloud registry, not when I build the same image in local.