feat: optimizes file copies to and from containers

[codefromthecrypt]

What does this PR do?

This changes code interacting with file copies to and from the container to use optimized functions. Doing so reduces buffering and gives a chance for to use Go 1.22's optimized paths for linux.

Why is it important?

I'm using rather large images in k3s. For example, nodejs images get easily over a GB each. I found this copy logic accounts for the majority of test fixture setup, in our case sometimes over a minute is spent here even when images are available locally.

Related issues

Originally added in #347

How to test this PR

you can use k3s and its LoadImages function which will hit all the paths here.

[netlify]

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	61a8c3d
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/6603aae270d3130008b05817
😎 Deploy Preview	https://deploy-preview-2450--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codefromthecrypt]

I think the failure is a flake?

           	            	-  RyukDisabled: (bool) true,
          	            	+  RyukDisabled: (bool) false,

[mdelapenya]

Hey @codefromthecrypt I've done a super quick and dirty benchmark for this improvement, and I'm sharing here the results.

I'm not doing them to demonstrate anything against the PR, but to learn myself from the process, as I've been thinking about benchmarks more and more in the recent times. So please take this as me doing an exercise to learn, and I'd love to receive feedback if possible.

Here I go!

Env

goos: darwin
goarch: arm64
GOMAXPROCS=8

Code

I created a LoadImagesOld method in the k3s module, which is a copy of the LoadImages one, but calling a copy of the old CopyFileToContainer method (before these changes). I also created a benchmark function to be run in sub-benchmarks: one for the new method, and one for the old one. Each benchmark will load the nginx image into the k3s cluster.

func BenchmarkLoadImages(b *testing.B) {
	// Give up to three minutes to run this test
	ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(3*time.Minute))
	defer cancel()

	k3sContainer, err := k3s.RunContainer(ctx,
		testcontainers.WithImage("docker.io/rancher/k3s:v1.27.1-k3s1"),
	)
	if err != nil {
		b.Fatal(err)
	}

	// Clean up the container
	defer func() {
		if err := k3sContainer.Terminate(ctx); err != nil {
			b.Fatal(err)
		}
	}()

	provider, err := testcontainers.ProviderDocker.GetProvider()
	if err != nil {
		b.Fatal(err)
	}

	// ensure nginx image is available locally
	err = provider.PullImage(ctx, "nginx")
	if err != nil {
		b.Fatal(err)
	}

	b.ResetTimer() // Reset the benchmark timer

	b.Run("Old copy method", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(3*time.Minute))
			defer cancel()
			err := k3sContainer.LoadImagesOld(ctx, "nginx")
			if err != nil {
				b.Fatal(err)
			}
		}
	})

	b.Run("New copy method", func(b *testing.B) {
		for i := 0; i < b.N; i++ {
			ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(3*time.Minute))
			defer cancel()
			err := k3sContainer.LoadImages(ctx, "nginx")
			if err != nil {
				b.Fatal(err)
			}
		}
	})
}

Benchmark execution

Run benchmarks 5 times, including memory profile (bytes and allocations per operation):

go test -bench=. -benchmem -count 5 -run=^#

Benchmarks results

Benchmark	ns/op	B/op	allocs/op
BenchmarkLoadImages/Old_copy_method-8	33,176,551,708	460,569,136	3,702
BenchmarkLoadImages/Old_copy_method-8	33,715,342,958	460,480,384	2,715
BenchmarkLoadImages/Old_copy_method-8	31,288,304,583	460,465,152	2,632
BenchmarkLoadImages/Old_copy_method-8	32,554,835,500	460,480,336	2,716
BenchmarkLoadImages/Old_copy_method-8	31,915,499,583	460,471,904	2,714
BenchmarkLoadImages/New_copy_method-8	33,774,419,125	269,623,592	2,720
BenchmarkLoadImages/New_copy_method-8	31,926,620,417	269,623,272	2,707
BenchmarkLoadImages/New_copy_method-8	31,896,055,375	269,616,432	2,720
BenchmarkLoadImages/New_copy_method-8	33,230,731,916	269,640,440	2,827
BenchmarkLoadImages/New_copy_method-8	33,030,320,250	269,622,944	2,712

Results

With the above numbers, it seems obvious that the Bytes per operation is way lower with the new method (from 460's to 270's). The other two values, ns and allocations per operation seems more or less the same: the code is not much faster nor produce less allocations, but it uses less memory.

[mdelapenya]

We test against both versions of the language, although we always develop in the lowest one. It could be probably interesting working the other way around: always develop in the latest release, and run the tests for both (current and current -1). Thoughts?

[codefromthecrypt]

To me, I think we should test at least 2 versions of Go anyway, and usually things like this don't come up too often.

In some projects I tend to do the develop in latest and test the floor model, just because a lot of devs always use latest first.

[mdelapenya]

LGTM, thanks!

[codefromthecrypt]

Thanks for sharing the results @mdelapenya, and also merging! I'm not too surprised about the benchmarks at the moment as most of the potential is only implemented in linux right now.

It could be neat to run the same in a linux container to see how much difference it makes where the go side is optimized, but not required on my side. Cheers!