Latest release kvm2 driver still linking to newer libvirt

[afbjorklund]

Seems that the driver was not built using the Docker image:

$ wget -q https://github.com/kubernetes/minikube/releases/download/v1.1.1/docker-machine-driver-kvm2
$ ldd docker-machine-driver-kvm2 
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt-lxc.so.0: version `LIBVIRT_LXC_2.0.0' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_2.2.0' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_3.0.0' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_1.3.3' not found (required by ./docker-machine-driver-kvm2)
./docker-machine-driver-kvm2: /usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_2.0.0' not found (required by ./docker-machine-driver-kvm2)

It is supposed to still be using libvirt 1.3.1, for compatibility.

FROM gcr.io/gcp-runtimes/ubuntu_16_0_4

[afbjorklund]

Related to #3689 (regression)

[afbjorklund]

Possibly a race condition, between the make -j16 and the hacky ways of kvm_in_docker

[medyagh]

I wonder if this is the root cause of all the "minikube stop" failing in KVM integration tests? #4495

I think it is important, I would be happy to review the PR fixing this

[josedonizetti]

I'll take a look at this.

[afbjorklund]

This issue affects all versions after v0.30.0:

docker-machine-driver-kvm2-v0.25.0	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.25.1	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.25.2	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.26.0	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.26.1	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.27.0	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.28.0	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.28.1	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.28.2	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.29.0	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.30.0	LIBVIRT_1.2.9
docker-machine-driver-kvm2-v0.31.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v0.32.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v0.33.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v0.33.1	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v0.34.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v0.34.1	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v0.35.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v1.0.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v1.0.1	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v1.1.0	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v1.1.1	LIBVIRT_3.0.0
docker-machine-driver-kvm2-v1.2.0	LIBVIRT_3.0.0

We thought we fixed this in PR #3701 (v1.0.0)

But due to this bug, docker build is never used...

[afbjorklund]

Until this is fixed, please use:

https://github.com/kubernetes/minikube/releases/download/v0.30.0/docker-machine-driver-kvm2

[josedonizetti]

hey, I'm still trying to understand how the release process works, and I have a question about the kvm release.

The way I understood, is that the release happens in Jenkins. It calls the script https://github.com/kubernetes/minikube/blob/master/hack/jenkins/release_build_and_upload.sh, which builds everything with make -j 16 all..., by all here I'm considering it builds minikube and the drivers (KVM, Hyperkit). Then, we have another script that uploads artifacts to Github https://github.com/kubernetes/minikube/blob/master/hack/jenkins/release_github_page.sh.

From this, I'm considering that the drivers are then built by the directive drivers called by make all https://github.com/kubernetes/minikube/blob/master/Makefile#L178, and the drivers directive doesn't use the kvm_in_docker, which was added to help with portability, by building KVM with an older OS.
https://github.com/kubernetes/minikube/blob/master/Makefile#L181
https://github.com/kubernetes/minikube/blob/master/Makefile#L401

The kvm_in_docker is called by make release-kvm-driver https://github.com/kubernetes/minikube/blob/master/Makefile#L426, which is not called in any file inside the minikube repo, at least ack release-kvm-driver returns only its declaration.

Does any of this make sense? If it does, could it be then, that the bug we are discussing here is because the current release isn't using kvm_in_docker build?

[afbjorklund]

Sounds plausible

[afbjorklund]

Should probably add a validation step for the max libvirt version used? So we fail the build, if it happens again for some reason

[afbjorklund]

Seems like the main problem here was that I didn't use BUILD_IN_DOCKER properly, for the kvm target.

So should do that, and then some of the kvm_in_docker will go back to out/docker-machine-driver-kvm2

[afbjorklund]

Verified that it was the -j 16 and the missing BUILD_IN_DOCKER that caused this issue to happen.

The all target builds cross and drivers, and neither call cross-tars or release-kvm-driver