Switch to the rkt iso by default #571

dlorenc · 2016-09-12T13:30:23Z

philips · 2016-09-14T23:28:50Z

@dlorenc is the plan to move the code into this tree?

dlorenc · 2016-09-16T16:24:35Z

Yeah, if @s-urbaniak agrees.

s-urbaniak · 2016-09-16T18:23:59Z

@dlorenc @philips sure thing, I agree, we should fix the OSX issues first though. I'll try to grab a Macbook during the weekend (Linux-only household here).

ryanj · 2016-09-16T23:46:40Z

@s-urbaniak have info on how to reproduce the OSX issues?
I was able to run a quick test of your v0.0.3 image using minikube 0.10. I didn't notice any major issues while testing.

philips · 2016-09-17T02:03:18Z

@ryanj on virtualbox or vmware fusion?

philips · 2016-09-17T02:31:53Z

OK, so a few things:

Something about minikube causes vmwarefusion to crash if the box doesn't come up perfectly. Doing open $HOME/.minikube/machines/minikube/minikube.vmx returns an error. Reboot required to fix.
Unpacking initramfs took 100 seconds (screenshot below)
The machine doesn't seem to have a network interface

@s-urbaniak so, I think you need to add some network drivers first to get this working

philips · 2016-09-17T02:37:29Z

@s-urbaniak I need vmxnet3 please as a kernel driver!

philips · 2016-09-17T02:39:13Z

s-urbaniak · 2016-09-17T08:38:57Z

@philips I added the driver in the minikube iso. Do you mind trying it out with https://github.com/coreos/minikube-iso/releases/download/v0.0.4/minikube-v0.0.4.iso?

aecolley · 2016-09-17T15:35:56Z

On OS X, using the new rkt v0.0.3 iso, docker run -v mounts no longer work. That is, they no longer mount directories from the OS X host into the docker container. The default iso manages this sorcery OK, and it would be nice if it was supported in the rkt iso too.

philips · 2016-09-17T21:21:48Z

@aecolley can you give us an example?

ryanj · 2016-09-17T21:22:09Z

the iso booted cleanly for me using Virtualbox 5.1 on OSX 10.10.2 and Virtualbox 5.0 on Fedora24. I didn't test the other providers, thanks for the additional info

I successfully deployed an image from dockerhub (didn't test volume mounts), and was able to view the running container in the output of rkt list after connecting to the node via minikube ssh.

philips · 2016-09-17T21:23:20Z

@aecolley can you give details on your OS, hypervisor, and version of minikube along with the example?

aecolley · 2016-09-17T22:32:03Z

@philips Certainly. Here's my test case script:

#!/bin/bash
minikube stop || :
minikube delete || :
rm -fr mountable || :

set -o errexit
minikube start "$@"
dvm use 1.12.1
eval "$(minikube docker-env)"
mkdir mountable
touch mountable/proof
output="$(docker run --rm -v $PWD/mountable:/mounted:ro busybox:latest /bin/ls -asCF /mounted)"
printf 'Output: %s\n' "${output}"
if [[ "${output}" == *proof* ]]; then
    echo PASS
    exit 0
else
    echo FAIL
    exit 1
fi

And here's an unedited copy-paste of the Terminal session in which I show it failing with the rkt iso and passing with the default setup. At the end, it shows the OS/virtualbox/minikube/docker versions as requested.

MacBook-Pro-2:~ aecolley$ tmp/testcase.sh --network-plugin=cni --container-runtime=rkt --iso-url=https://github.com/coreos/minikube-iso/releases/download/v0.0.3/minikube-v0.0.3.iso
Stopping local Kubernetes cluster...
Machine stopped.
Deleting local Kubernetes cluster...
Machine deleted.
Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.
Now using Docker 1.12.1
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
8ddc19f16526: Pulling fs layer
8ddc19f16526: Verifying Checksum
8ddc19f16526: Download complete
8ddc19f16526: Pull complete
Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
Status: Downloaded newer image for busybox:latest
Output: total 4
     0 ./     4 ../
FAIL
MacBook-Pro-2:~ aecolley$ tmp/testcase.sh
Stopping local Kubernetes cluster...
Machine stopped.
Deleting local Kubernetes cluster...
Machine deleted.
Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.
Now using Docker 1.12.1
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
8ddc19f16526: Pulling fs layer
8ddc19f16526: Verifying Checksum
8ddc19f16526: Download complete
8ddc19f16526: Pull complete
Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
Status: Downloaded newer image for busybox:latest
Output: total 4
     0 ./        4 ../       0 proof
PASS
MacBook-Pro-2:~ aecolley$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1004
MacBook-Pro-2:~ aecolley$ docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 31
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Security Options:
Kernel Version: 4.4.14-boot2docker
Operating System: Boot2Docker 1.11.1 (TCL 7.1); master : 901340f - Fri Jul  1 22:52:19 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.9 MiB
Name: minikube
ID: OXA2:LCBA:ACGN:HZZU:AUKE:YX3U:QCXI:B4WI:CLQH:5PC3:RFQT:KWNT
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 28
 Goroutines: 57
 System Time: 2016-09-17T22:16:05.318013662Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8
MacBook-Pro-2:~ aecolley$ minikube version
minikube version: v0.10.0
MacBook-Pro-2:~ aecolley$ VBoxManage --version
5.1.6r110634

The version numbers are the same in both cases, but the other parts of the docker info output is different. For completeness, here's the output of docker info with the rkt iso loaded:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Security Options:
Kernel Version: 4.7.2
Operating System: Buildroot 2016.08-rc3
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.4 MiB
Name: minikube
ID: U3OT:RIAO:ZX5D:RKBD:HC3J:O5TF:WNRA:CCKH:P7NK:FBQW:IFBI:P5R5
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

Finally, it also fails when using the docker runtime with the rkt iso (i.e. with testcase.sh --container-runtime=docker --iso-url=https://github.com/coreos/minikube-iso/releases/download/v0.0.3/minikube-v0.0.3.iso), so I believe it's a problem with the iso and not with the rkt support.

philips · 2016-09-18T19:42:32Z

@s-urbaniak put up a new version of the ISO which WFM! https://github.com/coreos/minikube-iso/releases/download/v0.0.4/minikube-v0.0.4.iso Confirmed working with minikube v0.10.0 and OSX and vmware fusion. Thanks @s-urbaniak

philips · 2016-09-28T22:01:48Z

@s-urbaniak what is the next step to move to this ISO by default?

s-urbaniak · 2016-09-29T06:52:11Z

@philips I can prepare a PR against minikube in the next week, I am still working on xhyve support though: https://github.com/coreos/minikube-iso/issues/17

@dlorenc Do you see xhyve support as a blocker right now or do you think I can proceed with a "boot2docker to minikube-iso" PR?

philips · 2016-10-06T03:00:51Z

I personally don't see xhyve as a blocker as it is still really hard to get that working. We need to figure out how to bundle it instead of relying on the libmachine external driver, etc.

dlorenc · 2016-10-07T01:24:29Z

I suppose we could make this iso default automatically for people passing the container-runtime=rkt flag, but there are quite a few xhyve users that I'd rather not break.

I think the xhyve issues are fixed now though, I'll do some final testing tomorrow.

@s-urbaniak do you want to send a PR to start moving the ISO build scripts from your repo here?

philips · 2016-10-07T13:36:29Z

@dlorenc @s-urbaniak mentioned he got xhyve working yesterday after I commented.

s-urbaniak · 2016-10-07T14:24:43Z

@dlorenc working on the changes to pull the ISO build scripts here right now. Not sure if I'll make it today though.

xhyve basically works according to https://github.com/coreos/minikube-iso/issues/17#issuecomment-250664776, @zchee is trying to push this upstream via moby/hyperkit#63.

zchee · 2016-10-07T16:03:29Z

@dlorenc @philips @s-urbaniak
As in the moby/hyperkit#63, I still haven't understood the method of calculating the ramdisk_start value.
But I was some debugging, It seems to be able to boot any ISO if about twice to ramdisk_start. (well, it also would be incorrect)

If there is no response from docker-dev until merge machine-drivers/docker-machine-driver-xhyve#138 ~~waiting for @dlorenc check~~ done), I was thinking to temporary fix of kexec.c for rkt.

FYI, I'm already using the zchee/libhyperkit that the static vendoring hyperkit and other patched(cgo problem, dispatch_loop(Apple GCD), lib9p server), so we do not need to wait for issue response.
We can support rkt immediately.

How you do you feel about it?

s-urbaniak · 2016-10-07T18:24:50Z

@zchee I'd say go with it :-)

dlorenc · 2016-10-07T18:26:08Z

+1, that sounds great to me.

zchee · 2016-10-08T01:36:16Z

@s-urbaniak @dlorenc @philips Done :)
Please check and test for it.
machine-drivers/docker-machine-driver-xhyve#140

I was tested and successful with

minikube start --show-libmachine-logs=true \
               --v=7 \
               --logtostderr=true \
               --vm-driver="xhyve" \
               --memory=4096 \
               --cpus=2 \
               --network-plugin=cni \
               --container-runtime=rkt \
               --iso-url=https://github.com/coreos/minikube-iso/releases/download/xhyve/minikube-xhyve.iso

If you want to pre-compile, Here is lib9p qcow2 static linked binary. build by my local env.
http://get.zchee.io/bin/docker-machine-driver-xhyve

s-urbaniak · 2016-10-25T05:37:18Z

I suggest to reopen this, as we have the scripts now available to generate the ISO, but inside minikube we are still pointing to the boot2docker based iso image as per [1].

@dlorenc @r2d4 I don't see pointers in the source on how the upload is happening on https://storage.googleapis.com/minikube. Please let me know if there is anything on how I can help to make this happen, so we can shut down https://github.com/coreos/minikube-iso/releases :-)

[1]

minikube/pkg/minikube/constants/constants.go

Line 62 in 3a61ad6

    
           DefaultIsoUrl       = "https://storage.googleapis.com/minikube/minikube-0.7.iso"

r2d4 · 2016-10-25T18:00:35Z

I reopened this, I think it got auto-closed from the PR. Right now, we are just pushing the iso with gsutil https://github.com/kubernetes/minikube/tree/master/deploy/iso/boot2docker

I can push the previous releases of coreos/minikube-iso to our storage bucket. I'll let you know when its done.

Another thing we might need to figure are names for the ISOs - the b2d one is currently called minikube.iso and this one is minikube-iso.iso. Until we make this one the default, we should probably rename it something else, possibly minikube-alpha.iso? Any suggestions @dlorenc or @s-urbaniak?

s-urbaniak · 2016-10-26T11:58:23Z

minikube-alpha.iso sgtm for the time being.

dlorenc · 2016-11-03T18:05:45Z

I think we're basically ready to make this default. Any concerns about doing this in 0.13?

dlorenc mentioned this issue Sep 12, 2016

README: add rkt instructions #570

Merged

r2d4 added the kind/feature Categorizes issue or PR as related to a new feature. label Sep 15, 2016

r2d4 added the runtime/rkt label Sep 19, 2016

r2d4 mentioned this issue Sep 22, 2016

Support CoreOS as the ISO image #187

Closed

s-urbaniak mentioned this issue Oct 13, 2016

add buildroot based minikube-iso build infrastructure #692

Merged

dlorenc mentioned this issue Oct 17, 2016

localkube.err should not fill up entire disk #700

Closed

dlorenc closed this as completed in #692 Oct 18, 2016

r2d4 reopened this Oct 25, 2016

r2d4 added the iso/minikube-iso label Oct 25, 2016

aecolley mentioned this issue Oct 25, 2016

Cut v0.12.0 release #739

Merged

This was referenced Oct 28, 2016

Add docker-env options to the buildroot provisioner. #765

Merged

Add an integration run with the new ISO. #761

Merged

r2d4 mentioned this issue Nov 11, 2016

Bump minikube iso to v1.0.1, the buildroot based ISO #820

Merged

r2d4 closed this as completed in #820 Dec 20, 2016

afbjorklund mentioned this issue Mar 19, 2019

Remove rkt container runtime #3906

Closed

afbjorklund mentioned this issue May 8, 2020

Upgrade Buildroot to 2020.02 LTS #7904

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to the rkt iso by default #571

Switch to the rkt iso by default #571

dlorenc commented Sep 12, 2016

philips commented Sep 14, 2016

dlorenc commented Sep 16, 2016

s-urbaniak commented Sep 16, 2016

ryanj commented Sep 16, 2016

philips commented Sep 17, 2016

philips commented Sep 17, 2016

philips commented Sep 17, 2016

philips commented Sep 17, 2016

s-urbaniak commented Sep 17, 2016

aecolley commented Sep 17, 2016

philips commented Sep 17, 2016

ryanj commented Sep 17, 2016 •

edited

Loading

philips commented Sep 17, 2016

aecolley commented Sep 17, 2016

philips commented Sep 18, 2016

philips commented Sep 28, 2016

s-urbaniak commented Sep 29, 2016

philips commented Oct 6, 2016

dlorenc commented Oct 7, 2016

philips commented Oct 7, 2016

s-urbaniak commented Oct 7, 2016

zchee commented Oct 7, 2016 •

edited

Loading

s-urbaniak commented Oct 7, 2016

dlorenc commented Oct 7, 2016

zchee commented Oct 8, 2016

s-urbaniak commented Oct 25, 2016

r2d4 commented Oct 25, 2016

s-urbaniak commented Oct 26, 2016

dlorenc commented Nov 3, 2016

Switch to the rkt iso by default #571

Switch to the rkt iso by default #571

Comments

dlorenc commented Sep 12, 2016

philips commented Sep 14, 2016

dlorenc commented Sep 16, 2016

s-urbaniak commented Sep 16, 2016

ryanj commented Sep 16, 2016

philips commented Sep 17, 2016

philips commented Sep 17, 2016

philips commented Sep 17, 2016

philips commented Sep 17, 2016

s-urbaniak commented Sep 17, 2016

aecolley commented Sep 17, 2016

philips commented Sep 17, 2016

ryanj commented Sep 17, 2016 • edited Loading

philips commented Sep 17, 2016

aecolley commented Sep 17, 2016

philips commented Sep 18, 2016

philips commented Sep 28, 2016

s-urbaniak commented Sep 29, 2016

philips commented Oct 6, 2016

dlorenc commented Oct 7, 2016

philips commented Oct 7, 2016

s-urbaniak commented Oct 7, 2016

zchee commented Oct 7, 2016 • edited Loading

s-urbaniak commented Oct 7, 2016

dlorenc commented Oct 7, 2016

zchee commented Oct 8, 2016

s-urbaniak commented Oct 25, 2016

r2d4 commented Oct 25, 2016

s-urbaniak commented Oct 26, 2016

dlorenc commented Nov 3, 2016

ryanj commented Sep 17, 2016 •

edited

Loading

zchee commented Oct 7, 2016 •

edited

Loading