Add Vagrantfile #32

robertgzr · 2017-06-14T09:58:49Z

Decide on Ubuntu vs. Fedora
~~Update the README on how to use it~~

alban · 2017-06-14T10:09:59Z

Trying this branch:

==> default: mesg: 
==> default: ttyname failed
==> default: : 
==> default: Inappropriate ioctl for device

Solved with:

diff --git a/Vagrantfile b/Vagrantfile
index 7bb225d..0a52228 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -25,6 +25,10 @@ Vagrant.configure("2") do |config|
       vb.customize ["modifyvm", :id, "--cpus", "1"]
   end
 
+  # Prevent TTY Errors (copied from laravel/homestead: "homestead.rb" file)... By default this is "bash -l".
+  # https://stackoverflow.com/questions/40815349/vagrant-ubuntu-box-inline-scripts-emitting-mesg-ttyname-failed-inappropriate-io/42397127#42397127
+  config.ssh.shell = "bash -c 'BASH_ENV=/etc/profile exec bash'"
+
   config.vm.provision "shell", inline: <<HERE
 profile=/home/ubuntu/.bash_profile
 if [ ! -e $profile ]; then touch $profile; fi

After that, the VM starts fine for me.

alban · 2017-06-14T10:45:59Z

Vagrantfile

+  # config.vbguest.auto_update = true
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder ".", "/go/src/github.com/kinvolk/kubeadm-nspawn", create: true


It does not build in the VM:

ubuntu@ubuntu-zesty:/go/src/github.com/kinvolk/kubeadm-nspawn$ make go build -o cni-noop ./cmd/cni-noop go build -o cnispawn ./cmd/cnispawn go build -o nspawn-runc ./cmd/nspawn-runc go build -o kubeadm-nspawn ./cmd/kubeadm-nspawn pkg/nspawntool/node.go:26:2: cannot find package "k8s.io/kubernetes/pkg/util/procfs" in any of: /go/src/github.com/kinvolk/kubeadm-nspawn/vendor/k8s.io/kubernetes/pkg/util/procfs (vendor tree) /usr/lib/go-1.7/src/k8s.io/kubernetes/pkg/util/procfs (from $GOROOT) /go/src/k8s.io/kubernetes/pkg/util/procfs (from $GOPATH) Makefile:5: recipe for target 'all' failed make: *** [all] Error 1

Maybe add more synced_folder for the dependencies?

Or vendor it

Looks like glide didn't run... Maybe still something with the Makefile

I have not tried make vendor before.

When trying:

$ make vendor package github.com/Masterminds/glide: mkdir /go/src/github.com/Masterminds/: permission denied Makefile:15: recipe for target 'glide' failed make: *** [glide] Error 1

What about preparing the GOPATH in /home/ubuntu/go instead of /go, and then make sure it belongs to the correct user?

alban · 2017-06-14T15:24:24Z

The user "ubuntu" should be added in the group "docker". Otherwise, the build of Kubernetes in the vagrant VM will fail:

ubuntu@ubuntu-zesty:/go/src/k8s.io/kubernetes$ build/run.sh make
+++ [0614 15:22:10] Verifying Prerequisites....
Can't connect to 'docker' daemon.  please fix and retry.

Possible causes:
  - Docker Daemon not started
    - Linux: confirm via your init system
    - macOS w/ docker-machine: run `docker-machine ls` and `docker-machine start <name>`
    - macOS w/ Docker for Mac: Check the menu bar and start the Docker application
  - DOCKER_HOST hasn't been set or is set incorrectly
    - Linux: domain socket is used, DOCKER_* should be unset. In Bash run `unset ${!DOCKER_*}`
    - macOS w/ docker-machine: run `eval "$(docker-machine env <name>)"`
    - macOS w/ Docker for Mac: domain socket is used, DOCKER_* should be unset. In Bash run `unset ${!DOCKER_*}`
  - Other things to check:
    - Linux: User isn't in 'docker' group.  Add and relogin.
      - Something like 'sudo usermod -a -G docker ${USER}'
      - RHEL7 bug and workaround: https://bugzilla.redhat.com/show_bug.cgi?id=1119282#c8
!!! Error in build/run.sh:28
  Error in build/run.sh:28. 'return 1' exited with status 1
Call stack:
  1: build/run.sh:28 main(...)
Exiting with status 1

alban · 2017-06-14T15:25:56Z

I have tried with passing the full GOPATH directory to the vagrant VM:

diff --git a/Vagrantfile b/Vagrantfile
index 7bb225d..0da8b65 100644
--- a/Vagrantfile
+++ b/Vagrantfile
@@ -2,7 +2,6 @@
 # vi: set ft=ruby :
 
 ENV["TERM"] = "xterm-256color"
-ENV["GOPATH"] = "/go"
 ENV["LC_ALL"] = "en_US.UTF-8"
 
 Vagrant.configure("2") do |config|
@@ -13,8 +12,7 @@ Vagrant.configure("2") do |config|
 
   # config.vbguest.auto_update = true
 
-  config.vm.synced_folder ".", "/vagrant", disabled: true
-  config.vm.synced_folder ".", "/go/src/github.com/kinvolk/kubeadm-nspawn", create: true
+  config.vm.synced_folder "#{ENV['GOPATH']}", "/go", create: true
   # config.vm.provider :virtualbox do |vbox|
   #     vbox.check_guest_additions = false
   #     vbox.functional_vboxsf = false

alban · 2017-06-14T15:57:56Z

Vagrantfile

+  # config.vm.box = "fedora/25-cloud-base"
+  # config.vm.provision "shell", inline: "dnf install -y go git docker"
+  config.vm.box = "ubuntu/zesty64"
+  config.vm.provision "shell", inline: "DEBIAN_FRONTEND=noninteractive apt-get install -y golang git docker.io systemd-container tmux"


Could add one more line after installing docker:

config.vm.provision "shell", inline: "usermod -aG docker ubuntu"

Could we maybe use the non-docker build?

I didn't know there was a non-docker build. But that line with usermod works for me ;)

Added the line.

alban · 2017-06-14T16:14:15Z

Vagrantfile

+  # end
+
+  config.vm.provider :virtualbox do |vb|
+      vb.customize ["modifyvm", :id, "--memory", "1024"]


Apparently, not enough to build Kubernetes:

Env for linux/amd64: GOOS=linux GOARCH=amd64 GOROOT=/usr/local/go_k8s_patched CGO_ENABLED= CC= go build k8s.io/kubernetes/cmd/kubelet/app: /usr/local/go_k8s_patched/pkg/tool/linux_amd64/compile: signal: killed # k8s.io/kubernetes/cmd/kube-controller-manager/app fatal error: runtime: out of memory

I am trying with 4096. It takes some time.

Still fails with the same error with 4094.

Actually, the error was:

Env for linux/amd64: GOOS=linux GOARCH=amd64 GOROOT=/usr/local/go_k8s_patched CGO_ENABLED= CC= # k8s.io/kubernetes/cmd/genman /usr/local/go_k8s_patched/pkg/tool/linux_amd64/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory !!! [0614 16:33:34] Call tree: !!! [0614 16:33:34] 1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:715 kube::golang::build_binaries_for_platform(...) !!! [0614 16:33:34] 2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:530 Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:530. '((i<4-1))' exited with status 2 Call stack: 1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:530 kube::golang::build_binaries_for_platform(...) 2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:715 kube::golang::build_binaries(...) 3: hack/make-rules/build.sh:27 main(...) Exiting with status 1 !!! [0614 16:33:34] Call tree: !!! [0614 16:33:34] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:711 Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:711. '((i<3-1))' exited with status 1 Call stack: 1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:711 kube::golang::build_binaries(...) 2: hack/make-rules/build.sh:27 main(...) Exiting with status 1 !!! [0614 16:33:34] Call tree: !!! [0614 16:33:34] 1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...) !!! Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:620 Error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:620. '((i<3-1))' exited with status 1 Call stack: 1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:620 kube::golang::build_binaries(...) 2: hack/make-rules/build.sh:27 main(...) Exiting with status 1 Makefile:89: recipe for target 'all' failed make: *** [all] Error 1 !!! [0614 16:33:35] Call tree: !!! [0614 16:33:35] 1: build/../build/common.sh:513 kube::build::run_build_command_ex(...) !!! [0614 16:33:35] 2: build/run.sh:30 kube::build::run_build_command(...) !!! Error in build/../build/common.sh:578 Error in build/../build/common.sh:578. '((i<4-1))' exited with status 2 Call stack: 1: build/../build/common.sh:578 kube::build::run_build_command_ex(...) 2: build/../build/common.sh:513 kube::build::run_build_command(...) 3: build/run.sh:30 main(...) Exiting with status 1

...

With the fork/exec error, I wonder if it is about pids limit in the pids cgroup controller. But it does not seem to be the case:

$ sudo cat /sys/fs/cgroup/pids//docker/d23fd83a4c92fa1854f111cb77d39c70496589d0d8c3a115bba292b9bacffc57/pids.max max

any ideas?

I am not sure what to do here...

I don't know either... I would use 4096 and investigate if the error ever happens again.

robertgzr · 2017-06-21T14:39:15Z

Vagrantfile

+ENV["LC_ALL"] = "en_US.UTF-8"
+
+Vagrant.configure("2") do |config|
+  # config.vm.box = "fedora/25-cloud-base"


Can't use fedora25 because machinectl doesn't work with systemd <=232
systemd/systemd#3996

dongsupark · 2017-07-10T13:09:16Z

AFAICT most of the comments for the PR are already out-of-date, because the whole Vagrantfile has been rewritten for Fedora 26 instead of Ubuntu.
The only remaining valid issue here is about the out-of-memory failure despite of 4GiB memory. I have not seen such an error, and I'm not sure either how to prevent it.
So I'd say, let's merge this PR for now, especially as other pending fixes depend on the changes in Vagrantfile.

/cc @robertgzr @alban

robertgzr · 2017-07-10T13:52:05Z

@dongsupark If I remember correctly the memory issue came up when building kubernetes (which is what we did originally)

I am not sure how to do that, if we need to be able to do this in the vagrant box?

dongsupark · 2017-07-11T08:37:46Z

@robertgzr Yeah, I already have done several complete k8s rebuilds in the Vagrant box, but haven't seen such a failure. Maybe more tight memory constraints are necessary for reproducing it.
Anyway as discussed with @blixtra, I'm going to merge this PR.

robertgzr self-assigned this Jun 14, 2017

robertgzr requested review from alban and schu June 14, 2017 09:58

robertgzr referenced this pull request Jun 14, 2017

Add Vagrantfile

ab3eec2

alban reviewed Jun 14, 2017

View reviewed changes

robertgzr mentioned this pull request Jun 15, 2017

Vagrantfile for testing / development #30

Closed

robertgzr force-pushed the robertgzr/vagrant branch 3 times, most recently from 2544d00 to b3562f7 Compare June 20, 2017 10:00

Add Vagrantfile

db11d75

robertgzr force-pushed the robertgzr/vagrant branch from b3562f7 to db11d75 Compare June 20, 2017 10:30

robertgzr commented Jun 21, 2017

View reviewed changes

Use a Fedora box

5bdde2e

dongsupark merged commit 67eb477 into master Jul 11, 2017

robertgzr mentioned this pull request Jul 11, 2017

docker/runc doesn't work on Fedora 26 #45

Closed

robertgzr deleted the robertgzr/vagrant branch July 11, 2017 15:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Vagrantfile #32

Add Vagrantfile #32

robertgzr commented Jun 14, 2017 •

edited

Loading

alban commented Jun 14, 2017

alban Jun 14, 2017

alban Jun 14, 2017

robertgzr Jun 14, 2017

alban Jun 14, 2017

alban commented Jun 14, 2017

alban commented Jun 14, 2017

alban Jun 14, 2017

robertgzr Jun 15, 2017

alban Jun 15, 2017

robertgzr Jun 20, 2017

alban Jun 14, 2017

alban Jun 14, 2017

alban Jun 15, 2017

alban Jun 15, 2017 •

edited

Loading

robertgzr Jun 20, 2017

alban Jun 20, 2017

robertgzr Jun 21, 2017

dongsupark commented Jul 10, 2017

robertgzr commented Jul 10, 2017

dongsupark commented Jul 11, 2017

Add Vagrantfile #32

Add Vagrantfile #32

Conversation

robertgzr commented Jun 14, 2017 • edited Loading

alban commented Jun 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alban commented Jun 14, 2017

alban commented Jun 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alban Jun 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongsupark commented Jul 10, 2017

robertgzr commented Jul 10, 2017

dongsupark commented Jul 11, 2017

robertgzr commented Jun 14, 2017 •

edited

Loading

alban Jun 15, 2017 •

edited

Loading