The goal is to mimic a eNovance environment in order to improve the continuous integration of specific development and puppet recipe.
The setup is based on virtualbox driven by vagrant and a specific development (machine-nics) which setups the host virtual networking environment.
The environment is composed of:
- a cluster eHaelix;
- a cluster of firewall;
- two machines providing mx and dns services;
- a monitoring machine;
- a puppet master;
- …
| Type | machine | ip | service | done |
|---|---|---|---|---|
| ehaelix | b1 | 172.143.115.36 | openvz | |
| drbd | ||||
| b2 | 172.143.115.37 | openvz | ||
| drbd | ||||
| fw | fw1 | gateway | ||
| iptables | ||||
| heartbeat1.3 | ||||
| pacemaker | ||||
| fw2 | gateway | |||
| iptables | ||||
| heartbeat1.3 | ||||
| pacemaker | ||||
| monitoring | mon_1 | centreon | ||
| nagios | ||||
| graph | ||||
| mon_2 | centreon | |||
| nagios | ||||
| graph | ||||
| mail/dns | dns_mx_1 | dns | ||
| mx | ||||
| dns_mx_2 | dns | |||
| mx | ||||
| backup | backup_1 | backuppc | ||
| backup_2 | backuppc | |||
| outsider | outside_1 | no service |
Eleven machines that create a realist production environment. Network bond and vlan are working inside each vm to mimic even more the production.
Not all machines need to be started for all needs. Only one firewall is required for the machine to get network connectivity. After if you test a puppet/backup recipe, you need only:
- fw1;
- backup_1;
- backup_2;
- puppet;
One machine consume roughly 475M of memory, but two machines do not make 950M. It roughly stay (> 500M) the same with no load on the machine. The maximum memory consumption is yet not know. Try it.
A version above 4.0.0 is required.
For mainstream Linux use the packages available here.
For FreeBSD you can use redports virtualbox.
As a side note you need dkms on the host before installing virtualbox.
apt-get install dkms /etc/init.d/virtualbox setup
You need :
- a version of the ruby interpreter above 1.9
- a version of the RubyGem above 1.8
- a version of bundler above 1.0
It is recommended to use native ruby tool for installation of the gem.
Here’s a example for Debian.
TODO: test http://deb.bearstech.com/squeeze/ruby-1.9.3/
sudo apt-get install ruby1.9.1
sudo apt-get install ruby1.9.1-dev
sudo apt-get install ri1.9
# required to get gem >= 1.8 the 1.3 does not work.
sudo update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby1.9.1 500 \
--slave /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby1.9.1.1.gz \
--slave /usr/share/man/man1/ri.1.gz ri.1.gz /usr/share/man/man1/ri1.9.1.1.gz \
--slave /usr/share/man/man1/rdoc.1.gz rdoc.1.gz /usr/share/man/man1/rdoc1.9.1.1.gz \
--slave /usr/share/man/man1/irb.1.gz irb.1.gz /usr/share/man/man1/irb1.9.1.1.gz \
--slave /usr/bin/ri ri /usr/bin/ri1.9.1 \
--slave /usr/bin/irb irb /usr/bin/irb1.9.1 \
--slave /usr/bin/gem gem /usr/bin/gem1.9.1 \
--slave /usr/bin/rdoc rdoc /usr/bin/rdoc1.9.1
sudo env REALLY_GEM_UPDATE_SYSTEM=1 gem1.9.1 update --system
# must be greater than 1.8
gem --version
sudo gem install bundler -v '~>1.0'
sudo gem install rake -v '~>0.9'
You need a specific version.
cd /usr/src sudo git clone --branch vendor/sathlan git://github.com/sathlan/vagrant.git cd vagrant bundle install rake build sudo gem install pkg/vagrant-*.gem
apt-get install sudo # this step is very optional adduser vagrant # add sudo passwordless right to the user who spawn the vm echo '%vagrant ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers cd /usr/src git clone git://github.com/sathlan/machine-nics.git cd machine-nics bundle install rake build sudo gem install pkg/machine-nics-*.gem
First we need the vagrant configuration and the associated rakefile (= makefile in ruby):
su - vagrant # or whatever user belonging to the
# sudo nopasswd group
cd ~/vps
mkdir ~/cluster_ehaelix
cd ~/cluster_ehaelix
git clone git://github.com/sathlan/cluster.git .
We are done with dependencies. Now we can spawn some vm.
First we need some network on the host.
# list available network configuration rake -T # make one. rake add_enovance_small
Here you’ll have to make sure that everything which goes out is SNATed.
Finally we start the virtual machines. The first time is very long as it has to download the host (3.2G base PXE ehaelix type server) and create an vm out of it.
vagrant up fw1 vagrant up b1 vagrant up puppet
Done, now you can enjoy the environment.
vagrant help vagrant ssh b1
We can have virtualbox snapshot at the command line for free with vagrant-snap.
sudo gem install virtualbox sudo gem install vagrant-snap cd ~/vps/cluster_ehaelix vagrant snap take b1 -n 'before big change' -d "I'm about to break everything."
Vagrant has puppet provisioning integrated. Here, it’s use to setup network connectivity. Another use can be to quickly test an puppet module. Coupled with snapshot, it enable the user to get test without hurting an new module.
The basic setup is found in for each machine in the
manifests/<machine>.pp file.
Setting up all the interface with nicpromis allow-all into virtualbox seems to resolve the issue. Seems related to Ticket VituralBox 4214.
On the host:
VBoxManage list vms for i in 2 3 4 5 ; do VBoxManage modifyvm <uuid> --nicpromisc$i allow-all; done
TODO: can be integrated to vagrant.
The ifenslave is required for the puppet network configuration I made. Unfortunately, it does not come by default with the ehaelix PXE setup.
So :
apt-get update apt-get install ifenslave hostname | reboot
TODO: include it in the default ehaelix image. TODO: make a rule in puppet.
The setup has some glitch under Linux, but in the end everything works fine.
To have the bond0.101 works on the fw1, the bond0 interface must be set promisc mode:
ifconfig bond0 promisc
This seems also related to Ticket VituralBox 4214 and requires further investigation (or a puppet rule)
By default the vagrant link (with 10....) address is still there. It
makes the command vagrant ssh b1 work. But it add a default route.
This must be (manually) removed.
TODO: A puppet rule to make it disappear.