-
Notifications
You must be signed in to change notification settings - Fork 9
Installation: Collectd Collectsphere
The first step is to deploy a system that is going to run Collectd and the Collectsphere plugin. The required resources depend highly on the size of your infrastructure. Collectsphere spawns one thread for every ESXi host and VM to fetch the data via the vSphere API. They spawn all at once utilizing a good amount of CPU power and even more memory. This is necessary in order to have a low execution time for the plugin.
The installation process will be demonstrated on Ubuntu 14.04 LTS:
apt-get install git-core
git clone https://github.com/evoila/collectsphere.git
apt-get install -y python-pip
pip install pysphere
cd collectsphere
cp run.py run-test.py
vim run-test.py
Edit the lines that configure collectsphere to look like this. Later, collectd will contain the configuration. This only for the manual run.
children.append(Conf('Name', ('test',)))
children.append(Conf('Host', ('r1.vxpertise.net',)))
children.append(Conf('Port', ('4430',)))
children.append(Conf('Username', ('root',)))
children.append(Conf('Password', ('*********',)))
Now execute run-test.py:
python run-test.py
The output should look something like this:
collectd [INFO]: configure_callback: Loaded config: name=test, host=r1.vxpertise.net, port=4430, verbose=True, username=root, password=******, host_metrics=5, vm_metrics=4, inventory_refresh_interval=600
collectd [INFO]: create_environment: URL: r1.vxpertise.net:4430
Conf/Init Time: 5.29135704041
collectd [INFO]: InventoryWatchDog: Running inventory refresh ...
collectd [INFO]: InventoryWatchDog: Discovered cluster lab cluster
collectd [INFO]: InventoryWatchDog: Cluster lab cluster never seen before
collectd [INFO]: InventoryWatchDog: Found 1 hosts in cluster lab cluster
collectd [INFO]: InventoryWatchDog: Found 76 registered VMs in cluster lab cluster
collectd [INFO]: InventoryWatchDog: Removed 0 VMs from my inventory cache
collectd [INFO]: InventoryWatchDog: Adding 76 VMs to cache. Spawning threads...
collectd [INFO]: InventoryWatchDog: Spawned 50 threads (0 - 49) to fetch VM objects in cluster lab cluster
collectd [INFO]: read_callback: entering environment: test
collectd [INFO]: read_callback: Inventory is empty. Skipping to next environment
collectd [INFO]: read_callback: Spawned a total of 0 threads to fetch Host and VM metrics.
collectd [INFO]: read_callback: Dispatched a total of 0 values to collectd.
Read Time: 0.00205302238464
collectd [INFO]: InventoryWatchDog: Spawned 26 threads (50 - 75) to fetch VM objects in cluster lab cluster
collectd [INFO]: InventoryWatchDog: All threads returned. Publishing inventory...
collectd [INFO]: InventoryWatchDog: Discovered 1 hosts and 76 VMs in 1 clusters in 56 seconds. Next refresh in 600 seonds.
collectd [INFO]: read_callback: entering environment: test
collectd [INFO]: read_callback: found 1 hosts and 76 VMs in the inventory cache
collectd [INFO]: read_callback: Cluster lab cluster: Spawning 1 threads to fetch host metrics.
collectd [INFO]: read_callback: Cluster lab cluster: Spawning 76 threads to fetch VM metrics.
collectd [INFO]: read_callback: Spawned a total of 77 threads to fetch Host and VM metrics.
host=None, plugin=collectsphere, plugin_instance=None, time=None, type=gauge, type_instance=lab cluster.host.node00.mem.all.usage.perc, values=[5040.0], meta=None
host=None, plugin=collectsphere, plugin_instance=None, time=None, type=gauge, type_instance=lab cluster.host.node00.cpu.all.usage.perc, values=[3134.0], meta=None
host=None, plugin=collectsphere, plugin_instance=None, time=None, type=gauge, type_instance=lab cluster.host.node00.disk.all.usage.KBps, values=[276.0], meta=None
sudo apt-get install -y collectd
vim /etc/collect.d/collectd.conf
If you would like to monitor the system collectd runs on, too, don't make any changes here. Otherwirse, disable all plugins by commenting out (#) the "LoadPlugin" lines and corresponding "Plugin" blocks below.
In order to send collected data to Graphite, enable the "write_graphite" plugin. As we are going to install graphite on the same host it should look something like this:
LoadPlugin write_graphite
<Plugin write_graphite>
<Node "localhost">
Host "localhost"
Port "2003"
Protocol "tcp"
LogSendErrors true
Prefix "collectd."
StoreRates true
AlwaysAppendDS false
EscapeCharacter "."
</Node>
</Plugin>
Go to the collectsphere repository you cloned before and copy the example collectsphere.conf file to /etc/collect.d/collectd.conf.d:
cd collectsphere
cp collectsphere.conf /etc/collectd/collectd.conf.d/
Now edit the file to match your environment:
vim /etc/collectd/collectd.conf.d/collectsphere.conf
<LoadPlugin "python">
Globals true
</LoadPlugin>
<Plugin "python">
ModulePath "/usr/share/collectd/plugins"
Import "collectsphere"
<Module "collectsphere">
Name "vc01"
Host "vc01.lab.invalid"
Port "443"
Verbose "True"
Username "root"
Password "vmware"
Host_Counters "cpu.usage,mem.usage,disk.usage"
VM_Counters "cpu.usage,mem.usage"
</Module>
</Plugin>
The names of metrics listed in Host_Counters and VM_Counters are those used by the vSphere API. An easy way to find possible values is demonstrated here: Counter Names
In the configuration above, we told collectd to load module called "collectsphere" from the directory /usr/share/collectd/plugins/. Now have to copy collectsphere to that directory, or we will get error messages from collectd:
mkdir -p /usr/share/collectd/plugins
cd collectsphere
cp collectsphere.py /usr/share/collectd/plugins/
tail -f /var/log/syslog | grep collectd &
service collectd restart
In /var/log/syslog you should be seeing the same messages you already know from our manual run before. You should see the InventoryWatchDog refreshing its cache. After that lines like
May 9 10:50:42 collectsphere collectd[2124]: read_callback: Cluster lab cluster: Spawning 1 threads to fetch host metrics.
May 9 10:50:42 collectsphere collectd[2124]: read_callback: Cluster lab cluster: Spawning 76 threads to fetch VM metrics.
May 9 10:50:45 collectsphere collectd[2124]: read_callback: Spawned a total of 77 threads to fetch Host and VM metrics.
May 9 10:50:49 collectsphere collectd[2124]: read_callback: Dispatched a total of 81 values to collectd.
confirm the proper functionality of collectsphere.
There are going to be error messages from the write_graphite plugin, which cannot send values to Graphite. Obviously, this is due to Graphite not being there yet.