From 38eed99dfe5965704a515fb252019e550fba668d Mon Sep 17 00:00:00 2001 From: Ivan Shvedunov Date: Wed, 11 Jul 2018 02:41:42 +0300 Subject: [PATCH] Add diagnostics docs --- docs/README.md | 3 +- docs/diagnostics.md | 94 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+), 1 deletion(-) create mode 100644 docs/diagnostics.md diff --git a/docs/README.md b/docs/README.md index 4975df190..5c7f162e1 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,4 +1,4 @@ -This directory holds files containing documentation. +This directory contains the Virtlet documentation. * For a basic example of VM pod definition, see [examples/cirros-vm.yaml](../examples/cirros-vm.yaml) * [Cloud-init data generation](cloud-init-data-generation.md) @@ -11,4 +11,5 @@ This directory holds files containing documentation. * [Environment variables](environment-variables.md) support * [Image Handling](images.md) * [Image Name Translation](image-name-translation.md) +* [Diagnostics](diagnostics.md) * [Update notes](update-notes.md) diff --git a/docs/diagnostics.md b/docs/diagnostics.md new file mode 100644 index 000000000..b51f8651d --- /dev/null +++ b/docs/diagnostics.md @@ -0,0 +1,94 @@ +# Diagnostics + +Virtlet provides a set of +[virtletctl diag](virtletctl/virtletctl_diag.md) commands that can +help with troubleshooting. The diagnostics can be invoked either +directly or buy means of a +[Sonobuoy](https://github.com/heptio/sonobuoy) plugin. + +## Direct invocation + +The most basic diagnostics command is [virtletctl diag dump](virtletctl/virtletctl_diag_dump.md): +``` +$ virtletctl diag out/ +$ ls -lR out +total 0 +drwxr-xr-x 3 user wheel 96 Jul 11 01:56 nodes + +out/nodes: +total 0 +drwxr-xr-x 12 user wheel 384 Jul 11 01:56 kube-node-1 + +out/nodes/kube-node-1: +total 5352 +-rwxr-xr-x 1 user wheel 1276000 Jul 11 01:56 criproxy.log +-rwxr-xr-x 1 user wheel 1787 Jul 11 01:56 ip-a.txt +-rwxr-xr-x 1 user wheel 322 Jul 11 01:56 ip-r.txt +drwxr-xr-x 3 user wheel 96 Jul 11 01:56 libvirt-logs +drwxr-xr-x 5 user wheel 160 Jul 11 01:56 libvirt-xml +-rwxr-xr-x 1 user wheel 9964 Jul 11 01:56 metadata.txt +-rwxr-xr-x 1 user wheel 1443 Jul 11 01:56 netns.txt +-rwxr-xr-x 1 user wheel 18214 Jul 11 01:56 stack.log +-rwxr-xr-x 1 user wheel 64314 Jul 11 01:56 virtlet-pod-libvirt.log +-rwxr-xr-x 1 user wheel 1349763 Jul 11 01:56 virtlet-pod-virtlet.log + +out/nodes/kube-node-1/libvirt-logs: +total 8 +-rwxr-xr-x 1 user wheel 2172 Jul 11 01:56 virtlet-1b2261ca-7ed6-cirros-vm.log + +out/nodes/kube-node-1/libvirt-xml: +total 24 +-rwxr-xr-x 1 user wheel 3511 Jul 11 01:56 domain-virtlet-1b2261ca-7ed6-cirros-vm.xml +-rwxr-xr-x 1 user wheel 445 Jul 11 01:56 pool-volumes.xml +-rwxr-xr-x 1 user wheel 1041 Jul 11 01:56 volume-virtlet_root_1b2261ca-7ed6-58e7-58de-0eef2c9d5320.xml +``` + +The following files and directories are produced for each Kubernetes +node that runs Virtlet: +* `criproxy.log` - the logs of CRI Proxy's systemd unit +* `ip-a.txt` - the output of `ip a` on the node +* `ip-r.txt` - the output of `ip r` on the node +* `metadata.txt` - the contents of Virtlet's internal metadata db in a text form +* `netns.txt` - the output of `ip a` and `ip r` for each network + namespace that's managed by Virtlet +* `stack.log` - the dump of Go stack of Virtlet process +* `virtlet-pod-libvirt.log` - the log of Virtlet pod's libvirt container +* `virtlet-pod-virtlet.log` - the log of Virtlet pod's virtlet container +* `livirt-logs` - a directory with libvirt/QEMU logs for each domain +* `libvirt-xml` - the dumps of all the domains, storage pools and storage volumes in libvirt + +It's also possible to dump Virtlet diagnostics as JSON to stdout using +`virtletctl diag dump --json`. The JSON file can be subsequently +unpacked into the aforementioned directory structure using +[virtletctl diag unpack](virtletctl/virtletctl_diag_unpack.md). + +## Sonobuoy + +Virtlet diagnostics can be run as a +[Sonobuoy](https://github.com/heptio/sonobuoy) plugin. Unfortunately, +right now Sonobuoy's plugin support is somewhat limited, see +heptio/sonobuoy#405 Because of that problem, Sonobuoy run must be +done in two phases, first generating YAML and then using `virtletctl` +to patch it (inject Virtlet sonobuoy plugin): +``` +$ cat sonobuoy.json +{ + "plugins": [ { "name": "virtlet" } ] +} +$ sonobuoy gen --config sonobuoy.json --e2e-focus nosuchtest | + virtletctl diag sonobuoy | + kubectl apply -f - +$ # wait till sonobuoy run is complete +$ sonobuoy status +PLUGIN STATUS COUNT +virtlet complete 1 + +Sonobuoy has completed. Use `sonobuoy retrieve` to get results. +$ sonobuoy retrieve +``` + +The diagnostics results are placed under `plugins/virtlet/results` and +can be unpacked using [virtletctl diag unpack](virtletctl/virtletctl_diag_unpack.md): +``` +$ virtletctl diag unpack out/