Add tracing test

[jodh-intel]

As shown on kata-containers/runtime#1277, we broke runtime tracing but didn't notice.

Clearly, we need a test to ensure tracing works that does atleast the following:

• Install the Jaeger docker image.
• Enable tracing for all components.
• Run a docker workload.
• Check that Jaeger allows us to query a "reasonable number" of trace spans / component:
  • The minimum number of expected spans is going to differ per component.
  • We'd always expect >0.
  • The actual figures are a little difficult to tie down. We are going to require some magic numbers alas (shudder) since as the code changes, so do the number of trace spans generated.

[jodh-intel]

Hi @chavafg - any thoughts on how best to arrange the pieces here? The plan is to run a basic busybox true docker test initially and check we get some traces so I was thinking of doing something like this:

• Create a script to install Jaeger for trace collection + display (".ci/install_jaeger.sh"?)
• Enable all tracing options in configuration.toml (".ci/configure_tracing_for_kata.sh enable"?)
• Main shell script test (Where should this live?)
  • Run "docker run busybox true"
  • Run the test to ensure Jaeger has collected a number of trace spans.
• Disable all tracing options in configuration.toml (".ci/configure_tracing_for_kata.sh disable"?)

So we'd always install Jaeger for a CI run, but only actually use it for this single test (which would then undo the config to stop tracing). Alternatively, we could create custom Jenkins jobs and run all CI tests with Jaeger, but I think that's overkill tbh.

[jodh-intel]

Related:

• Add opentracing support agent#322
• tracing: Add OpenTracing support agent#415
• rootfs: Support agent tracing osbuilder#200

[chavafg]

Hi @jodh-intel, seems like your approach should work. I'll check this.

[chavafg]

Hi @jodh-intel,
I tried to collect some traces but I got the following issue:
BTW, I am using: kata-containers/runtime#1317

[fuentess@centos-prll kata-containers]$ sudo docker run -ti busybox true
docker: Error response from daemon: OCI runtime create failed: Shim tracing requires disable_new_netns for Jaeger agent communication: unknown.

I went and modify the disable_new_netns = true, then I got:

[fuentess@centos-prll kata-containers]$ sudo docker run -ti busybox true
docker: Error response from daemon: OCI runtime create failed: config disable_new_netns only works with 'none' internetworking_model: unknown.

Modified the internetworking model to none and got the next error:

[fuentess@centos-prll kata-containers]$ sudo docker run -ti busybox true
docker: Error response from daemon: OCI runtime create failed: exit status 1: stdout: , stderr: time="2019-03-04T20:47:29Z" level=fatal msg="failed to add interface veth6d874f2 to sandbox: error renaming interface \"veth6d874f2\" to \"eth0\": file exists": unknown.
[fuentess@centos-prll kata-containers]$ sudo docker run -ti busybox true                                                                                                                                           
docker: Error response from daemon: OCI runtime create failed: exit status 1: stdout: , stderr: time="2019-03-04T20:48:11Z" level=fatal msg="failed to add interface veth0ac47c9 to sandbox: error renaming interface \"veth0ac47c9\" to \"eth0\": file exists": unknown.

Any idea if I am doing something wrong?

[jodh-intel]

Hi @chavafg - oh thanks! I was happy to add this test, but not a problem if you want to ;)

I've just checked and it looks like shim tracing is broken now too ;(( I've raised kata-containers/shim#148 for that.

As such, I'd just enable tracing for the runtime for now to gather some traces. I'll clean up my scripts to check for traces programatically and hand them over later...

[jodh-intel]

Hi @chavafg - here's a basic script that performs some basic Jaeger tests. It might need a bit of refactoring (had to add .png to allow the file to be attached :-)

That's a cut-down version of the script I've got to test agent tracing.

A few other points:

• It should probably be using set -e.
• It assumes debug and tracing is enabled for the runtime.

[chavafg]

Thanks @jodh-intel

[jodh-intel]

@chavafg - I keep forgetting that you need to run docker with --net=none when using the none internetworking model. It's a pita but that information isn't encoded into the config.json so we cannot produce a sensible error message and just fall over when tweaking network stuff. I've just double-checked and shim tracing works fine with that final piece of the puzzle in place ;)

[chavafg]

oh, good to know, thanks @jodh-intel. I'll work on this today.