tracing: improve testing of shadow tracers and fix logging regression

[tbg]

Rearrange the existing test, and test the Zipkin tracer as well. Phase
out the mockTracer; we can just use a zipkin tracer with an in-mem
collector.

Note that the updated tests also verify that if we create a span from a
tracer configured with a shadow tracer (zipkin), that span does indeed
record verbose log messages (cc @andreimatei).

Release justification: testing improvement
Release note: None

Previously (i.e. pre the current cycle's tracing work), a trace that had
a shadow trace attached to it was always regarded as verbose (and there
was no explicit notion of verbosity; the span was recording, which
equalled it being verbose).

Now, verbose spans are those that collect all log messages in the
crdb-internal span. The log package had not been updated to reflect
this, so when a non-verbose span was set up with, say, a zipkin trace,
it was not regarded as worth logging into by log.Event and friends.

This commit fixes this as follows:

• a span that has a shadow span but isn't explicitly set to verbose
  remains non-verbose, but
• the logging framework (ExpensiveLogEnabled) asks the tracer whether
  an auxiliary logging sink is currently set up; if so it does relay all
  log messages to the trace span nevertheless, in effect restoring the
  old behavior.

As a nice improvement, we are now giving log messages only to the
shadow span. It used to be such that the messages were added to the
shadow span and duplicated in the crdb-internal span, for no good
reason. This alone should cut the memory overhead of running with an
external trace collector by around 50%.

Release justification: fixes a recent regression in tracing functionality
Release note: None

[cockroach-teamcity]

This change is 

[knz]

I believe this commit also fixes a bug (with the && tr != nil condition)? If so maybe mention that in the commit message.

Reviewed 4 of 4 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @angelapwen and @irfansharif)

[tbg]

This will need another review, as a substantial commit was added.

[knz]

with nit

also you need a round of make bazel-generate on the 2nd commit

Reviewed 3 of 3 files at r2, 5 of 5 files at r3.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @angelapwen, @irfansharif, and @tbg)

---

pkg/util/log/trace_client_test.go, line 23 at r2 (raw file):

func TestTrace(t *testing.T) {
	ctx := context.Background()

might need a defer leaktest.AfterTest(t)()

---

pkg/util/log/trace_client_test.go, line 58 at r2 (raw file):

func TestTraceWithTags(t *testing.T) {
	ctx := context.Background()

ditto

[tbg]

bors r=knz

[craig]

Build succeeded:

• GitHub CI (Cockroach)