util/tracing: disable span reuse under deadlock detector

[andreimatei]

In #73883 we started reusing tracing spans through a sync.Pool. This
causes the mutexes in the spans to be acquired by a goroutine
repeatedly, in a random order. Multiple spans can be locked at the same
time, but the ordering well defined (a child can be locked while the
parent is also locked). This erroneously looks like a possible deadlock
to the deadlock detector. The following test demonstrates the problem -
it's flagged by the deadlock detector even though the code is
deadlock-free.

This patch fixes the issue by disabling span reuse under deadlock
builds.

I guess the moral of the story is that recycling structs with locks
through sync.Pool is incompatible with the deadlock detector? One
option, I guess, is to contribute a way to mark some mutexes as excluded
from the deadlock detector.

func TestXXX(t *testing.T) {
	p := sync.Pool{New: func() interface{} {
		return new(syncutil.Mutex)
	},
	}

	var m1, m2 *syncutil.Mutex
	m1 = p.Get().(*syncutil.Mutex)
	m2 = p.Get().(*syncutil.Mutex)

	m1.Lock()
	m2.Lock()
	m2.Unlock()
	m1.Unlock()

	p.Put(m1)
	p.Put(m2)
	m2 = p.Get().(*syncutil.Mutex)
	m1 = p.Get().(*syncutil.Mutex)

	m1.Lock()
	m2.Lock()
	m2.Unlock()
	m1.Unlock()
}

Fixes #75351 and a million others.

Release note: None

[cockroach-teamcity]

This change is 

[RaduBerinde]

Rather than disabling span reuse, we could add a syncutil.ReusableMutex or something like that that is always a simple mutex, even for deadlock builds. We should file an issue for github.com/sasha-s/go-deadlock to provide some kind of Reset function and then we could expose that through ReusableMutex.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @aayushshah15, @erikgrinaker, and @nvanbenschoten)

[andreimatei]

Yeah, I was thinking similar things. But let's do this patch first, I'd say.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @aayushshah15, @erikgrinaker, and @nvanbenschoten)

[andreimatei]

bors r+

[craig]

Build failed:

• GitHub CI (Cockroach)

[andreimatei]

bors r+

[craig]

Build failed:

• GitHub CI (Cockroach)

[andreimatei]

keeps flaking on #68395

bors r+

[craig]

Build failed (retrying...):

• GitHub CI (Cockroach)

[craig]

Build succeeded:

• GitHub CI (Cockroach)