kv: dequeue request from lock table wait-queues on scan error

Informs cockroachdb#111352.
Informs cockroachdb#111530.
Informs cockroachdb#111564.
Informs cockroachdb#111893.

In 8205b43, we added a case to the lock table where a request's initial scan
could throw an error. This was not being handled properly if the request had
already entered any other lock wait-queues. In these cases, the request's entries
in those wait-queues would be abandoned and the locks would get stuck. This commit
fixes that issue by dequeuing the request from the lock table when throwing an
error from ScanAndEnqueue.

This was one of the causes of the recent kvnemesis instability, but we believe
that there is at least one other issue that is still causing timeouts.

Release note: None

pkg/kv/kvserver/concurrency/lock_table.go

@@ -3785,6 +3785,9 @@ func (t *lockTableImpl) ScanAndEnqueue(req Request, guard lockTableGuard) (lockT
 
 	err := g.resumeScan(true /* notify */)
 	if err != nil {
+		// We're not returning the guard on this error path, so we need to
+		// release the guard in case it has already entered any wait-queues.
+		t.Dequeue(g)
 		return nil, kvpb.NewError(err)
 	}
 	if g.notRemovableLock != nil {

pkg/kv/kvserver/concurrency/testdata/lock_table/shared_locks

@@ -1137,10 +1137,7 @@ lock promotion from Shared to Exclusive is not allowed
 
 print
 ----
-num=2
- lock: "a"
-   queued locking requests:
-    active: false req: 58, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000002
+num=1
  lock: "b"
   holder: txn: 00000000-0000-0000-0000-000000000002 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 0)]