Locking fixes for versions of Dashmap v5.1.0 and beyond

[carllin]

Problem

Bumping dashmap version to v5.1.0 caused validator with secondary indexes to stall, and thus had to be reverted: #23592

Dashmap v5.1.0 switched over to using parking_lot RwLocks. Because these locks can be write-prioritized, this means that behavior like:

Thread 1:  Grabs and holds Read(A) lock, tries to grab Read(A) later while holding original Read(A)
Thread 2: Grabs Write(A)

can deadlock if Thread 2 grabs the write lock between the first and second attempts to grab Read(A) by Thread 1.

Summary of Changes

The attempt to grab read lock here:

solana/runtime/src/secondary_index.rs

Lines 132 to 137 in 857576d

	let outer_keys = self.reverse_index.get(inner_key).unwrap_or_else(|| {
	self.reverse_index
	.entry(*inner_key)
	.or_insert(RwLock::new(Vec::with_capacity(1)))
	.downgrade()
	});

, then later here

solana/runtime/src/secondary_index.rs

Line 150 in 857576d

("num_secondary_keys", self.index.len() as i64, i64),

is an example of such a scenario where the deadlock described above can occur. We fix this by ensuring the first lock drops out of scope.

TODO: Check use cases across the rest of the code base

Fixes #

[t-nelson]

change here lgtm. are you planning to examine/address other DashMap usage in this PR or separately?

[carllin]

are you planning to examine/address other DashMap usage in this PR

Think I'll do it in this PR