perf: convert m_nodes_mutex to a non-recursive mutex

[PastaPastaPasta]

Issue being fixed or feature implemented

If we are able to get this merged, I'll probably try to upstream this.

m_nodes_mutex is heavily contended; likely more-so in dash than bitcoin, due to master nodes aggressively sending out signature shares and related messages, and additional logic that uses stuff like ForEachNode.

Long term, I may want to convert the non-recursive mutex into a shared mutex, similar to the logic in 6468 as m_nodes is only actually mutated (hence needing an exclusive lock) when nodes are added or removed. CNode is internally thread safe.

As you can see below, m_nodes_mutex represents ~90% of all contention on ~latest nightly on testnet during tx/islock spamming. Making this into a non-recursive mutex will likely significantly reduce the time a contention takes, although, the contention itself likely won't go away.

What was done?

Convert m_nodes_mutex from RecursiveMutex -> Mutex and fix all compilation errors.

How Has This Been Tested?

This should be tested with a tsan reindex and otherwise tsan test suite, which has not been done yet.

Breaking Changes

N/a

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation
• I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

[github-actions]

This pull request has conflicts, please rebase.

[UdjinM6]

CI is pretty unhappy...

[PastaPastaPasta]

I had changes that fixed this, but they're on my dead system... will have to wait a week or so before I can recover them.

[github-actions]

This pull request has conflicts, please rebase.