fix: concurrency insert between pending and delivered

[Freyskeyd]

Description

This PR is covering a corner case which involve the delivery of a prev certificate and a child certificate.

The scenario is this one:

• A prev certificate (C1) has been delivered (by the broadcast) and need to be persisted
  The persist method will hold a lock while performing multiple insert/delete to avoid
  insert race condition.
• At the same time, another node is sending a certificate (C2) which have C1 as prev_id.
  C2 is looking at the storage to find if the prev_id C1 is delivered but find nothing as
  the persist method is still working at creating the WriteBatch. It led the node to put
  C2 in the precedence_pool waiting for C1 to be delivered while it is in fact already
  delivered.

To avoid that and as a first step, when trying to insert a certificate in the pending pool,
The node will try to acquire a lock guard on the certificate but also on the prev_id.

PR Checklist:

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added or updated tests that comprehensively prove my change is effective or that my feature works

[codecov]

Codecov Report

Attention: Patch coverage is 87.87879% with 12 lines in your changes are missing coverage. Please review.

Project coverage is 71.69%. Comparing base (a5299c8) to head (37d59d6).

Files	Patch %	Lines
...es/topos-p2p/src/runtime/handle_event/peer_info.rs	55.55%	8 Missing ⚠️
crates/topos-tce-storage/src/fullnode/mod.rs	89.47%	2 Missing ⚠️
crates/topos-p2p/src/behaviour/gossip.rs	50.00%	1 Missing ⚠️
crates/topos-tce/src/app_context/api.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #467      +/-   ##
==========================================
- Coverage   71.87%   71.69%   -0.18%     
==========================================
  Files         225      227       +2     
  Lines       12675    12743      +68     
==========================================
+ Hits         9110     9136      +26     
- Misses       3565     3607      +42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dvdplm]

LGTM, and thank you for the high quality comments explaining what is going on.

Beyond this PR, I think this need for strong synchronization is unfortunate. It makes the code less clear and introduces delays that could harm performance. Not sure how we could do better but I'll try to give it some thought. I have a gut feeling that treating pending and delivered certificates separately is part of the problem, but I'll have to think more on this.