triage: prover node quotes aren't getting gossiped

[just-mitch]

A fresh bot is sending a transaction which gets added to the mempool by the bootnode, but then is dropped with no explanation.

The reason was in part because the TX was submitted, then a reorg occurred.

The reorg occurred in part because the prover node's quote wasn't gossiped beyond the boot node.

Steps to repro

./scripts/earthly-local ./yarn-project/+export-e2e-test-images

cd yarn-project/end-to-end

AZTEC_DOCKER_TAG=yournewtag NAMESPACE=smoke FRESH_INSTALL=true VALUES_FILE="48-validators.yaml" ./scripts/network_test.sh ./src/spartan/smoke.test.ts

Assuming it comes up, restart the bot and watch its logs.

Explore-logs-2024-11-01 14_27_30.json

[just-mitch]

Re-ran the repro with aztec:libp2p_service logs enabled on the boot node. I'm seeing:

Already connected to peer 16Uiu2HAkxo...
Handling discovered peer 16Uiu2HAkxo...

Multiple times per second. I filed #9674

[just-mitch]

It's looking like the transaction got dropped after a reorg. At least 2 issues come out of this:

1. I saw a node try to build the block when it only had like 5 seconds left. I filed node needs a minimum time for block building #9675
2. The prover node's quote isn't getting gossiped. I can see it lands on the boot node but goes no further.

I will repurpose this issue to address point 2. Next triage is actually to re-run with libp2p debug logs on all the nodes to ensure they in fact receive nothing (though I don't see anyone attempting to even claim the quote, but that could be due to other issues).

I also need to enable discv5 logs on the prover node- it should be broadcasting to more than just the boot node.

There is, however, still a serious, separate issue: I started the 48 validator test in AWS and got TX timeouts from the bot before a reorg was possible. I tracked that on in #9676

[just-mitch]

I ran the 48 validator setup, and we had over 75 consecutively proven epochs. Closing this until we have a new concrete occurrence.