neutrino: improve perf in high latency networks

[djkazic]

After working with @hsjoberg and @niteshbalusu11 to troubleshoot Blixt performance we realized two things were consistently going wrong in networks with > around 300ms latency:

1. Retries would be exhausted quickly
2. Due to that --^ timeouts being doubled did not help much

After investigating, we decided to experiment with changing some constants in neutrino to work better.

This PR addresses both issues by:

• Decreasing maxBatch to 10
• Increasing query retries from 2 to 8
• Increasing rescan retries from 1 to 2
• Increasing broadcast query retries from 1 to 8

The result is that we can now use Blixt with networks that we could not previously.

[djkazic]

OK, it appears that queryAllPeers breaks the unit test when the numRetries of the defaultQueryOptions struct is > 1.

[djkazic]

Cool, so I fixed the unit test.

Now, the Blixt team will do some testing to make sure that we haven't also rolled back the perf improvements.

[ellemouton]

would it not make sense for high latency environments to instead make the timeout longer instead of bumping num retries? (or both lol)

also - wondering if we should make this configurable rather than hard code it?

[djkazic]

1. So the timeouts and retries are directly connected to one another, as when we have a timeout we'll do a retry (bounded by the retry count). What we found from trace logs was that the networks didn't consistently timeout, but rather during higher bandwidth utilization there was a chance of hitting the low retry count limit. That's why we raised the retries, since raising the timeout would also indirectly influence the amount of extra timeout added (it doubles each time it fails).

In my highly non-scientific testing there seemed to be a trend where if we bumped timeouts alone, reliability would still be bad. So this is just kind of the solution that we converged.

2. Yes, I've thought about making them configurable settings. However, we did a grid search and these were the best settings that balanced perf for slow and fast networks. I figured a follow-up PR where we break out these as configurables would be good, but let me know if we should bundle the changes. This PR is mostly focused on "saner" defaults.

[hsjoberg]

would it not make sense for high latency environments to instead make the timeout longer instead of bumping num retries? (or both lol)

Increasing timeout is kinda a bigger change as the number gets increased exponentially after each try.
I think increasing both retry and timeout makes sense. But I could look into that in a follow-up PR instead.

also - wondering if we should make this configurable rather than hard code it?

Yeah, I've thought of that too. It would be a very nice thing to have.

[ellemouton]

gotcha gotcha

[ellemouton]

cool LGTM - before the final ✅ though, can you open an LND PR that points to this so that we can just check the CI on that side?

[djkazic]

cool LGTM - before the final ✅ though, can you open an LND PR that points to this so that we can just check the CI on that side?

Created here: lightningnetwork/lnd#8797

[ellemouton]

cool thanks - will ACK this once the neutrino test on that side is green

[lightninglabs-deploy]

@djkazic, remember to re-request review from reviewers when ready

[ellemouton]

Looking good on the LND side - thanks!

[guggero]

One question about disk performance on fast systems, otherwise LGTM.