fix(tee-prover): mitigate panic on redeployments

[pbeza]

What ❔

We experienced tee-prover panic, likely due to the automatic redeployment of the proof-data-handler in the staging environment. We've been getting 503 Service Unavailable errors for an extended period when trying to reach http://server-v2-proof-data-handler-internal.stage.matterlabs.corp/tee/proof_input, which resulted in a panic after reaching the retry limit.

Relevant code causing the panic:

zksync-era/core/bin/zksync_tee_prover/src/tee_prover.rs

Lines 201 to 203 in 8ed086a

	if !err.is_retriable() || retries > self.config.max_retries {
	return Err(err.into());
	}

Relevant logs.

Why ❔

To mitigate panics on proof-data-handler redeployments.

Checklist

• PR title corresponds to the body of PR (we generate changelog entries from PRs).
• Tests for the changes have been added / updated.
• Documentation comments have been added / updated.
• Code has been formatted via zk fmt and zk lint.

[pbeza]

With this code change, tee-prover will retry connecting to http://server-v2-proof-data-handler-internal.stage.matterlabs.corp/tee/proof_input for a maximum of:

$$\begin{aligned} \sum_{i=0}^{9} \min(2^i, 128) &=\sum_{i=0}^{7} 2^i + \min(2^8, 128) + \min(2^9, 128) \\ &= 2^{8} - 1 + 128 + 128 \\ &= 255 + 256 \\ &= 511 \end{aligned} $$

seconds, which is 8 minutes and 31 seconds if my math is correct. :P

Not sure if just increasing the number of retries is good enough, though. I welcome any other suggestions. Perhaps we can allow panics during redeployments?

[pbeza]

@popzxc, PTAL.

I’ve addressed all your comments. I think the failing tests are just being flaky.