Intermittent test panics in non_finalized_state::tests::prop::finalized_equals_pushed_genesis

[teor2345]

Motivation

We're seeing some failures in the merge queue due to an unrelated test bug:

---- service::non_finalized_state::tests::prop::finalized_equals_pushed_genesis stdout ----
The application panicked (crashed).
Message: only called while blocks is populated
Location: zebra-state/src/service/non_finalized_state/chain.rs:921

https://github.com/ZcashFoundation/zebra/actions/runs/4672510901/jobs/8274774308#step:14:3305

https://github.com/ZcashFoundation/zebra/actions/runs/4684761663/jobs/8301221723?pr=6496#step:14:2893

Analysis

This could be a test bug, or it could be revealing a possible panic in production.

We should check the code that's panicking to decide how important this issue is.

[teor2345]

I think we're going to need a full backtrace with debug info to diagnose this issue. In CI, I can't see the calling functions, so I don't know what code is actually causing this test failure.

[teor2345]

Failed PR #6515:
https://github.com/ZcashFoundation/zebra/actions/runs/4696422367/jobs/8327044365?pr=6515#step:3:13194

@mpguerra another test that is failing regularly and should be fixed so we can reliably merge PRs.

[teor2345]

Failed PR #6516:
https://github.com/ZcashFoundation/zebra/actions/runs/4696756744/jobs/8327729472?pr=6516#step:3:23435

[teor2345]

We might be able to diagnose this issue by changing the panic to a test error, and then committing the proptest seed to our git repository.

If it's a reproducible error that only happens with some test data, it will happen all the time once we commit the seed. If it's a timing issue, then it won't happen reliably with the seed. (But we'll still have learnt something.)

[teor2345]

Failed in PR #6537:
https://github.com/ZcashFoundation/zebra/actions/runs/4737607161/jobs/8411027811?pr=6537#step:3:3051

[mpguerra]

Let's devote some cycles this sprint to figuring out what's going on here

[teor2345]

It's possible that the test doesn't wait long enough for the blocks to commit to the state. Or the blocks are actually invalid, so they get rejected, but we're not checking for block commit errors.

[arya2]

Message: Test failed: only called while blocks is populated; minimal failing input: (chain, end_count, network, empty_tree) = (alloc::vec::Vec<zebra_state::request::PreparedBlock><zebra_state::request::PreparedBlock>, len=104, 103, Mainnet, HistoryTree(None))

Looks like a test bug, there won't be any blocks in the partial chain if end_count > chain.len() - 2.

[mpguerra]

@arya2 can you please add a size for this issue?