zebra_zcash_listener_conflict test occasionally fails

[teor2345]

Motivation

The zebra_zcash_listener_conflict zebrad acceptance test occasionally fails with:

conflicted node2 was still running, but the test expected a panic

One of the launched processes does panic:

Message: Opening Zcash network protocol listener 127.0.0.1:58261 failed: Os { code: 98, kind: AddrInUse, message: "Address already in use" }. Hint: Check if another zebrad or zcashd process is running. Try changing the network listen_addr in the Zebra config.

The launch timing is right for node2, but the panic check isn't seeing the panic (it runs just before these lines):

    return node2
        .kill_on_error::<(), _>(Err(eyre!(

This has happened about once a day for the past week:
https://github.com/ZcashFoundation/zebra/runs/5114636264?check_suite_focus=true#step:13:1744

Diagnostic Suggestions

Turn on full backtraces in the acceptance tests using COLORBT_SHOW_HIDDEN=1.

Fix Suggestions

Use expect_stdout_line_matches right after node1 is launched, rather than using wait_with_output.
(node1 could be blocking on output, and never getting to run the conflicting port or file code.)

If this doesn't fix the bug, it will help with diagnostics.

Related Work

[ftm1000]

Hey team! Please add your planning poker estimate with ZenHub @dconnolly @gustavovalverde @conradoplg @jvff @oxarbitrage @teor2345 @upbqdn