Handle host ports with emulator tests

[elizabethengelman]

Troubleshooting emulator tests

Closes #1461 🤞

What

We were still seeing emulator test failures.

Why/Issue Description

The speculos emulator dockerfile exposes a bunch of ports that we don't necessarily need for our tests. We relied on testcontainer's port publishing, which publishes all ports by default. I think that there were conflicts in which ports were being used for which tests. Since the tests are run in parallel, sometimes a test would try to access the API/UI endpoint on the GH runner host, and another test container was also using that port. 😵

This PR explicitly only publishes the two ports that we need, and makes sure to get two open ports per test.

Known limitations

We are on an older version of testcontainers, which we can hopefully update as part of #1471. I'm just pointing this out here, because our current version of testcontainers doesn't make it easy to publish just the ports with want, at dynamic ports on the host. Maybe we can clean this up when testcontainers is updated.

Shout out to @Ifropc for digging into this too!

[Ifropc]

The instability of tests seems to still be there 😵‍💫😵‍💫

failures:

---- emulator_tests::test_sign_tx::when_the_device_is_nanox stdout ----
listener1: TcpListener { addr: 127.0.0.1:36175, fd: 84 }
Retry count: 0
Wait time: 1s
Error occurred while exchanging with Ledger device: Error connecting to ledger device
thread 'emulator_tests::test_sign_tx::when_the_device_is_nanox' panicked at cmd/crates/stellar-ledger/tests/test/emulator_tests.rs:185:13:
assertion failed: false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- emulator_tests::test_sign_tx::when_the_device_is_nanos_plus stdout ----
listener1: TcpListener { addr: 127.0.0.1:36975, fd: 63 }
Retry count: 0
Wait time: 1s
Retry count: 1
Wait time: 2s
Error occurred while exchanging with Ledger device: Error connecting to ledger device
thread 'emulator_tests::test_sign_tx::when_the_device_is_nanos_plus' panicked at cmd/crates/stellar-ledger/tests/test/emulator_tests.rs:185:13:
assertion failed: false

---- emulator_tests::test_sign_tx_hash_when_hash_signing_is_enabled::when_the_device_is_nanos stdout ----
listener1: TcpListener { addr: 127.0.0.1:42571, fd: 93 }
Retry count: 0
Wait time: 1s
thread 'emulator_tests::test_sign_tx_hash_when_hash_signing_is_enabled::when_the_device_is_nanos' panicked at cmd/crates/stellar-ledger/tests/test/emulator_tests.rs:266:13:
Unexpected result: Error occurred while exchanging with Ledger device: Error connecting to ledger device

---- emulator_tests::test_sign_tx::when_the_device_is_nanos stdout ----
listener1: TcpListener { addr: 127.0.0.1:33041, fd: 30 }
Retry count: 0
Wait time: 1s
Retry count: 1
Wait time: 2s
Error occurred while exchanging with Ledger device: Error connecting to ledger device
thread 'emulator_tests::test_sign_tx::when_the_device_is_nanos' panicked at cmd/crates/stellar-ledger/tests/test/emulator_tests.rs:185:13:
assertion failed: false


failures:
    emulator_tests::test_sign_tx::when_the_device_is_nanos
    emulator_tests::test_sign_tx::when_the_device_is_nanos_plus
    emulator_tests::test_sign_tx::when_the_device_is_nanox
    emulator_tests::test_sign_tx_hash_when_hash_signing_is_enabled::when_the_device_is_nanos

test result: FAILED. 11 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 77.78s

Good thing is that it's only Error connecting to ledger device error now, no port issues anymore 🎉

[elizabethengelman]

Bummer! Thanks for checking this out - I am struggling to repro this locally, even when running the test a bunch of times in short order. 🤔

Looking at it again this morning - it looks like we are waiting for the UI port to be running, but are not running for the emulated device itself. I will push an update for that shortly.

[Ifropc]

Still get the failures on the latest HEAD...

failures:

---- emulator_tests::test_sign_tx::when_the_device_is_nanos stdout ----
Error occurred while exchanging with Ledger device: Error connecting to ledger device
thread 'emulator_tests::test_sign_tx::when_the_device_is_nanos' panicked at cmd/crates/stellar-ledger/tests/test/emulator_tests.rs:190:13:
assertion failed: false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    emulator_tests::test_sign_tx::when_the_device_is_nanos

test result: FAILED. 14 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 41.10s

error: test failed, to rerun pass `--test test`
ifro@nixos:~/projects/stellar-cli/ > git log
commit 0b8850348d168ffac72a44b0d46f5e4b18003413 (HEAD, AhaLabs/fix/troubleshooting-emulator-tests)
Author: Elizabeth Engelman <[email protected]>
Date:   Wed Jul 31 12:29:21 2024 -0400

    Make sure that ports are not conflicting between test threads

Seems to be passing on CI though 😅
Do you think you could add a dumb of container logs just before the failure? Maybe it will give us some ideas.
Worst case we can maybe we restart the test all together if getting Error connecting to ledger device 🤔

[Ifropc]

Ran the test 30 times locally, the only flaky test is emulator_tests::test_sign_tx::when_the_device_is_nanos and it failed 28/30 times. Not sure what makes this test special 🤔

[elizabethengelman]

Ran the test 30 times locally, the only flaky test is emulator_tests::test_sign_tx::when_the_device_is_nanos and it failed 28/30 times. Not sure what makes this test special 🤔

I don't know why I didn't see these failures before, but i am seeing it now!

I think what's happening now is that even though I was checking if the port is in use, I was checking at 127.0.0.1, but it looks like docker is publishing ports at all network interfaces, not just 127.0.0.1. I'll push up a change shortly to check if the port is open on all interfaces (0.0.0.0).

Oh yeah, and why that test is so special... I wonder if it is just the order in which the tests are run, that one tends to be the first one to reuse a port that was already in use? Not 100% sure.

[Ifropc]

I was actually trying to run just this test before (in isolation), and it was still flaky (I was also thinking that maybe there was some environment issue). It was before the changes introduced in this PR though

[elizabethengelman]

I was able to get this to pass a few times in CI on my fork, with the workflow running on a couple of branches at the same time. Are we comfortable merging this in, and then I can reenable and test it out for real? @Ifropc @leighmcculloch

If we start seeing issues again, we can disable again, and reevaluate.