E2E testing using testing framework

[rylev]

This starts the process of moving e2e testing over to using the new testing framework. As you can see, this resulted in roughly 5 times the amount of code being deleted as added.

The ultimate aim is that:

• The testing framework can easily handle all scenarios being tested in e2e tests
• Since integration and e2e tests will use the same framework, they can simply be combined into one type test type
• A higher-level integration test framework can be added on top of the lower level testing framework (in much the same way runtime tests work) so that adding new tests involves less boilerplate.

How things currently look

Here's an example of all the code needed to write a fairly complex e2e test that starts Spin as an inbound redis handler. This test depends on Redis running which is all handled by the testing framework. The only thing that needs to be installed on the machine to run this test (besides a Rust toolchain) is Docker which will run redis for the user.

As you can see in this test, the user has controlled access to the testing environment allowing the user to do things like make calls against the redis database without needing to know which port redis is actually running on locally. The user can also query the test environment to see if a log file was written in the expected place.

#[test]
fn redis_smoke_test() -> anyhow::Result<()> {
    run_test(
        "redis-smoke-test",
        testing_framework::SpinMode::Redis,
        [], // extra Spin CLI args which this test doesn't need
        testing_framework::ServicesConfig::new(vec!["redis".into()])?,
        move |env| {
            let redis_port = env.services_mut().get_port(6379)?.unwrap();

            let mut redis = redis::Client::open(format!("redis://localhost:{redis_port}"))
                .context("could not connect to redis in test")?;
            redis
                .publish("my-channel", "msg-from-test")
                .context("could not publish test message to redis")?;
            assert_eventually!({
                match env.read_file(".spin/logs/hello_stdout.txt") {
                    Ok(logs) => {
                        let logs = String::from_utf8_lossy(&logs);
                        logs.contains("Got message: 'msg-from-test'")
                    }
                    Err(e) if e.kind() == std::io::ErrorKind::NotFound => false,
                    Err(e) => return Err(anyhow::anyhow!("could not read stdout file: {e}").into()),
                }
            });
            Ok(())
        },
    )?;

    Ok(())
}

Next Steps

The next thing to do is continue to move more integration and e2e tests over to this framework and eventually delete all the old e2e and integration testing support code.

[rajatjindal]

I've heard some chat about support multiple types of trigger at some stage. would that become a different SpinMode at that time?

[rylev]

Yes. The SpinMode is used to indicate how we know how to communicate with Spin.