Docker Service for runtime tests

[rylev]

This PR introduces Docker based services for runtime tests to compliant Python based services introduced in #2180. With this, we can now run all runtime tests as long as the user has Python and Docker installed and running.

Features

The docker based services are run for each test and will be spun down after the test is run.

Ports

In addition to these services, new support for templating ports in the runtime test's spin.toml manifest was added along with changes to the templating language.

The interpolation now happens through {{key=value}} annotations where key is one of a limited number of keys the test runner supports. The supported keys are:

• source: This support already existed and allows the manifest to reference pre-built Spin compliant WebAssembly modules that can be found in the test-components folder in the Spin repo. The value is substituted for the name of the test component to be used. For example {{source=sqlite}} will use the test-component named "sqlite" found in the test-components directory.
• port: The manifest can reference a port that has been exposed by a service. For example, if the test runner sees {{port=1234}} it will look for a service that exposes the guest port 1234 on some randomly assigned host port and substitute {{port=1234}} for that randomly assigned port.

This ensures that the service does not need to hardcode a port which might already be taken on the host.

Drawbacks

This approach has a few drawbacks:

• Each test runs each service independently instead of tests sharing a service. This means we pay the start up and shut down costs for a service for each test that uses that service instead of paying only once per test suite run. I believe this is worth it to keep each test isolated and to reduce the need for complicated book keeping that somehow needs to understand that other tests rely on a service and so the service should not be shut down.
• The service has no knowledge on whether the actual underlying dependency is in a good state. For example, the docker service does check that the docker container is running but it has no idea if the service inside that container is in a good state. We have to hardcode around this for now with MySQL which takes a while to start up before it can start accepting connections.

FAQ

How does one add a Docker service?

Just like a Python service, you just need to add a service configuration file to the services directory. For Docker services this configuration file is just a Dockerfile.

Why both Python and Docker based services?

When looking to add a new service, always prefer the Python based service as it's generally much quicker and lighter weight to run a Python script than a Docker container. Only use Docker when the service you require is not possible to achieve in cross platform way as a Python script.

For example, we need to be able to run against a PostgreSQL database for some runtime tests. This is not possible to achieve with Python in a cross platform way, do we use Docker. However, the TCP server based tests can easily be achieved with a Python script.

[lann]

How would you feel about using different template delimiters, like ${...}? I already get confused by both templates and variables using {{...}}.

[lann]

Am I reading this right that it would preclude the use of spin variables in a test manifest?

[rylev]

I think the point you brought up above about using a different template delimiter applies here. I just completely forgot that {{}} already has semantic meaning. Perhaps ${{}} would work?

[lann]

${{}} might be confusing as it could be read as "$ + {{variable}}".

[endocrimes]

Fwiw you may want to look at https://github.com/testcontainers/testcontainers-rs - at least in Go they've got some useful helpers for things like "waiting for a service to actually be up"

[rylev]

Took a look at the Rust library, and I'm not sure it's worth taking on the dependency especially since the API they expose is sort of directly at odds with how we want to do things. We could make it work, but the code we have now isn't too bad, and I think it ultimately will be easier to maintain.

[rylev]

I implemented health checking now too which was only a few lines of code, so it seems like we're fine without a library for now.

[endocrimes]

Not about the code but more about the preference for Python - it's relatively hard to write cross platform Python and also have reliable dependency management that will be consistently available - we probably need some guidelines to go along with this (I may have missed some?)

[rylev]

I think the issue is that we don't have a great alternative. Ideally whatever we pick has these properties:

• Relatively widely known or easy to write by cargo culting
  • e.g., Lua might tick some of these other boxes but is a bit of a strange pick
• Can start running relatively quickly
  • this is why I didn't chose Rust despite the rest of the project being written in it
• Is possible to run cross platform
  • e.g., I'd like to avoid bash scripts so that we can run this easily on Windows
• Can be used consistently
  • this is why I didn't pursue using WebAssembly for this since I'd rather not have each service written in a different language

That all being said, I'll definitely try to write some best practice guidelines so we can make the best out of Python.

[endocrimes]

Python 3 is not consistently aliases as python3 across systems and distributions - especially ones that are removing Python 2

[rylev]

👍 This needs to be improved in other ways too. For example, we should ensure sooner rather than later that the Python we try to use is actually there.