Parallel e2e tests #765
Merged
Parallel e2e tests #765
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closed
Choraden
reviewed
Apr 4, 2024
e2e/setup/runner.go
Outdated
| } | ||
|
|
||
| setups := slices.Clone(r.Setups) | ||
| if r.Parallel != 1 { |
There was a problem hiding this comment.
nit: r.Parallel != 1 && !r.Debug
|
LGTM, the debug mode satisfies my needs. I'm a little surprised the e2e job takes 5m. I thought we would be able to reduce it even more, but it is still a great improvement as non parallel takes 8m on average. |
mmatczuk
force-pushed
the
mmt/parallel_e2e
branch
from
d596cec to
f4035e8
Compare
|
Adding more parallelism does not make it run faster due to constrained resources. |
mmatczuk
force-pushed
the
mmt/parallel_e2e
branch
from
f4035e8 to
ad9b8eb
Compare
mmatczuk
force-pushed
the
mmt/parallel_e2e
branch
from
ad9b8eb to
b4fe841
Compare
This enabled building in rootful podman. Otherwise we hit this error Error: building at STEP "RUN apk add --no-cache bash bash-completion ca-certificates curl jq": setup network: pasta failed with exit code 1: Don't run as root. Changing to nobody... No routable interface for IPv6: IPv6 is disabled Couldn't open network namespace /proc/171325/ns/net: Permission denied
Apply single responsibility principle and make Compose a data type. The command helper is moved to separate type. Modernize code and prepare for expanding to other commands in the future.
The standard --wait flag does not work with podman compose and work on all services in the compose file. This implementation works only on the started containers.
Volumes are problematic when running on remote machines - don't use them.
Ports are now accessible via test container. The mapped ports cause problems when containers are not properly cleaned.
New runner implementation based on updated Compose utility. Tests must run in a container "test" that needs to be part of each setup. Debug mode preserves compatibility with the previous version. Only single setup running in parallel and compose file is accessible locally. To speed things up compose teardown is executed asynchronously. In CI we do not wait for teardown completion.
Refactor dns and sc-2450 to separate containers. This allows to have healthcheck on them that simplifies the waiting.
The .env file is not read by podman-compose. Also, when running with tmp files it's not practical. Move everything to envvars.
Automatically print compose file and logs on test failure
$ make run-e2e
=== setup defaults-http-http
services:
httpbin:
image: saucelabs/forwarder:${FORWARDER_VERSION}
command: test httpbin
environment:
FORWARDER_API_ADDRESS: :10000
FORWARDER_PROTOCOL: http
ports:
- 10002:10000
proxy:
image: saucelabs/forwarder:${FORWARDER_VERSION}
environment:
FORWARDER_API_ADDRESS: :10000
FORWARDER_PROTOCOL: http
ports:
- 3128:3128
- 10000:10000
>>>> Executing external compose provider "/opt/homebrew/bin/docker-compose". Please refer to the documentation for details. <<<<
httpbin-1 | 2024/03/28 14:00:23.703134 [INFO] configuration
httpbin-1 | api-address=:10000
httpbin-1 | protocol=http
httpbin-1 | 2024/03/28 14:00:23.703319 [server] [INFO] HTTP server listen address=[::]:8080 protocol=http
httpbin-1 | 2024/03/28 14:00:23.703354 [api] [INFO] HTTP server listen address=[::]:10000 protocol=http
proxy-1 | 2024/03/28 14:00:23.661974 [INFO] Forwarder Unknown (Unknown)
proxy-1 | 2024/03/28 14:00:23.661986 [INFO] configuration
proxy-1 | api-address=:10000
proxy-1 | protocol=http
proxy-1 | 2024/03/28 14:00:23.662682 [proxy] [INFO] no upstream proxy specified
proxy-1 | 2024/03/28 14:00:23.662722 [proxy] [INFO] localhost proxying mode=deny
proxy-1 | 2024/03/28 14:00:23.662990 [proxy] [INFO] PROXY server listen address=[::]:3128 protocol=http
proxy-1 | 2024/03/28 14:00:23.663153 [api] [INFO] HTTP server listen address=[::]:10000 protocol=http
exit status 2
exit status 1
make: *** [Makefile:13: run-e2e] Error 1
Rename additional network to internal. If there is a network defined automatically add test container to it.
Run all test with a single command.
mmatczuk
force-pushed
the
mmt/parallel_e2e
branch
from
b4fe841 to
ad90c33
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patchset redefines our e2e tests framework allowing to run the tests in parallel.
It also allows to run remotely with
podman-remoteas the container runtime.Check e2e/README.md for details.