end2end tests part 1 #316
end2end tests part 1 #316
Conversation
trzysiek
force-pushed
the
e2e-1
branch
13 times, most recently
from
f8119e8
to
4ce7419
Compare
trzysiek
force-pushed
the
e2e-1
branch
2 times, most recently
from
4400d63
to
8931633
Compare
bin/save_test.py
Outdated
| @@ -0,0 +1,79 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
Maybe we can use more specific name here: e2e_save_tests.py
There was a problem hiding this comment.
We should be able to control recording. Right now all request are being recorded.
Example recording scenario:
./bin/e2e_start_recording.py
# some kibana exploration here
./bin/e2e_stop_recording.py
./bin/e2e_save_tests.py
# or
./bin/e2e_clean_recording.py
There was a problem hiding this comment.
I would even add own directory for that:
./bin/e2e/start_recording.py
docker/mitmproxy/test_saver.py
Outdated
| self.w.add(flow) | ||
|
|
||
|
|
||
| writer = Writer() |
There was a problem hiding this comment.
mitmproxy doesn't start on my setup:
| @@ -0,0 +1,142 @@ | |||
| package end2end | |||
There was a problem hiding this comment.
File should be named e2e_integration_test.go and has following build tag
//go:build integration
otherwise tests will be run as normal unit tests.
There was a problem hiding this comment.
I would also keep e2e name consistent if possible
bin/save_test.py
Outdated
| @@ -0,0 +1,79 @@ | |||
| #!/usr/bin/env python3 | |||
There was a problem hiding this comment.
I would even add own directory for that:
./bin/e2e/start_recording.py
| @@ -0,0 +1,142 @@ | |||
| package end2end | |||
There was a problem hiding this comment.
I would also keep e2e name consistent if possible
|
Error I see: |
|
Ok, I started to understand how this works: |
MVP of e2e tests.
I'll write some
README, but maybe in next PR, once this is accepted.You can e.g. see below, that it's often the case that count responses from Quesma and Elastic differ by 1 for the basic
Explorer's date_histogram, which is the first testcase I "recorded" and included. It's most likely because of a timezone bug that I mentioned for a second during today's standup (#307)(error and info messages on both screens would already be better)
_(async_)searchand_field_caps) requests are being saved.bin/save_test.pywhich will copy either all of those requests, or some subset chosen by you, to a proper directory with testcases:quesma/tests/end2end/testcases/{new-testcase-nr}/I've done it and saved first 2 requests.
local-dev-dual-comparison.ymlconfig running, you can simply rungo testand all requests from all testcases will be run via Elastic/Quesma+Clickhouse, and responses compared.Of course there's lots of TODO, but it's something, and it seems to work.
Happy to hear any feedback/suggestions.