TLS Anvil Server Tests in Nightly CI #3651

FAlbertDev · 2023-07-31T08:15:18Z

(Refers to #3444)
This adds TLS Anvil server tests to Botan's nightly CI. Unfortunately, a current bug of the origin TLS-Anvil repo required me to create a fork and work around it (I already created an issue for the TLS-Anvil repo). Until the bug is fixed, the CI script uses my fork.
The job's workflow is the following:

Build Botan and TLS-Anvil
Run the script src/scripts/ci/ci_tlsanvil_test.py to setup and start a botan tls_server and the TLS-Anvil process. The TLS-Anvil process tests the server by playing a client (repeatedly) connecting to the server. TLS-Anvil creates a TestSuiteResults folder for storing the results with additional test information.
The TestSuiteResults folder and the TLS-Anvil log files are uploaded as an artifact, so we can analyze them if necessary.
The script src/scripts/ci/ci_tlsanvil_check.py checks if the tests are successful, i.e., if the results in TestSuiteResults meet our expectations. Currently, some tests fail. I will look into these together with @reneme after his holidays. Until then, I allowed the respective tests to fail.

An example workflow execution with failing tests (without deactivating the currently failing tests) is here. When allowing the tests to fail, the workflow looks like this. Since the tests take around 1.5h, they are only executed nightly.
I also plan to add the TLS-Anvil client tests in a future PR.

coveralls · 2023-07-31T08:50:07Z

coverage: 91.703% (-0.02%) from 91.719% when pulling 4fb301b on FAlbertDev:ci/tls-anvil into 8c72bab on randombit:master.

randombit

Thanks for setting this up @FAlbertDev, I'll be happy when this is part of our test coverage.

I have a few minor comments/questions but overall looks very good.

randombit · 2023-08-03T10:40:12Z

src/scripts/ci/ci_tlsanvil_test.py

+    cert_path = os.path.join(Config.key_and_cert_storage_path, Config.tmp_cert_file_name)
+
+    with open(key_path, 'w', encoding='utf-8') as keyfile:
+        subprocess.run([botan_exe_path, "keygen", "--algo=RSA"], stdout=keyfile, check=True)


keygen with RSA defaults to 3072 bits since that is ~~ 128 bit security level, but for testing this is just some needless computational overhead - let's generate 2048 bit instead since that is still sufficient and may save a bit of test time.

Seems to be not significantly faster with 2048 bit. Won't hurt, though.

randombit · 2023-08-03T10:41:45Z

src/scripts/ci/ci_tlsanvil_test.py

+    tls_anvil_cmd = [
+        "java", "-jar", tls_anvil_jar_path,
+        "-strength", "1",
+        "-parallelHandshakes", "1",


The runners we get have 2 cores - would it make sense / be possible to use 2-3 handshakes in parallel here?

Yeah, parallel handshakes would be great. However, I run TLS-Anvil against the ./botan tls_server process, which only allows one connection at a time.

Could we use tls_http_server instead?

randombit · 2023-08-03T10:42:45Z

src/scripts/ci/ci_tlsanvil_test.py

+        action="store_true",
+        default=False,
+        help="Test the Botan TLS client",
+    )


Maybe instead a single option --test-target= which could take on any of client, server, or client,server?

randombit · 2023-08-03T10:44:25Z

.github/workflows/nightly.yml

+          cache-key: linux-clang-x86_64-static
+
+      - name: Build Botan
+        run: ./configure.py --compiler-cache=ccache --build-targets=static,cli --without-documentation && make -j8


TBH I don't know if this even works in GH Actions but should this be something like

make -j$(nproc)

so the build scales to using all (and only) the number of cores available?

It seems to work 👍

randombit · 2023-08-03T10:49:01Z

.github/workflows/nightly.yml

+        uses: ./.github/actions/setup-build-agent
+        with:
+          target: static
+          cache-key: linux-clang-x86_64-static


This cache key seems wrong since we are using gcc to build

I think if we use the linux-gcc-x86_64-static key, then we can share caches with this build and the normal static library GCC CI build.

FAlbertDev · 2023-08-04T10:13:57Z

Thanks for your review, @randombit! I addressed your suggestions in ba484ec :)

reneme

Nice! Just a few general remarks below. Detailed review will need to come later.

reneme · 2023-08-07T11:05:19Z

.github/workflows/nightly.yml

+      - name: Fetch TLS-Anvil fork for Botan tests
+        uses: actions/checkout@v3
+        with:
+          repository: FAlbertDev/TLS-Anvil
+          path: ./tlsanvil


If we need a fork, maybe it should live under @randombit's GitHub account. Similarly to the boringssl fork we maintain for BoGo.

(I'm guessing you hope to get rid of the fork after your upstream issue is fixed)

(I'm guessing you hope to get rid of the fork after your upstream issue is fixed)

True. In fact, the same hour you wrote this comment, the issue was fixed ;)

reneme · 2023-08-07T11:07:02Z

.github/workflows/nightly.yml

+      - name: Build Botan
+        run: ./configure.py --compiler-cache=ccache --build-targets=static,cli --without-documentation && make -j$(nproc)


Maybe configure and build botan in the ci_tlsanvil_test.py instead?

reneme · 2023-08-07T11:09:06Z

.github/workflows/nightly.yml

+        env:
+          DOCKER: 1 # TLS-Anvil specific to disable the loading bar


Please set this inside the ci_tlsanvil_test.py script, if possible. Rationale: We try to keep the CI YAML files as "dumb" as possible and move as much of the configuration and tool-intricacies into the dedicated scripts.

reneme · 2023-08-07T11:12:50Z

src/scripts/ci/ci_tlsanvil_test.py

+    with open(key_path, 'w', encoding='utf-8') as keyfile:
+        subprocess.run([botan_exe_path, "keygen", "--algo=RSA", "--params=2048"], stdout=keyfile, check=True)
+
+    with open(cert_path, 'w', encoding='utf-8') as certfile:
+        subprocess.run([botan_exe_path, "gen_self_signed", key_path, "localhost"], stdout=certfile, check=True)


As a general remark: At one point we should spin-off some internal Utilities python module that we can reuse across the internal python machinery. As a center piece this will likely include a convenience wrapper around subprocess.run and/or subprocess.Popen. No idea, how many scripts already have a copy of run_commend(). 🤡

Yeah. But we need to evaluate if an additional abstraction layer is worth it. I think, in this case, the calls are slim enough.

FAlbertDev · 2023-08-08T07:16:58Z

My current state for this PR:
I'm currently working on the usage of the tls_http_server instead of tls_server to speed up the tests by parallelization. Some test results are currently inconsistent between both CLI modules (probably because of a small issue with the tls_http_server). I'll investigate this and will probably open another PR addressing this issue after understanding the underlying problem.
Also, I need to implement your suggestions, @reneme, and redirect the checkout of TLS-Anvil to the origin repo, which is fixed now.

FAlbertDev · 2023-08-09T15:12:03Z

.github/workflows/nightly.yml

+        uses: ./.github/actions/setup-build-agent
+        with:
+          target: static
+          cache-key: linux-gcc-x86_64-static


Do we need a separate cache key because we build with --with-boost?

Oh probably so - maybe best to just use a new unique cache id for each job and let the cache management system sort it out rather than trying to share caches and causing conflicts and cache misses if/when the configuration of different jobs changes over time.

Also: if we use a unique target here (eg anvil) we can install boost in setup_gh_actions.sh as a side effect of setting up the build agent, rather than having an explicit install of boost later on in the yaml

Done. (see a589ed2)

Thanks for your feedback!

FAlbertDev · 2023-08-09T15:14:33Z

I applied your suggestions (109f2e4). However, this PR should only be merged after #3660 is merged.

randombit · 2023-08-11T13:06:27Z

LGTM. Only remaining comment from me would be please squash before merging.

FAlbertDev · 2023-08-11T13:34:54Z

please squash before merging

Done.

randombit reviewed Aug 3, 2023

View reviewed changes

reneme reviewed Aug 7, 2023

View reviewed changes

FAlbertDev mentioned this pull request Aug 9, 2023

Fix: TLS HTTP server alert handling #3660

Merged

FAlbertDev commented Aug 9, 2023

View reviewed changes

FAlbertDev force-pushed the ci/tls-anvil branch 2 times, most recently from 9836910 to a589ed2 Compare August 11, 2023 07:26

GitHub CI Action for TLS-Anvil Server Tests

4fb301b

FAlbertDev force-pushed the ci/tls-anvil branch from a589ed2 to 4fb301b Compare August 11, 2023 13:34

randombit approved these changes Aug 13, 2023

View reviewed changes

reneme approved these changes Aug 15, 2023

View reviewed changes

reneme merged commit fc4b427 into randombit:master Aug 15, 2023
36 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TLS Anvil Server Tests in Nightly CI #3651

TLS Anvil Server Tests in Nightly CI #3651

FAlbertDev commented Jul 31, 2023 •

edited

Loading

coveralls commented Jul 31, 2023 •

edited

Loading

randombit left a comment

randombit Aug 3, 2023

FAlbertDev Aug 4, 2023

randombit Aug 3, 2023

FAlbertDev Aug 4, 2023

randombit Aug 4, 2023

randombit Aug 3, 2023

randombit Aug 3, 2023

FAlbertDev Aug 4, 2023

randombit Aug 3, 2023

FAlbertDev commented Aug 4, 2023

reneme left a comment

reneme Aug 7, 2023

reneme Aug 7, 2023

FAlbertDev Aug 8, 2023

reneme Aug 7, 2023

reneme Aug 7, 2023

reneme Aug 7, 2023

FAlbertDev Aug 8, 2023

FAlbertDev commented Aug 8, 2023 •

edited

Loading

FAlbertDev Aug 9, 2023

randombit Aug 10, 2023

randombit Aug 10, 2023

FAlbertDev Aug 11, 2023

FAlbertDev commented Aug 9, 2023 •

edited

Loading

randombit commented Aug 11, 2023

FAlbertDev commented Aug 11, 2023

		- name: Build Botan
		run: ./configure.py --compiler-cache=ccache --build-targets=static,cli --without-documentation && make -j$(nproc)

		env:
		DOCKER: 1 # TLS-Anvil specific to disable the loading bar

TLS Anvil Server Tests in Nightly CI #3651

TLS Anvil Server Tests in Nightly CI #3651

Conversation

FAlbertDev commented Jul 31, 2023 • edited Loading

coveralls commented Jul 31, 2023 • edited Loading

randombit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FAlbertDev commented Aug 4, 2023

reneme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FAlbertDev commented Aug 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FAlbertDev commented Aug 9, 2023 • edited Loading

randombit commented Aug 11, 2023

FAlbertDev commented Aug 11, 2023

FAlbertDev commented Jul 31, 2023 •

edited

Loading

coveralls commented Jul 31, 2023 •

edited

Loading

FAlbertDev commented Aug 8, 2023 •

edited

Loading

FAlbertDev commented Aug 9, 2023 •

edited

Loading