Add a stub Rust crate #12595

erikjohnston · 2022-04-29T14:45:19Z

This is a straw-man proposal for how we might want to integrate Rust into Synapse, and is not intended to land is is.

This bundles in Rust into Synapse proper, rather than e.g. having the rust code as a separate package. The advantage of this approach is that it allows very tight coupling between the Synapse and Rust code. The rust code is built and accessible at synapse.synapse_rust.*.

We achieve this by using a Poetry build script (which is a bit of an unstable feature python-poetry/poetry#2740) to add the rust as an extension using setuptools-rust.

Using poetry install correctly builds the rust code.

Changes to the rust code have to be manually rebuilt by running poetry install.

Building of a source package via poetry build -f sdist builds a sdist that includes the rust source.

Building of wheels is done via the docker manylinux package and a script to use poetry to manually build wheels for each python version. Based on https://setuptools-rust.readthedocs.io/en/latest/building_wheels.html

DMRobertson · 2022-04-29T14:55:50Z

We achieve this by using a Poetry build script (which is a bit of an unstable feature python-poetry/poetry#2740)

Is there any particular version of poetry we need to use for this to make sense? We're split between 1.1.12 and 1.2.0bsomething in CI.

erikjohnston · 2022-04-29T14:59:04Z

We achieve this by using a Poetry build script (which is a bit of an unstable feature python-poetry/poetry#2740)

Is there any particular version of poetry we need to use for this to make sense? We're split between 1.1.12 and 1.2.0bsomething in CI.

It seems to work fine with 1.1.13 which is what I have locally. Haven't tried the 1.2.0b yet

erikjohnston · 2022-08-17T16:53:49Z

This is now blocked on python-poetry/poetry#6154 I think (or alternatively we add setuptools_rust as a normal dep). Frustratingly they fixed that issue for poetry build in 1.2.0b2 (which now just works), but poetry install doesn't work.

Otherwise, pip install works out of the box, so I think that is pretty much the only blocker?

erikjohnston · 2022-08-18T14:31:17Z

Slightly annoyingly poetry 1.1.14 doesn't let you specify a minimum version of setuptools, so unless you manually upgrade it poetry install doesn't work.

It also looks like they're not planning to fix python-poetry/poetry#6154 imminently, so I've temporarily setuptools_rust to the dev-dependencies. Not sure if that is really the right place for it.

reivilibre · 2022-08-22T14:26:10Z

.github/workflows/release-artifacts.yml

@@ -91,7 +91,34 @@ jobs:

  build-sdist:
    name: "Build pypi distribution files"
-    uses: "matrix-org/backend-meta/.github/workflows/packaging.yml@v1"
+    steps:


should whatever changed here be folded back into the backend-meta repo?

Ah yup, probably. The change is using poetry build rather than pip build

reivilibre · 2022-08-22T14:26:42Z

.gitignore

+
+# rust
+/target/
+/synapse/*.so


do we care about ... dylibs? dlls? Whatever those other OS people do :)

Yeah, probably I guess? Though do we support other OSes?

I think @clokep is on Mac at least. Perhaps he can comment about what that looks like over there

Yes, I'm on macOS (and others who occasionally contribute to Synapse use macOS too) -- are you mostly just hoping I run this and make sure it doesn't break / what other files we might need to ignore?

Yeah, pretty much

On macOS this did seem to only generate a .so file, but it was under the build/ directory. This was when installing an editable install via pip, so might be OK if you use poetry?

/build/ is ignored separately fwiw, so thats fine.

Cargo.toml

pyproject.toml

DMRobertson · 2022-08-22T14:30:52Z

.github/workflows/release-artifacts.yml

+        uses: actions/setup-python@v2
+
+      - name: Install tools
+        run: python -m pip install build twine


I don't think you need build here, if we're using poetry build instead.

No objections to using poetry build instead of python -m build in the backend meta repo. (It should hopefully be equivalent anyway---I think build should invoke poetry-core via pyproject.toml?)

For whatever reason python -m build wasn't installing the build deps iirc?

tests/test_rust.py

reivilibre

Looks promising to me; I would double check that the sdist is good enough to use as an installation source before merging.

Will re-add to queue since you asked for general opinions on this and mine is but one.

erikjohnston · 2022-08-23T12:21:19Z

We also need to figure out a minimum supported rustc version

DMRobertson · 2022-09-06T15:53:30Z

.github/workflows/release-artifacts.yml

+
+    steps:
+      - name: Build sdist
+        run: python -m build --sdist


Does this work? Do you need to setup-python or pip install build?

Sigh, good point. Lets test that.

Erm, I think you're missing an actions/checkout too.

Yup, fixed now!

DMRobertson

Other than building the sdist lgtm. (I have no idea about the debian packaging.)

Something to think about for the future anticipate us writing unit tests in rust at all?

erikjohnston · 2022-09-06T16:00:23Z

Something to think about for the future anticipate us writing unit tests in rust at all?

Yeah, I plan to do a follow up PR that adds Rust specific CI (i.e. cargo fmt|check|clippy|test). Just don't want to clutter this PR up even more

DMRobertson · 2022-09-06T18:29:44Z

.github/workflows/release-artifacts.yml

-    uses: "matrix-org/backend-meta/.github/workflows/packaging.yml@v1"
+    name: Build sdist
+    runs-on: ubuntu-latest
+    if: ${{ !startsWith(github.ref, 'refs/pull/') }}


Hmm. Should we be building the sdist on PRs? (I'm not sure it picks anything hugely important up?)

jplatte · 2022-09-07T08:17:49Z

.github/workflows/tests.yml

+      - name: Install Rust
+        uses: actions-rs/toolchain@v1
+        with:
+            toolchain: 1.61.0
+            override: true


After toolchain installation, I recommend adding - uses: Swatinem/rust-cache@v2 as a step, which will take care of caching dependencies.

ooooooooooooooooooooooooooooo

jplatte · 2022-09-07T08:18:35Z

Cargo.toml

@@ -0,0 +1,5 @@
+# We make the whole Synapse folder a workspace so that we can run `cargo`


No corresponding Cargo.lock?

Oh, good point!

jplatte · 2022-09-07T08:21:03Z

docs/deprecation_policy.md

@@ -31,3 +37,10 @@ long process.
 By following the upstream support life cycles Synapse can ensure that its
 dependencies continue to get security patches, while not requiring system admins
 to constantly update their platform dependencies to the latest versions.
+
+For Rust, the situation is a bit different given that a) the Rust foundation


This should be "the Rust project", the foundation only does trademark, financing and PR stuff, it's not involved with maintenance / suppport or such.

NickCao · 2022-09-21T14:10:57Z

I noticed that there is no Cargo.lock in the release tarball or 1.68rc1, is this intended or will be addressed in the final release?

DMRobertson · 2022-09-21T14:12:56Z

I noticed that there is no Cargo.lock in the release tarball or 1.68rc1, is this intended or will be addressed in the final release?

It was an oversight, see #13858, which will be included in the final 1.68 release.

@dakariakin

Synapse 1.68.0 (2022-09-27) =========================== Please note that Synapse will now refuse to start if configured to use a version of SQLite older than 3.27. In addition, please note that installing Synapse from a source checkout now requires a recent Rust compiler. Those using packages will not be affected. On most platforms, installing with `pip install matrix-synapse` will not be affected. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.68/upgrade.html#upgrading-to-v1680). Bugfixes -------- - Fix packaging to include `Cargo.lock` in `sdist`. ([\matrix-org#13909](matrix-org#13909)) Synapse 1.68.0rc2 (2022-09-23) ============================== Bugfixes -------- - Fix building from packaged sdist. Broken in v1.68.0rc1. ([\matrix-org#13866](matrix-org#13866)) Internal Changes ---------------- - Fix the release script not publishing binary wheels. ([\matrix-org#13850](matrix-org#13850)) - Lower minimum supported rustc version to 1.58.1. ([\matrix-org#13857](matrix-org#13857)) - Lock Rust dependencies' versions. ([\matrix-org#13858](matrix-org#13858)) Synapse 1.68.0rc1 (2022-09-20) ============================== Features -------- - Keep track of when we fail to process a pulled event over federation so we can intelligently back off in the future. ([\matrix-org#13589](matrix-org#13589), [\matrix-org#13814](matrix-org#13814)) - Add an [admin API endpoint to fetch messages within a particular window of time](https://matrix-org.github.io/synapse/v1.68/admin_api/rooms.html#room-messages-api). ([\matrix-org#13672](matrix-org#13672)) - Add an [admin API endpoint to find a user based on their external ID in an auth provider](https://matrix-org.github.io/synapse/v1.68/admin_api/user_admin_api.html#find-a-user-based-on-their-id-in-an-auth-provider). ([\matrix-org#13810](matrix-org#13810)) - Cancel the processing of key query requests when they time out. ([\matrix-org#13680](matrix-org#13680)) - Improve validation of request bodies for the following client-server API endpoints: [`/account/3pid/msisdn/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidmsisdnrequesttoken), [`/org.matrix.msc3720/account_status`](https://github.com/matrix-org/matrix-spec-proposals/blob/babolivier/user_status/proposals/3720-account-status.md#post-_matrixclientv1account_status), [`/account/3pid/add`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidadd), [`/account/3pid/bind`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidbind), [`/account/3pid/delete`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3piddelete) and [`/account/3pid/unbind`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidunbind). ([\matrix-org#13687](matrix-org#13687), [\matrix-org#13736](matrix-org#13736)) - Document the timestamp when a user accepts the consent, if [consent tracking](https://matrix-org.github.io/synapse/latest/consent_tracking.html) is used. ([\matrix-org#13741](matrix-org#13741)) - Add a `listeners[x].request_id_header` configuration option to specify which request header to extract and use as the request ID in order to correlate requests from a reverse proxy. ([\matrix-org#13801](matrix-org#13801)) Bugfixes -------- - Fix a bug introduced in Synapse 1.41.0 where the `/hierarchy` API returned non-standard information (a `room_id` field under each entry in `children_state`). ([\matrix-org#13506](matrix-org#13506)) - Fix a long-standing bug where previously rejected events could end up in room state because they pass auth checks given the current state of the room. ([\matrix-org#13723](matrix-org#13723)) - Fix a long-standing bug where Synapse fails to start if a signing key file contains an empty line. ([\matrix-org#13738](matrix-org#13738)) - Fix a long-standing bug where Synapse would fail to handle malformed user IDs or room aliases gracefully in certain cases. ([\matrix-org#13746](matrix-org#13746)) - Fix a long-standing bug where device lists would remain cached when remote users left and rejoined the last room shared with the local homeserver. ([\matrix-org#13749](matrix-org#13749), [\matrix-org#13826](matrix-org#13826)) - Fix a long-standing bug that could cause stale caches in some rare cases on the first startup of Synapse with replication. ([\matrix-org#13766](matrix-org#13766)) - Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests. ([\matrix-org#13789](matrix-org#13789)) - Delete associated data from `event_failed_pull_attempts`, `insertion_events`, `insertion_event_extremities`, `insertion_event_extremities`, `insertion_event_extremities` when purging the room. ([\matrix-org#13825](matrix-org#13825)) Improved Documentation ---------------------- - Note that `libpq` is required on ARM-based Macs. ([\matrix-org#13480](matrix-org#13480)) - Fix a mistake in the config manual introduced in Synapse 1.22.0: the `event_cache_size` _is_ scaled by `caches.global_factor`. ([\matrix-org#13726](matrix-org#13726)) - Fix a typo in the documentation for the login ratelimiting configuration. ([\matrix-org#13727](matrix-org#13727)) - Define Synapse's compatability policy for SQLite versions. ([\matrix-org#13728](matrix-org#13728)) - Add docs for the common fix of deleting the `matrix_synapse.egg-info/` directory for fixing Python dependency problems. ([\matrix-org#13785](matrix-org#13785)) - Update request log format documentation to mention the format used when the authenticated user is controlling another user. ([\matrix-org#13794](matrix-org#13794)) Deprecations and Removals ------------------------- - Synapse will now refuse to start if configured to use SQLite < 3.27. ([\matrix-org#13760](matrix-org#13760)) - Don't include redundant `prev_state` in new events. Contributed by Denis Kariakin (@dakariakin). ([\matrix-org#13791](matrix-org#13791)) Internal Changes ---------------- - Add a stub Rust crate. ([\matrix-org#12595](matrix-org#12595), [\matrix-org#13734](matrix-org#13734), [\matrix-org#13735](matrix-org#13735), [\matrix-org#13743](matrix-org#13743), [\matrix-org#13763](matrix-org#13763), [\matrix-org#13769](matrix-org#13769), [\matrix-org#13778](matrix-org#13778)) - Bump the minimum dependency of `matrix_common` to 1.3.0 to make use of the `MXCUri` class. Use `MXCUri` to simplify media retention test code. ([\matrix-org#13162](matrix-org#13162)) - Add and populate the `event_stream_ordering` column on the `receipts` table for future optimisation of push action processing. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13703](matrix-org#13703)) - Rename the `EventFormatVersions` enum values so that they line up with room version numbers. ([\matrix-org#13706](matrix-org#13706)) - Update trial old deps CI to use Poetry 1.2.0. ([\matrix-org#13707](matrix-org#13707), [\matrix-org#13725](matrix-org#13725)) - Add experimental configuration option to allow disabling legacy Prometheus metric names. ([\matrix-org#13714](matrix-org#13714), [\matrix-org#13717](matrix-org#13717), [\matrix-org#13718](matrix-org#13718)) - Fix typechecking with latest types-jsonschema. ([\matrix-org#13724](matrix-org#13724)) - Strip number suffix from instance name to consolidate services that traces are spread over. ([\matrix-org#13729](matrix-org#13729)) - Instrument `get_metadata_for_events` for understandable traces in Jaeger. ([\matrix-org#13730](matrix-org#13730)) - Remove old queries to join room memberships to current state events. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13745](matrix-org#13745)) - Avoid raising an error due to malformed user IDs in `get_current_hosts_in_room`. Malformed user IDs cannot currently join a room, so this error would not be hit. ([\matrix-org#13748](matrix-org#13748)) - Update the docstrings for `get_users_in_room` and `get_current_hosts_in_room` to explain the impact of partial state. ([\matrix-org#13750](matrix-org#13750)) - Use an additional database query when persisting receipts. ([\matrix-org#13752](matrix-org#13752)) - Preparatory work for storing thread IDs for notifications and receipts. ([\matrix-org#13753](matrix-org#13753)) - Re-type hint some collections as read-only. ([\matrix-org#13754](matrix-org#13754)) - Remove unused Prometheus recording rules from `synapse-v2.rules` and add comments describing where the rest are used. ([\matrix-org#13756](matrix-org#13756)) - Add a check for editable installs if the Rust library needs rebuilding. ([\matrix-org#13759](matrix-org#13759)) - Tag traces with the instance name to be able to easily jump into the right logs and filter traces by instance. ([\matrix-org#13761](matrix-org#13761)) - Concurrently fetch room push actions when calculating badge counts. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13765](matrix-org#13765)) - Update the script which makes full schema dumps. ([\matrix-org#13770](matrix-org#13770)) - Deduplicate `is_server_notices_room`. ([\matrix-org#13780](matrix-org#13780)) - Simplify the dependency DAG in the tests workflow. ([\matrix-org#13784](matrix-org#13784)) - Remove an old, incorrect migration file. ([\matrix-org#13788](matrix-org#13788)) - Remove unused method in `synapse.api.auth.Auth`. ([\matrix-org#13795](matrix-org#13795)) - Fix a memory leak when running the unit tests. ([\matrix-org#13798](matrix-org#13798)) - Use partial indices on SQLite. ([\matrix-org#13802](matrix-org#13802)) - Check that portdb generates the same postgres schema as that in the source tree. ([\matrix-org#13808](matrix-org#13808)) - Fix Docker build when Rust .so has been built locally first. ([\matrix-org#13811](matrix-org#13811)) - Complement: Initialise the Postgres database directly inside the target image instead of the base Postgres image to fix building using Buildah. ([\matrix-org#13819](matrix-org#13819)) - Support providing an index predicate clause when doing upserts. ([\matrix-org#13822](matrix-org#13822)) - Minor speedups to linting in CI. ([\matrix-org#13827](matrix-org#13827)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmMy4FoACgkQkD7OEIo5 # 3t009g/6A4S26H6NG4GM44JD9+OB25fO59m9UAWWLrePmOKsBaGXVp86scPq9epI # vQbr4Czi8WEqCJlMRxIWLXv7BL3TLXnLF1vC0wSE6YiJqrPU9vMZ0UYxWNErl8Sr # eFBpuHXDlfppQUXs903iNmXVbdTpXCVjdTEwaZmgU1/FKydgU0o90PQseb/hnegX # hcJrepL6xhcs37fP2zdlixissLQ85WE10x6h7FX+SkCEHGvkiKrqSvXsS4ZN0Scn # vGCy3GD69j/ZpRu7RczdDwwzCRerg6r7spokRK/b6pzz9sXmCyY8SrrUyEEdzveN # uEEe4A8vmR3v0sR4Ao2cGZ7zy/jq6WyrWjmfOd+hfYD9tXx/g8190RFkRQrrkYVU # jTNdD6Zom0rtENEgHuFQ8joD96MNsaq4dvDefYYpcXDOh3YZnA/fiwfmG9XZRq6u # B42RZEtUZ3sjZ3VdRb3AvhPTTrY4kiwEqVBFSUTBKEAKXQtdrsiqv2QPvQTSC5BJ # YFXMwj32E7Zi3mTYjl60yggtBAxYt49KsHL8qAq75t6A38HpnYEyEW1R3S6VDmtp # rIR5ktxyzoGvxpJ4YPUdC4A2hbaOztwPvGE3iEDiPgUdkfb6m8x3MhaWhShid4Df # v2BQu+SFIl1wXPLxjlX2rmkQMhUGD2RGYmkUgYfoWnjdTtQV10w= # =4eYj # -----END PGP SIGNATURE----- # gpg: Signature made Tue Sep 27 12:36:58 2022 BST # gpg: using RSA key D79D3CA0B61429A8A760525A903ECE108A39DEDD # gpg: Can't check signature: No public key # Conflicts: # docker/Dockerfile # poetry.lock # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_tools.py # synapse/storage/databases/main/event_push_actions.py # synapse/storage/databases/main/events_worker.py # tests/replication/slave/storage/test_events.py

ri0t · 2022-10-02T17:01:08Z

Upgrading a roughly year old installation on Debian (from git) yields:

# synctl restart
synapse.app.homeserver not running
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/synapse/synapse/synapse/__init__.py", line 23, in <module>
    from synapse.util.rust import check_rust_lib_up_to_date
  File "/home/synapse/synapse/synapse/util/rust.py", line 20, in <module>
    from synapse.synapse_rust import get_rust_file_digest
ModuleNotFoundError: No module named 'synapse.synapse_rust'

I have installed rustc via apt-get (not soo cool to have to, version 1.60 - maybe too new?) then did pip install --upgrade . which ran without errors/problems. Am now installing tox et al to maybe get a few insights.

Edit: Everything 3.9 is borked anyway. I'm now upgrading to 3.10 first.

ri0t · 2022-10-02T18:35:13Z

Nope, not even doing a clean (pypi based) install in a new venv doesn't work, yields the same problem.

ri0t · 2022-10-02T18:48:09Z

Ah, well. A very thorough cleanup (deleting everything..!) with partial restoring from backup cleared it up.

DMRobertson · 2022-10-03T10:20:07Z

ModuleNotFoundError: No module named 'synapse.synapse_rust'

This error appears in at least two circumstances:

You have not build the Rust part of Synapse.
You have built the Rust part of Synapse, but you are running Synapse directly from its source tree. This won't work because Python is importing Synapse from the source checkout, and not where it's actually installed.

I suspect the latter in your case.

babolivier · 2022-10-03T10:44:01Z

Also: please use #synapse:matrix.org for support rather than trying to revive long-closed PRs.

erikjohnston added 2 commits August 17, 2022 17:11

Add a stub Rust crate

178d866

Ignore the .so lib

09bc570

erikjohnston force-pushed the erikj/rust_app branch from 413f82e to 09bc570 Compare August 17, 2022 16:39

Add explicit setuptools_rust dep

b4f8dcd

erikjohnston force-pushed the erikj/rust_app branch from 7911307 to b4f8dcd Compare August 18, 2022 11:48

erikjohnston added 8 commits August 18, 2022 13:11

Merge remote-tracking branch 'origin/develop' into erikj/rust_app

e9a887e

Fix lock file

6f7d07c

Use poetry build

9e50d78

Fix type hints

7c6c3fc

Fix debian packaging

fb841d4

Newsfile

7ae6651

Make a workspace

d1216ff

Add rust to docker build deps

d29b599

erikjohnston force-pushed the erikj/rust_app branch from 553b252 to d29b599 Compare August 18, 2022 14:29

reivilibre self-requested a review August 22, 2022 14:20

reivilibre reviewed Aug 22, 2022

View reviewed changes

Cargo.toml Show resolved Hide resolved

reivilibre reviewed Aug 22, 2022

View reviewed changes

pyproject.toml Show resolved Hide resolved

DMRobertson reviewed Aug 22, 2022

View reviewed changes

reivilibre reviewed Aug 22, 2022

View reviewed changes

tests/test_rust.py Show resolved Hide resolved

reivilibre previously approved these changes Aug 22, 2022

View reviewed changes

Use recent poetry

0ce8a1d

erikjohnston added 2 commits August 24, 2022 14:04

Install rust toolchains

bc119dd

Set toolchain

1036994

erikjohnston added 2 commits September 6, 2022 16:47

Use a specific rustc version in tests

1a601fa

Add typing

fa8f079

erikjohnston requested a review from DMRobertson September 6, 2022 15:51

DMRobertson reviewed Sep 6, 2022

View reviewed changes

erikjohnston added 2 commits September 6, 2022 16:54

Temporarily always build sdist

bbd3d09

Fix sdist

43e38a9

DMRobertson approved these changes Sep 6, 2022

View reviewed changes

erikjohnston added 3 commits September 6, 2022 17:12

Fix sdist

fa57b03

Fix sdist warning

bc8f80a

Only build sdist on non-PRs CI

19c6a5f

erikjohnston merged commit c9b7e97 into develop Sep 6, 2022

erikjohnston deleted the erikj/rust_app branch September 6, 2022 18:01

DMRobertson reviewed Sep 6, 2022

View reviewed changes

jplatte reviewed Sep 7, 2022

View reviewed changes

erikjohnston mentioned this pull request Sep 7, 2022

Fix latest deps CI #13734

Merged

DMRobertson mentioned this pull request Sep 7, 2022

Call out changes in Synapse 1.68 requirements when announcing Synapse 1.67 #13744

Closed

2 tasks

This was referenced Sep 22, 2022

Fix sdist to include rust build script #13866

Merged

Correctly add Cargo.lock to sdist #13909

Merged

DMRobertson mentioned this pull request Oct 7, 2022

Parse SYNAPSE_ASYNC_IO_REACTOR env variable & log the reactor on startup #14092

Merged

erikjohnston mentioned this pull request Oct 25, 2022

Cache rust build deps in trial CI #14287

Merged

clokep mentioned this pull request Jul 11, 2023

Don't build wheels for Python 3.7 #15917

Merged

		@@ -0,0 +1,5 @@
		# We make the whole Synapse folder a workspace so that we can run `cargo`

Add a stub Rust crate #12595

Add a stub Rust crate #12595

Conversation

erikjohnston commented Apr 29, 2022

DMRobertson commented Apr 29, 2022

erikjohnston commented Apr 29, 2022

erikjohnston commented Aug 17, 2022

erikjohnston commented Aug 18, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

erikjohnston commented Aug 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DMRobertson left a comment

Choose a reason for hiding this comment

erikjohnston commented Sep 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NickCao commented Sep 21, 2022

DMRobertson commented Sep 21, 2022

ri0t commented Oct 2, 2022 • edited Loading

ri0t commented Oct 2, 2022

ri0t commented Oct 2, 2022

DMRobertson commented Oct 3, 2022

babolivier commented Oct 3, 2022

ri0t commented Oct 2, 2022 •

edited

Loading