common crate, expose more tests in wasm #1366

insipx · 2024-12-03T21:38:44Z

This PR adds a common crate & some utilities for testing WebAssembly. It mostly moves things around and should not effect functionality other than a few minor bug fixes for tests which previously did not run in web assembly

the bug fixes:

fixed is_ready in register_identity to toggle once the identity signature has been verified with backend, fixing SequenceId errors
fixed some webassembly tests hanging because of sleep implementation, replaced with yield
fixed Notify/Delivery test utilities panicking streams in webassembly environment because of std::time, replaced with xmtp_common::time

Common Crate:

creates common crate that houses:
- the rand_xx fns that were duplicated across a few crates
- web assembly time:: shims, that are now accessible via xmtp_common::time. It should be a drop-in replacement for std::time, and preferable to use instead of std::time
- xmtp_common::yield_ which delegates to tokio or the browsers executor depending on environment. Should be preferred over tokio::task::yield, since a WebAssembly future may not necessarily be running in a tokio executor
- retry moved from xmtp_mls to common. Retry is general enough that this makes sense, and allows it to be used without including all of xmtp_mls.
- traced_test moved from xmtp_mls to common. Also general enough to be used w/o mls, and useful elsewhere
- xmtp_common includes a common logger initializer toggling on arch, xmtp_common::logger
assert_x and optify macros moved to common
new test utilities in store/mod.rs create triggers to track intent metadata (created/published/deleted) and exposed for tests on DbConnection
replaced some uses of cfg_attr/cfg_attr pair, with the more succinct wasm_bindgen_test(unsupported = tokio::test) macro

insipx · 2024-12-03T21:39:00Z

common crate, expose more tests in wasm #1366 : 2 dependent PRs (#1376 , #1392 ) 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

insipx · 2024-12-11T23:18:02Z

xmtp_mls/src/groups/device_sync/consent_sync.rs

        storage::consent_record::{ConsentState, ConsentType},
        utils::test::wait_for_min_intents,
    };
    use xmtp_cryptography::utils::generate_local_wallet;
    use xmtp_id::InboxOwner;

-    #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[wasm_bindgen_test(unsupported = tokio::test(flavor = "multi_thread", worker_threads = 1))]
+    #[cfg_attr(target_family = "wasm", ignore)]


these tests (and all of device sync on web) is blocked on xmtp/xmtp-message-history-server#6, then the ignore can be removed

xmtp_mls/src/groups/device_sync/consent_sync.rs

cameronvoell · 2024-12-12T01:19:13Z

xmtp_mls/src/storage/mod.rs

+    use super::*;
+    use diesel::{connection::LoadConnection, deserialize::FromSqlRow, sql_query, RunQueryDsl};
+    impl DbConnection {
+        /// Create a new table and register triggers for tracking column updates


this is a cool idea, nice 😎

cameronvoell

awesome problem solving on the browser executor for streaming. and clean up in this pr is fantastic thanks! 🥇

codabrink · 2024-12-12T14:55:28Z

common/src/lib.rs

+pub fn rand_array<const N: usize>() -> [u8; N] {
+    let mut buffer = [0u8; N];
+    crypto_utils::rng().fill_bytes(&mut buffer);
+    buffer
+}


That's cool. I hadn't thought about using a generic like that before.

codabrink · 2024-12-12T15:00:56Z

common/src/test.rs

+    Fut: Future<Output = T>,
+    T: std::fmt::Debug + PartialEq,
+{
+    let result = crate::time::timeout(crate::time::Duration::from_secs(60), async {


Can we reduce the default timeout to something like 2-3 seconds? Otherwise tests will take a long time to fail.

I can reduce it a bit, but reducing too much causes tests in webassembly to flake out. I figure the common case of test success it will still finish quickly, but we need to give liberal time for wasm. I'll lower it to ~20sec?

codabrink

Thank you for making this. Huge QoL improvement.

insipx mentioned this pull request Dec 4, 2024

Start the sync worker in the builder #1365

Merged

insipx force-pushed the 11-30-common_crate_for_wasm_shims branch from c33b5ba to cf10919 Compare December 4, 2024 19:41

insipx mentioned this pull request Dec 4, 2024

feat(wasm): unblock streams in the browser #1376

Closed

insipx force-pushed the 11-30-common_crate_for_wasm_shims branch from cf10919 to f8d41b5 Compare December 6, 2024 19:45

insipx mentioned this pull request Dec 6, 2024

benchmarks for create_client #1392

Merged

insipx force-pushed the 11-30-common_crate_for_wasm_shims branch 10 times, most recently from 3df89c2 to 919f767 Compare December 11, 2024 16:25

insipx mentioned this pull request Dec 11, 2024

Take advantage of sqlite3_update_hook in testing #1405

Closed

insipx force-pushed the 11-30-common_crate_for_wasm_shims branch 5 times, most recently from 718950b to 9b9da95 Compare December 11, 2024 23:12

insipx commented Dec 11, 2024

View reviewed changes

insipx force-pushed the 11-30-common_crate_for_wasm_shims branch from 9b9da95 to 0b8d1fe Compare December 11, 2024 23:19

common crate for wasm shims

fd98665

insipx force-pushed the 11-30-common_crate_for_wasm_shims branch from 0b8d1fe to fd98665 Compare December 11, 2024 23:31

insipx changed the title ~~common crate for wasm shims~~ common crate for wasm, expose more tests in wasm Dec 11, 2024

insipx changed the title ~~common crate for wasm, expose more tests in wasm~~ common crate, expose more tests in wasm Dec 11, 2024

insipx marked this pull request as ready for review December 11, 2024 23:40

insipx requested a review from a team as a code owner December 11, 2024 23:40

cameronvoell reviewed Dec 12, 2024

View reviewed changes

xmtp_mls/src/groups/device_sync/consent_sync.rs Outdated Show resolved Hide resolved

cameronvoell reviewed Dec 12, 2024

View reviewed changes

cameronvoell approved these changes Dec 12, 2024

View reviewed changes

codabrink reviewed Dec 12, 2024

View reviewed changes

codabrink approved these changes Dec 12, 2024

View reviewed changes

feedback

54cd430

insipx merged commit 600a8e2 into main Dec 12, 2024
16 checks passed

insipx deleted the 11-30-common_crate_for_wasm_shims branch December 12, 2024 16:38

This was referenced Dec 12, 2024

Write some Message/Consent/Device Sync History integration tests for WebAssembly #1201

Closed

Separate grpc channel connection from create_client #1415

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

common crate, expose more tests in wasm #1366

common crate, expose more tests in wasm #1366

insipx commented Dec 3, 2024 •

edited

Loading

insipx commented Dec 3, 2024 •

edited

Loading

insipx Dec 11, 2024 •

edited

Loading

cameronvoell Dec 12, 2024

cameronvoell left a comment

codabrink Dec 12, 2024

codabrink Dec 12, 2024 •

edited

Loading

insipx Dec 12, 2024

codabrink left a comment

common crate, expose more tests in wasm #1366

common crate, expose more tests in wasm #1366

Conversation

insipx commented Dec 3, 2024 • edited Loading

Common Crate:

insipx commented Dec 3, 2024 • edited Loading

insipx Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

cameronvoell Dec 12, 2024

Choose a reason for hiding this comment

cameronvoell left a comment

Choose a reason for hiding this comment

codabrink Dec 12, 2024

Choose a reason for hiding this comment

codabrink Dec 12, 2024 • edited Loading

Choose a reason for hiding this comment

insipx Dec 12, 2024

Choose a reason for hiding this comment

codabrink left a comment

Choose a reason for hiding this comment

insipx commented Dec 3, 2024 •

edited

Loading

insipx commented Dec 3, 2024 •

edited

Loading

insipx Dec 11, 2024 •

edited

Loading

codabrink Dec 12, 2024 •

edited

Loading