Cleanup Testing code #471

josephlr · 2024-06-10T11:38:29Z

Depends on #473

This stops using weird include hacks to reuse testing code. Instead, we move the code to a normal tests.rs file and uses helper functions to avoid code duplication. Now when running cargo test the tests appear as normal unit tests:

   Compiling getrandom v0.2.15 (/home/joe/dev/rust/getrandom)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running unittests src/lib.rs (target/debug/deps/getrandom-1d050696e8dd96df)

running 6 tests
test error::tests::test_size ... ok
test tests::fill_large ... ok
test tests::fill_zero ... ok
test tests::multithreading ... ok
test tests::fill_small ... ok
test tests::fill_huge ... ok

Follow up PRs (which explain some of the design logic in this PR):

Adds back RDRAND testing: Test RDRAND on x86 / x86_64 targets #476
Tests getrandom_uninit: Test getrandom_uninit #474
Tests file-based fallback: Test fallback file implementation on Linux/Android #475

josephlr · 2024-06-10T12:02:44Z

CC @briansmith who is doing other testing improvements.

josephlr · 2024-06-10T12:07:08Z

Haiku failure is unrelated, fixed by from rust-lang/rust#126212

src/lib.rs

briansmith

This PR tries to do three things, AFAICT:

Refactoring how common logic is shared between tests.
Add support for testing the _uninit() API directly.
Change how we test custom implementations and what is being tested in those cases.

Each of these types of changes should be its own PR with its own discussion. I think if I had submitted this PR, nobody would want to review it because it mixes all of these things.

src/rdrand.rs

src/tests.rs

josephlr · 2024-06-12T01:47:25Z

This PR tries to do three things, AFAICT:
* Refactoring how common logic is shared between tests.

* Add support for testing the `_uninit()` API directly.

* Change how we test custom implementations and what is being tested in those cases.
Each of these types of changes should be its own PR with its own discussion. I think if I had submitted this PR, nobody would want to review it because it mixes all of these things.

This is a very good point. I've reduced the scope of this PR (see the updated description), and I've split out some of the prerequisites into their own PRs (#473, #472). I've also moved testing of RDRAND / getrandom_uninit / file-fallback to followup PRs (#476 , #475 , #474).

@newpavlov @briansmith let me know if you thing the Byte-based generic code makes sense. Having a single method which is "give me a random Vec<u8>" seemed like the easiest way to express the differences between the u8 and MaybeUninit<u8> code.

@briansmith

This simplifies lib.rs and allows for our testing code to be less convoluted. Split out from #471. CC @briansmith The MSRV for rand 0.9 is 1.61 and Debian stable is 1.63, so this is a fine MSRV bump. Signed-off-by: Joe Richey <[email protected]>

newpavlov · 2024-06-19T12:02:08Z

I would prefer to keep bulk of our tests in the tests/ folder. I will try to experiment with the testing code a bit later.

briansmith

This PR is hard to reivew in the GitHub UI because the move of tests/common/mod.rs to src/tests.rs is combined with extensive edits of the file. If you could, please split moving the unmodified file into a separate commit and then rebase this commit over it, so that the PR consists of two commits. Then each commit in the PR can be reviewed in the GitHub UI individually.

briansmith · 2024-06-19T17:50:30Z

src/lib.rs

+    test,
+    not(all(target_family = "wasm", target_os = "unknown", feature = "custom"))
+))]
+pub(crate) mod tests;


I think the "normal" tests should still be defined in tests/normal.rs (See comment below).

I think we're all keen to avoid duplicating complex cfg all over. I suggest we add a new api:

/// true if the implementation defined by `register_custom_getrandom!` would be used; false if it will be ignored. pub const CUSTOM_USED: bool;

Then we can condition all the entropy checks in the common tests on that new API.

I think this API would also be useful to external users.

I also expected to see here something like:

cfg_if::cfg_if! { if #[cfg(any(target_arch = "x86", target_arch = "x86_64")] { mod rdrand; #[cfg(test)] mod rdrand_tests { tests::define_tests!(rdrand::getrandom_uninit); } } }

This way, tests/rdrand.rs is properly replaced instead of just being lost.

Then presumably in future PRs we would do the same for use_file, etc.

I think we're all keen to avoid duplicating complex cfg all over. I suggest we add a new api:

After rearranging some stuff, this cfg is no longer needed. We can just do #[cfg(test)] and everything works as expected. Note that the CI tests are updated to run wasm-pack test --node --features=custom --test=custom when testing on an unsupported platform. This makes more sense to me, as the in-library unit tests will not build if you just enable the custom feature on an unsupported platform (you have to register a custom handler, which is only done in our integration test).

I also expected to see here something like:

cfg_if::cfg_if! { if #[cfg(any(target_arch = "x86", target_arch = "x86_64")] { mod rdrand; #[cfg(test)] mod rdrand_tests { tests::define_tests!(rdrand::getrandom_uninit); } } }

This way, tests/rdrand.rs is properly replaced instead of just being lost.

I changed this PR to just unconditionally have mod rdrand present if the "rdrand" feature is enabled. This is similar to what we do for "custom", and allows us to just have normal unit tests inside rdrand.rs. We can't put such things inside the cfg_if, but having the mod rdrand outside of that macro ends up being more readable anyway.

briansmith · 2024-06-19T17:52:39Z

src/tests.rs

+}
+
+// A function which fills a buffer with random bytes.
+type FillFn<B> = fn(&mut [B]) -> Result<(), Error>;


I think this should be type FillFn<B, R> = fn(&mut [B]) -> Result<R, Error> so that all of getrandom::getrandom_uninit, getrandom::getrandom, and the internal getrandom_impl functions can all be used with it.

Unfortunately, this doesn't work. Due to lifetime-related problems, there's no way to have a type alias which refers to the type of all 3 of those functions. Specifically, there's no R for which:

getrandom_uninit: FillFn<B,R>

as the full type is:

getrandom_uninit: for<'a> fn(&'a mut [MaybeUninit<u8>]) -> Result<&'a mut [u8], Error>

Instead, I changed things to have FillFn be a trait (implemented by the relevant Fn objects) which defines a make_vec method. The only downside is that it requires a dummy generic parameter to avoid conflicting implementations. Let me know what you think.

briansmith · 2024-06-19T17:53:43Z

src/tests.rs

+}
+pub(crate) use define_tests;
+
+define_tests!(crate::getrandom);


I would expect that instead of having this line here, we could have it in tests/normal.rs:

#[path = "../src/tests.rs"] mod tests; mod init { define_tests!(getrandom::getrandom); } mod uninit { define_tests!(getrandom::getrandom_uninit); }

This way we're defining integration tests in the normal way.

Conceptually, I consider the tests created by define_tests! to be unit tests rather than integration tests (they just happen to test the public API), so it makes more sense to me for them to use the unit testing framework. Having them as unit tests also avoids #[path = "..."] shenanigans (which can be brittle and annoying at times).

Signed-off-by: Joe Richey <[email protected]>

josephlr · 2024-06-20T12:40:19Z

This PR is hard to reivew in the GitHub UI because the move of tests/common/mod.rs to src/tests.rs is combined with extensive edits of the file. If you could, please split moving the unmodified file into a separate commit and then rebase this commit over it, so that the PR consists of two commits. Then each commit in the PR can be reviewed in the GitHub UI individually.

Split into 3 commits:

Rename/Delete files
Common tests for getrandom and getrandom_uninit
Restoring RDRAND testing

I also closed #474 and #476 as those changes seem to work better as part of this PR (#475 is still its own PR).

briansmith

The main objection I still have is moving all these tests from integration tests to unit tests. The #[path] stuff is suboptimal and we should reduce our usage of it. But, I think we should also still stick to preferring having integration tests for getrandom::getrandom and getrandom::getrandom_uninit and more generally the whole public API.

briansmith · 2024-06-28T19:21:07Z

src/tests.rs

+
+        #[test]
+        fn zero() {
+            $fill.make_vec(0);


getrandom::getrandom_uninit() does the !dest.is_empty() check before to avoid calling getrandom_inner with an empty dest, custom::getrandom_inner() doesn't have a similar check, but also we guarantee that the custom implementation will never be called with a zero-length dest. But, having this test here implies to me that we do expect every getrandom_inner() to correctly handle an empty dest. It seems contradictory. Perhaps instead zero() should be an integration test that tests specifically/only getrandom::getrandom() and getrandom::getrandom_uninit()? Or, perhaps we should move that check out of getrandom_uninit and into the getrandom_inner implementations since that would be more robust.

Regardless of how we address this, we should have a comment here explaining what we're expecting this test to be testing.

briansmith · 2024-06-28T19:38:46Z

src/tests.rs

+        fn zero() {
+            $fill.make_vec(0);
+        }
+        #[test]


Let's separate the functions inside the macro body with blank lines.

briansmith · 2024-06-28T19:39:53Z

src/tests.rs

+        }
+        #[test]
+        fn huge() {
+            $fill.make_vec(100_000);


We need a comment here about why were not using check_diff_large

newpavlov · 2024-10-07T14:15:06Z

I think #504 has resolved most of the issues targeted by this PR.

josephlr requested a review from newpavlov June 10, 2024 11:38

josephlr mentioned this pull request Jun 10, 2024

Test getrandom_uninit() with initially-uninitialized buffer. #462

Closed

josephlr added the testing label Jun 10, 2024

newpavlov approved these changes Jun 10, 2024

View reviewed changes

src/lib.rs Outdated Show resolved Hide resolved

briansmith suggested changes Jun 10, 2024

View reviewed changes

src/rdrand.rs Outdated Show resolved Hide resolved

src/tests.rs Outdated Show resolved Hide resolved

src/tests.rs Outdated Show resolved Hide resolved

josephlr force-pushed the testing branch from 6e9520f to 731567f Compare June 12, 2024 01:04

This was referenced Jun 12, 2024

MSRV 1.60: Conditionally import lasy.rs module #472

Merged

Cleanup Custom Tests #473

Merged

josephlr force-pushed the testing branch from 731567f to e350e3b Compare June 12, 2024 01:16

josephlr mentioned this pull request Jun 12, 2024

Test getrandom_uninit #474

Closed

josephlr force-pushed the testing branch from e350e3b to e2cd3f8 Compare June 12, 2024 01:35

This was referenced Jun 12, 2024

Test fallback file implementation on Linux/Android #475

Closed

Test RDRAND on x86 / x86_64 targets #476

Closed

josephlr force-pushed the testing branch 5 times, most recently from bae28dd to 14ba262 Compare June 19, 2024 10:26

josephlr requested review from briansmith and newpavlov June 19, 2024 10:26

briansmith suggested changes Jun 19, 2024

View reviewed changes

josephlr force-pushed the testing branch 2 times, most recently from 79f3444 to a04b6f3 Compare June 20, 2024 08:31

Move common integration tests to unit tests

788d6d5

Signed-off-by: Joe Richey <[email protected]>

josephlr force-pushed the testing branch from a04b6f3 to f078f64 Compare June 20, 2024 12:12

Define common tests via define_tests! macro

71a2ab8

Signed-off-by: Joe Richey <[email protected]>

josephlr force-pushed the testing branch from f078f64 to bfad968 Compare June 20, 2024 12:16

Test RDRAND

b028603

Signed-off-by: Joe Richey <[email protected]>

josephlr force-pushed the testing branch from bfad968 to b028603 Compare June 20, 2024 12:23

josephlr requested a review from briansmith June 21, 2024 01:45

briansmith suggested changes Jun 28, 2024

View reviewed changes

newpavlov closed this Oct 7, 2024

newpavlov deleted the testing branch October 11, 2024 00:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cleanup Testing code #471

Cleanup Testing code #471

josephlr commented Jun 10, 2024 •

edited

Loading

josephlr commented Jun 10, 2024

josephlr commented Jun 10, 2024

briansmith left a comment

josephlr commented Jun 12, 2024

newpavlov commented Jun 19, 2024

briansmith left a comment

briansmith Jun 19, 2024

josephlr Jun 20, 2024

briansmith Jun 19, 2024

josephlr Jun 20, 2024

briansmith Jun 19, 2024

josephlr Jun 20, 2024

josephlr commented Jun 20, 2024

briansmith left a comment

briansmith Jun 28, 2024

briansmith Jun 28, 2024

briansmith Jun 28, 2024

newpavlov commented Oct 7, 2024

Cleanup Testing code #471

Cleanup Testing code #471

Conversation

josephlr commented Jun 10, 2024 • edited Loading

josephlr commented Jun 10, 2024

josephlr commented Jun 10, 2024

briansmith left a comment

Choose a reason for hiding this comment

josephlr commented Jun 12, 2024

newpavlov commented Jun 19, 2024

briansmith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

josephlr commented Jun 20, 2024

briansmith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

newpavlov commented Oct 7, 2024

josephlr commented Jun 10, 2024 •

edited

Loading