How to use a global runtime for all tests?

[ufoscout]

Is it possible to use the same runtime instance to run all tests?

The use case:
An instance of my application is created and stored in a lazy_static object to be used by all tests; this application internally uses an async connection pool based on tokio.

The problem:
[tokio::test] creates a new runtime for each test, so:

• the async connection pool of the application is initialized when the first test accesses it
• the async connection pool is then bound to the runtime of the first test
• when the first test ends, its runtime is closed and consequently, the pool is also closed
• then, all other tests fail because the pool is closed.

To solve this I would like [tokio::test] to use a single runtime instance for all tests.

For example, an idea could be:

lazy_static! {
    static ref MY_APP: MyApp = MyApp::default();
 }

#[tokio::test(runtime = "rt()")] // <-- HERE I WOULD LIKE TO PASS A CUSTOM RUNTIME
async fn test_1() {
   MY_APP.do_something().await;
}

#[tokio::test(runtime = "rt()")] // <-- HERE I WOULD LIKE TO PASS A CUSTOM RUNTIME
async fn test_2() {
   MY_APP.do_something().await;
}

fn rt() -> &'static tokio::runtime::Runtime {
    lazy_static! {
        static ref RT: tokio::runtime::Runtime = tokio::runtime::Builder::new()
        .threaded_scheduler()
        .enable_all()
        .build()
        .expect("Should create a tokio runtime");
    }
    &RT
}

[LucioFranco]

I would suggest that using a global connection pool like that is generally an anti pattern and generates surprising effects. I personally would suggest removing the global pool and using the same tokio::test macro. This is by far the best way to reduce tests affecting each other in weird ways.

[ufoscout]

@LucioFranco
The application and the pool are global/static only for the integration tests because considering that the app starts in a couple of seconds and that we have thousands of tests, then the execution time of the tests would grow considerably if we restart the app for each test.

In addition, in this particular case, the application is a web server that uses postgres as the storage layer and, of course, it internally manages all accesses to the DB through a connection pool.
As the web server is supposed to be completely stateless, an eventual side effect of a test on another one would mean that there is a bug in the application; so, I would be extremely happy to find it during the tests instead of in production.
In fact, we have also a set of tests explicitly written to try to identify bad usage of the cache for example.

[LucioFranco]

So if you have a global runtime you can just in your tests do

RT.block_on(async move {
	// test
})

which is basically what the macro does but because of certain internals block_on takes &mut self, so you will want to do the Handle::current + futures_executor::block_on trick.

Overall, I still suggest spinning up a single connection pool per test anyways, I don't suspect this should be too expensive if you only need say one connection per test?

[ufoscout]

@LucioFranco
thanks! I ended up with something like:

pub fn test<F: std::future::Future>(f: F) -> F::Output {
    rt().handle().enter(|| futures::executor::block_on(f))
}

#[test]
fn a_test() {
    test(async {
        println!(" this is async");
    });
}

Anyway, it would be great to have it on tokio itself so there would be no need for all that boilerplate code and the nested "test()" call.

[LucioFranco]

I think we are likely to not support something like that since we generally want to promote the usage of tokio::test since in my opinion that is the right way. I am going to close this issue, please re-open if you have any more questions! Hope everything works! :)

[casey]

I believe I have run into a scenario where I am truly unable to use a per-test runtime as set up by #[tokio::test].

I am writing a Discord bot. Discord's API is large and complex, so I can't mock it. I also can't run it locally, because it is proprietary. So, my tests must run against the live API.

Let's call the type that represents a session with the Discord API as a Session.

Discord rate-limits log in to once every 5 seconds. So I can't create a new Session in each test.

I tried to use a global Session object, however, I believe that this is problematic because the session will be created in the runtime the #[tokio::test]-annotated test that initializes it, and the rest of the tests will be using a Session object that has been initialized in a different runtime.

I'm not sure, but I've seen some surprising errors, like this one, which I believe are being caused by running futures on the wrong runtime.

So, I think I have no choice but to use a global runtime.

I understand the desire to guide users towards #[tokio::test], however, it would be nice to provide a way to use #[tokio::test] with a global runtime, when needed. Not being able to use the macro, and having to wrap every test in a helper function, introducing another level of indentation, is less than ideal.

[casey]

I ran into another situation where I was forced to use a global runtime for all tests. The tests in question were browser tests, which use the chromiumoxide crate to interact with a headless browser. Initializing a browser is slow, so I needed to share a browser instance between all tests. However, the browser instance must run on the same runtime as the test, so I had to remove #[tokio::test] and use a shared, global runtime.

Any chance this issue could get a second look? So far, it's been a pretty common (2 for 2) that I've needed to use a global runtime when writing async tests, so I suspect it's common for others. It would be nice if there was an easy way to do this, without needing to use something like lazy_static to initialize a global runtime, and doing:

#[test]
fn test() {
  RUNTIME.block_on(async {
    // test code
  });
}

In every test.

[Darksonn]

I am open to suggestions for how to achieve a shared runtime.

[casey]

How about a shared_runtime argument to the tokio::test attribute that causes the test to run with a lazily initialized shared runtime:

#[tokio::test(shared_runtime)]
async fn test() {
  ...
}

[jsoneaday]

I'm doing something similar to @ufoscout as shown below, but not sure I understand what's happening. In the code below I'm lazy initializing a static variable RT by creating a new runtime and then running some async setup code inside of the rt.block_on call. This code runs as expected and blocks until its async closure completes entirely.

However the subsequent function run_test has a call test_rt().block_on, which is what I use to execute my actual tests, and these calls return immediately without waiting for the internal future to complete.

What is the difference between the two calls, why does the first one wait and the other does not?

pub fn test_rt() -> &'static tokio::runtime::Runtime {
    lazy_static! {
        static ref RT: tokio::runtime::Runtime = {
            println!("log: start setup RT");
            let rt = tokio::runtime::Builder::new_current_thread()
                .enable_all()
                .build()
                .unwrap();

            rt.block_on(async {
                println!("log: start setup_test_db_data()");
                let conn = get_conn_pool().await;
                _ = setup_test_db_data(&conn).await;
                println!("log: end setup_test_db_data()");
            });
            println!("log: end setup RT");

            rt
        };
    }
    
    &RT
}

pub fn run_test<F: std::future::Future>(f: F) -> F::Output {
    test_rt().block_on(f)
}

[gwy15]

I ran into this problem too and created this crate to provide #[tokio_shared_rt::test] that acts like #[tokio::test] but uses a shared runtime.

https://crates.io/crates/tokio-shared-rt

#[tokio_shared_rt::test]
async fn t1() {
    db_pool().foo().await;
}
#[tokio_shared_rt::test(shared)]
async fn t2() {
    db_pool().foo().await;
}
#[tokio_shared_rt::test(shared = true)]
async fn t3() {
    db_pool().foo().await;
}
#[tokio_shared_rt::test(shared = false)]
async fn delicate_runtime_test() {
    db_pool().foo().await;
}

[elichai]

I am open to suggestions for how to achieve a shared runtime.

I'd love to hear what the problems are preventing just sticking it in a OnceCell might be worth it at least for tests. is it because the runtime will never be dropped?

[txbm]

I am also facing a recent testing issue that would benefit from Tokio supporting a parameter to enable a shared runtime between tests. I am surprised that this feature has not already been implemented given how commonly it seems to be needed for a variety of use cases.

[Tockra]

I ran into a similar issue.
I want to setup a test-container for the whole test suite. For that I'm using a OnceCell. But the problem here is that the container is never dropped. So everytime I do this, I have a running docker container as left-over:

static KEYCLOAK: OnceCell<(ContainerAsync<GenericImage>, String)> = OnceCell::const_new();

async fn start_auth_container() -> &'static (ContainerAsync<GenericImage>, String) {
    KEYCLOAK
        .get_or_init(|| async { start_keycloak_container().await })
        .await
}

#[tokio::test]
async fn test1(
) {
    let keycloak = start_auth_container().await;

    println!("{:?}", keycloak.1);
    //let app = start_app("some").await;
}

#[tokio::test]
async fn test2(
) {
    let keycloak = start_auth_container().await;

    println!("{:?}", keycloak.1);
    //let app = start_app("some").await;
}

A shared runtime, where I could do some specific clean up call at the end, would be a great solutions and other testing frameworks like junit have such stuff for years...