test_with::env executes even if env var is unset

[WhyNotHugo]

I've checked out #55 and I have test with macros in this order:

#[serial]
#[test_with::env(CALDAV_SERVER)]
#[tokio::test]
async fn test_fetch_missing() {

However, the test is executed even if CALDAV_SERVER is unset. This is my dependency tree:

[dependencies]
async-trait = "0.1.66"
base64 = "0.21.0"
domain = { version = "0.7.2", features = ["resolv"] }
http = "0.2.9"
hyper = { version = "0.14.24", features = ["http1", "client"] }
hyper-rustls = "0.23.2"
log = "0.4.17"
quick-xml = { version = "0.27.0" }
thiserror = "1.0.38"
tokio = "1.25.0"

[dev-dependencies]
env_logger = "0.10.0"
rand = "0.8.5"
serial_test = { version = "1.0.0", default-features = false, features = ["async"] }
test-with = { version = "0.9.6", default-features = false }

Any ideas? The full repo is at https://git.sr.ht/~whynothugo/vdirsyncer-rs, branch test_with.

[yanganto]

hmm... I check on this, with following files.

// src/main.rs
#[serial_test::serial]
#[test_with::env(CALDAV_SERVER)]
#[tokio::test]
async fn test_fetch_missing() {}

# Cargo.toml
[dependencies]
tokio = { version = "1.25.0", features = ["full"] }

[dev-dependencies]
serial_test = { version = "1.0.0", default-features = false, features = ["async"] }
test-with = { version = "0.9.6", default-features = false }

running 1 test
test test_fetch_missing ... ignored, because variable CALDAV_SERVER not found

test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s

Could you do cargo clean && cargo test for me again?

If it still have issue, could you let me know about what rust version you are using?

[WhyNotHugo]

Huh, I ran cargo clean and now things have changed:

On the terminal where CALDAV_SERVER is unset the test was correctly ignored. But then switching back to the terminal where CALDAV_SERVER is set:

> cargo test test_fetch_missing
    Finished test [unoptimized + debuginfo] target(s) in 0.05s
     Running unittests src/lib.rs (target/debug/deps/libdav-98ea0c7ef4059abc)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.00s

     Running tests/caldav.rs (target/debug/deps/caldav-63cd50a1eca166b7)

running 1 test
test test_fetch_missing ... ignored, because variable CALDAV_SERVER not found

test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 3 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/vdirsyncer-430987d2581c901c)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/vstorage-fe538d3663535938)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out; finished in 0.00s


> echo $CALDAV_SERVER
http://localhost:8000

[WhyNotHugo]

Is it possible that the variable is being checked at build-time and not at runtime?

[yanganto]

Yeh, I hope I can solve this in runtime, so I write it down in the Readme and also make a post on Reddit.
https://www.reddit.com/r/rust/comments/10fudip/integration_test_issue_in_rust_development/

A lot of people want this issue to be solved.
rust-lang/rust#68007

And there is also an implementation but not accepted
rust-lang/rust#96574

I hope the Rust team can consider this, but @Mark-Simulacrum is against solving the issue here.
rust-lang/rust#68007 (comment)

Somehow Rust is a little away from a useful system programing language. If this is not a reasonable use case, people will not write a macro inside a string like this.
https://slides.com/yanganto/rust-ignore#/4/1

[yanganto]

BTW, the runtime issue is already tracked by another issue, if there is nothing more please close it. Thanks.

[WhyNotHugo]

It's right there in the readme, I must have missed it 🤦

Sorry for the noise.

[yanganto]

@WhyNotHugo no worries, it is not intuitive.

However, for most cases, you will change the code in development, or start empty in the CI server, you only need to cargo test.