Add environment variable query #130883

madsmtm · 2024-09-26T15:10:05Z

Generally, rustc prefers command-line arguments, but in some cases, an environment variable really is the most sensible option. We should make sure that this works properly with the compiler's change-tracking mechanisms, such that changing the relevant environment variable causes a rebuild.

This PR is a first step forwards in doing that.

Part of the work needed to do #118204, see #129342 for some discussion.

r? @petrochenkov

compiler/rustc_interface/src/passes.rs

madsmtm · 2024-09-26T15:11:37Z

compiler/rustc_interface/src/passes.rs

@@ -336,6 +337,20 @@ fn early_lint_checks(tcx: TyCtxt<'_>, (): ()) {
    )
 }

+fn env_var(tcx: TyCtxt<'_>, key: Symbol) -> Option<Symbol> {
+    let var = match std::env::var(key.as_str()) {


Is there something similar to Clippy's disallowed-methods that I could use to lint against usage of std::env::var in the compiler, and direct users towards the query instead?

See also rust-lang/cargo#11588.

There's some rustc-internal lints in https://github.com/rust-lang/rust/blob/master/compiler/rustc_lint/src/internal.rs

madsmtm · 2024-09-27T16:58:55Z

compiler/rustc_interface/src/passes.rs

+    // Also add the variable to Cargo's dependency tracking
+    tcx.sess.psess.env_depinfo.borrow_mut().insert((key, var));


Hmm, a complication here is that this query will only work with the dependency info files that Cargo reads if it's called before passes::write_dep_info at:

rust/compiler/rustc_driver_impl/src/lib.rs

Line 437 in a3f76a2

passes::write_dep_info(tcx);

So everything done during passes::analysis and Linker::codegen_and_build_linker will need some other mechanism for populating ParseSess::env_depinfo.

Is it possible for us to move the write_dep_info pass to the very end, or just before linking? Or is it important that it happens as soon as possible for parallelizing Cargo, or something?

Or is it important that it happens as soon as possible for parallelizing Cargo, or something?

It is important for depinfo to be generated as early as possible, without performing full compilation.
All file dependencies in particular, and env var dependencies from env! and proc macros are ready after name resolution is completed.

I think it's fine to still lose some env depinfo in this PR, it will still better than what we have now.
Could you add a comment to this code, explaining that env vars added after the call to write_dep_info will be lost from the dep file?

(We can think what to do with the Apple env vars specifically later.)

I have added a comment to the env_var query, and added rustc_codegen_ssa::register_environment_variables as an extension point for where to put this. Suggestions very welcome on how to do it differently!

compiler/rustc_borrowck/src/nll.rs

petrochenkov · 2024-10-22T12:56:32Z

compiler/rustc_middle/src/query/mod.rs

+    /// detection and target-dependent compiler flags.
+    ///
+    /// Will emit an error and return `None` if the variable is not UTF-8.
+    query env_var(key: Symbol) -> Option<Symbol> {


Is it possible to use OsStr(ing)s in query inputs and outputs? What are the obstacles?
Many places in the compiler use env::var_os.

Also, Symbols shouldn't be needed here, you can take a OsStr/str and return &'tcx OsString/&'tcx String like env::var(_os) returns.

OsStr isn't really set up with the query system, so we'd have to do that first. I've done parts of it in the last commit, but it also needs to be Key, which I'm unsure would be correct? str for example isn't Key either.

Also, returning &'tcx OsStr requires that the environment variables are actually stored inside TyCtxt<'tcx>, which I'm unsure how to do wrt. lifetimes?

compiler/rustc_middle/src/query/mod.rs

…g, r=petrochenkov Move versioned Apple LLVM targets from `rustc_target` to `rustc_codegen_ssa` Fully specified LLVM targets contain the OS version on macOS/iOS/tvOS/watchOS/visionOS, and this version depends on the deployment target environment variables like `MACOSX_DEPLOYMENT_TARGET`, `IPHONEOS_DEPLOYMENT_TARGET` etc. We would like to move this to later in the compilation pipeline, both because it feels impure to access environment variables when fetching target information, but mostly because we need access to more information from rust-lang#130883 to do rust-lang#118204. See also rust-lang#129342 (comment) for some discussion. The first and second commit does the actual refactor, it should be a non-functional change, the third commit adds diagnostics for invalid deployment targets, which are now possible to do because we have access to the session. Tested with the same commands as in rust-lang#130435. r? `@petrochenkov`

…g, r=petrochenkov Move versioned Apple LLVM targets from `rustc_target` to `rustc_codegen_ssa` Fully specified LLVM targets contain the OS version on macOS/iOS/tvOS/watchOS/visionOS, and this version depends on the deployment target environment variables like `MACOSX_DEPLOYMENT_TARGET`, `IPHONEOS_DEPLOYMENT_TARGET` etc. We would like to move this to later in the compilation pipeline, both because it feels impure to access environment variables when fetching target information, but mostly because we need access to more information from rust-lang#130883 to do rust-lang#118204. See also rust-lang#129342 (comment) for some discussion. The first and second commit does the actual refactor, it should be a non-functional change, the third commit adds diagnostics for invalid deployment targets, which are now possible to do because we have access to the session. Tested with the same commands as in rust-lang#130435. r? ```@petrochenkov```

…g, r=petrochenkov Move versioned Apple LLVM targets from `rustc_target` to `rustc_codegen_ssa` Fully specified LLVM targets contain the OS version on macOS/iOS/tvOS/watchOS/visionOS, and this version depends on the deployment target environment variables like `MACOSX_DEPLOYMENT_TARGET`, `IPHONEOS_DEPLOYMENT_TARGET` etc. We would like to move this to later in the compilation pipeline, both because it feels impure to access environment variables when fetching target information, but mostly because we need access to more information from rust-lang#130883 to do rust-lang#118204. See also rust-lang#129342 (comment) for some discussion. The first and second commit does the actual refactor, it should be a non-functional change, the third commit adds diagnostics for invalid deployment targets, which are now possible to do because we have access to the session. Tested with the same commands as in rust-lang#130435. r? ````@petrochenkov````

…g, r=petrochenkov Move versioned Apple LLVM targets from `rustc_target` to `rustc_codegen_ssa` Fully specified LLVM targets contain the OS version on macOS/iOS/tvOS/watchOS/visionOS, and this version depends on the deployment target environment variables like `MACOSX_DEPLOYMENT_TARGET`, `IPHONEOS_DEPLOYMENT_TARGET` etc. We would like to move this to later in the compilation pipeline, both because it feels impure to access environment variables when fetching target information, but mostly because we need access to more information from rust-lang#130883 to do rust-lang#118204. See also rust-lang#129342 (comment) for some discussion. The first and second commit does the actual refactor, it should be a non-functional change, the third commit adds diagnostics for invalid deployment targets, which are now possible to do because we have access to the session. Tested with the same commands as in rust-lang#130435. r? `````@petrochenkov`````

…g, r=petrochenkov Move versioned Apple LLVM targets from `rustc_target` to `rustc_codegen_ssa` Fully specified LLVM targets contain the OS version on macOS/iOS/tvOS/watchOS/visionOS, and this version depends on the deployment target environment variables like `MACOSX_DEPLOYMENT_TARGET`, `IPHONEOS_DEPLOYMENT_TARGET` etc. We would like to move this to later in the compilation pipeline, both because it feels impure to access environment variables when fetching target information, but mostly because we need access to more information from rust-lang#130883 to do rust-lang#118204. See also rust-lang#129342 (comment) for some discussion. The first and second commit does the actual refactor, it should be a non-functional change, the third commit adds diagnostics for invalid deployment targets, which are now possible to do because we have access to the session. Tested with the same commands as in rust-lang#130435. r? ``````@petrochenkov``````

Rollup merge of rust-lang#131037 - madsmtm:move-llvm-target-versioning, r=petrochenkov Move versioned Apple LLVM targets from `rustc_target` to `rustc_codegen_ssa` Fully specified LLVM targets contain the OS version on macOS/iOS/tvOS/watchOS/visionOS, and this version depends on the deployment target environment variables like `MACOSX_DEPLOYMENT_TARGET`, `IPHONEOS_DEPLOYMENT_TARGET` etc. We would like to move this to later in the compilation pipeline, both because it feels impure to access environment variables when fetching target information, but mostly because we need access to more information from rust-lang#130883 to do rust-lang#118204. See also rust-lang#129342 (comment) for some discussion. The first and second commit does the actual refactor, it should be a non-functional change, the third commit adds diagnostics for invalid deployment targets, which are now possible to do because we have access to the session. Tested with the same commands as in rust-lang#130435. r? ``````@petrochenkov``````

This currently works because it's part of expansion, and that isn't yet tracked by the query system. But we want to ensure it continues working, even if that is changed.

madsmtm · 2024-11-16T18:23:48Z

@rustbot ready

Note that I've also added a test that env!/option_env! works with incremental compilation, I felt it was relevant to this PR, but feel free to tell me to do it in a different one.

rustbot assigned petrochenkov Sep 26, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 26, 2024

madsmtm commented Sep 26, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

madsmtm force-pushed the env-var-query branch from 506b0df to af26a40 Compare September 26, 2024 19:43

This comment has been minimized.

Sign in to view

madsmtm force-pushed the env-var-query branch 2 times, most recently from dee9b61 to 0ef9570 Compare September 27, 2024 14:06

This comment has been minimized.

Sign in to view

madsmtm force-pushed the env-var-query branch from 0ef9570 to 455e468 Compare September 27, 2024 15:56

This comment has been minimized.

Sign in to view

madsmtm force-pushed the env-var-query branch from 455e468 to 46276c8 Compare September 27, 2024 16:10

madsmtm commented Sep 27, 2024

View reviewed changes

madsmtm mentioned this pull request Sep 29, 2024

Move versioned Apple LLVM targets from rustc_target to rustc_codegen_ssa #131037

Merged

petrochenkov reviewed Oct 22, 2024

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 22, 2024

This comment was marked as resolved.

Sign in to view

petrochenkov mentioned this pull request Nov 10, 2024

Rebuild on changes to the deployment target when compiling Apple targets #129342

Open

madsmtm added 4 commits November 16, 2024 19:14

Test that env! works with incremental compilation

31766c6

This currently works because it's part of expansion, and that isn't yet tracked by the query system. But we want to ensure it continues working, even if that is changed.

Add TyCtx::env_var

8d1d4be

Add environment variable tracking in places where it was convenient

2ee0861

Make OsStr and CStr a bit more usable with queries

916e404

madsmtm force-pushed the env-var-query branch from 46276c8 to 916e404 Compare November 16, 2024 18:15

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add environment variable query #130883

Add environment variable query #130883

madsmtm commented Sep 26, 2024

madsmtm Sep 26, 2024

clubby789 Sep 28, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

madsmtm Sep 27, 2024 •

edited

Loading

petrochenkov Oct 22, 2024

madsmtm Nov 16, 2024

petrochenkov Oct 22, 2024

madsmtm Nov 16, 2024 •

edited

Loading

This comment was marked as resolved.

madsmtm commented Nov 16, 2024 •

edited

Loading

		// Also add the variable to Cargo's dependency tracking
		tcx.sess.psess.env_depinfo.borrow_mut().insert((key, var));

Add environment variable query #130883

Are you sure you want to change the base?

Add environment variable query #130883

Conversation

madsmtm commented Sep 26, 2024

madsmtm Sep 26, 2024

Choose a reason for hiding this comment

clubby789 Sep 28, 2024

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

madsmtm Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

petrochenkov Oct 22, 2024

Choose a reason for hiding this comment

madsmtm Nov 16, 2024

Choose a reason for hiding this comment

petrochenkov Oct 22, 2024

Choose a reason for hiding this comment

madsmtm Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

This comment was marked as resolved.

madsmtm commented Nov 16, 2024 • edited Loading

madsmtm Sep 27, 2024 •

edited

Loading

madsmtm Nov 16, 2024 •

edited

Loading

madsmtm commented Nov 16, 2024 •

edited

Loading