Global preferences

[aplavin]

How to make global preferences for a package? I tried putting them to ~/.julia/environments/v1.7/Project.toml, but they are only available when this environment is in the LOAD_PATH. Code running from tests or from Pluto notebooks doesn't have this load path entry, so those preferences are unavailable.
I'm talking about machine-specific preferences, like some system directory path, or hardware-related settings. They seem to be a good fit for Preferences.jl, but such preferences should always be available in order to make sense at all.

[vchuravy]

Global preferences essentially work through the JULIA_LOAD_PATH. See https://juliaparallel.org/tutorials/preferences/#interactions_with_the_julia_load_path

[staticfloat]

Yeah, you'll need to provide a global entry on the LOAD_PATH for this to work.

[aplavin]

Just checking – any way to have global (machine/depot wide) preferences now? :)
In some common setups, LOAD_PATH doesn't contain the default environment – eg in notebooks or in tests. While machine-specific preferences should be used no matter how the code is run. Stuff like defining library paths or hardware settings.

[carstenbauer]

Just checking – any way to have global (machine/depot wide) preferences now? :)

What I do a lot (on different HPC clusters where I want global preferences) is to save a Project.toml that has all the preferences I need on this machine at /some/path/Project.toml and then set

export JULIA_LOAD_PATH=:/some/path

Note that the colon after the equal sign is crucial as it means "append". This works very well for me. See https://juliahpc.github.io/sysadmin_julia/#providing_global_package_preferences.

[aplavin]

JULIA_LOAD_PATH and LOAD_PATH variables often don't propagate: some examples I know of are running tests (Pkg.test()) and notebooks (Pluto). So this solution isn't truly "global", it only applies in limited situations.

[carstenbauer]

That's unfortunate, indeed. Although I would argue that setting environment variables is a pretty canonical way to set global settings (well beyond Julia) and would therefore see this as more of a Pluto or Pkg.test() issue.

(And, to be fair, both Pluto and Pkg.test() are known for their "sandboxing" approach, which makes propagating global information more difficult. I would say the issue is rather specific to them.)

[aplavin]

Julia environments carry a lot more than preferences – most notably, packages. Naturally, tests and notebooks don't want other packages (not under their control) to influence the setup, so they don't propagate LOAD_PATH. I assume this is an explicit decision, not an oversight.
Still, global preferences (data or library paths, hardware settings) make sense to apply there as well.

And, to be fair, both Pluto and Pkg.test() are known for their "sandboxing" approach, which makes propagating global information more difficult. I would say the issue is rather specific to them.

So there's indeed no way to set Preferences globally. Tests and notebooks are common, these aren't some niche usecases.
Would be nice to have a depot-wide Preferences.toml file!

[carstenbauer]

I could also imagine a JULIA_PREFERENCES_LOAD_PATH environment variable that only loads preferences (and that defaults to the regular load path). Or alternatively, the preferences from these paths could be added on top of what LOAD_PATH already gives.

[3f6a]

JULIA_LOAD_PATH and LOAD_PATH variables often don't propagate: some examples I know of are running tests (Pkg.test()) and notebooks (Pluto). So this solution isn't truly "global", it only applies in limited situations.

I just want to say that Pluto is precisely the context where I'd like to be able to set global preferences for a computer / user account.
If JULIA_LOAD_PATH and LOAD_PATH do not work for this purpose, it would be nice to have some other solution. I think it would be generally useful.

[aplavin]

Are there any issues with adding ~/.julia/Preferences.toml to the default preferences files list?

[carstenbauer]

What if my depot isn't under the default path (in home)?
For example, I'm thinking of a scenario where a system admin wants to set the global preferences for all users (say, on an HPC cluster). In this case, the issue is that the sys admin doesn't control where a user puts their Julia depot. (And it is common to put the depot on a parallel filesystem, i.e. outside of home.)

Also, communicating this information through a file at a fixed place isn't optimal. It's fragile (the user might wipe the depot at some point) and non trivial (when/how should the sys admin create this file? It's much easier to set an environment variable).

If we would choose this "global fixed file" approach, I'd probably (in addition) want a Preferences.toml in a subfolder of Sys.BINDIR (c.f. startup.jl)

[aplavin]

What if my depot isn't under the default path (in home)?

Sorry for ambiguity, I meant Preferences.toml in the depot folder. Or any specific subfolder, like config that is already used for stuff like startup.jl.

It's fragile (the user might wipe the depot at some point)

And that's fine, isn't it? Wipe the depot => the entire Julia installation is removed, including its settings.

non trivial (when/how should the sys admin create this file? It's much easier to set an environment variable)

The presence of a global preferences file doesn't preclude adding env vars for preferences, or to customize the location of this file for complex setups. I'm just saying there should be a way to put the global preferences file somewhere so that it's read by default no matter how Julia is started. The vast majority of computers are effectively single-user, so "truly global" and "user-global" are typically the same.

[carstenbauer]

I think we should entangle two things here.

The first point is:

Julia environments carry a lot more than preferences – most notably, packages. Naturally, tests and notebooks don't want other packages (not under their control) to influence the setup, so they don't propagate LOAD_PATH. I assume this is an explicit decision, not an oversight.
Still, global preferences (data or library paths, hardware settings) make sense to apply there as well.

I think it's fair to ask for a way to set preferences of packages without making them available to load (i.e. using). That is, I can imagine a mechanism that looks at LOAD_PATH to see which packages are available and sets preferences for those packages not just based on the environments specified in LOAD_PATH directly but also other Preferences.toml files.

I think it's fair to ask for a way to load preferences from Preferences.tomls or Project.tomls without making their [deps] (if any) available to load (i.e. using).

The second point is:

How should it be possible to specify the location of these Preferences.toml/Project.toml files that should be considered in this "preferences-only" fashion? The two proposals so far are

1. A fixed-location file approach, where a Preferences.toml can be placed at a fixed location in a Julia depot (similar to config/startup.jl).
2. An environment variable approach, where a new JULIA_PREFERENCES_PATH is used to indicate the locations.

The advantage of 1) is that it is simple. The disadvantage of 1), at least in its current form, is that it doesn't give "truly global" but only "user depot global" preferences, which can be an issue in some scenarios (e.g. multi-user server settings like HPC). However, this could be amended by including a fixed location etc/Preferences.toml in the Julia install directory (not the depot). This is similar to the global etc/startup.jl which we already support (although most people likely don't know about this).

The advantage of 2) is that it is more flexible. The disadvantage is that Pluto and Pkg.test etc. would likely need to explicitly support this new environment variable (I assume that they drop all env vars and only keep the information that they need).

[aplavin]

I think it's fair to ask for a way to set preferences of packages without making them available to load (i.e. using).

Oh, I didn't even know it's not possible :) So, setting a Preference for a package is currently not allowed without having the package installed in the same env? Sorry for these basic questions, didn't really play with preferences after I understood that there's no support for global.

[carstenbauer]

I misspoke a little bit and have updated my post above. It is possible to create a Preferences.toml that only provides preferences and not any packages (i.e. it doesn't have any [deps]). See for example https://juliahpc.github.io/sysadmin_julia/#example_mpijl_and_cudajl.

However, loading preferences and making packages loadable (i.e. using) is currently somewhat intertwined in the sense that we use LOAD_PATH to find preferences but also packages (i.e. [deps]). And I thinks it is fair to think about entangling this a little bit.

Note that a tangent to this is that it would be nice to have public API here that allows one to access available preferences for a package that is not a [deps] for any environment in LOAD_PATH. In JuliaGPU/CUDA_Runtime_Discovery.jl#13, we use Base.get_preferences() for this but that's not public API (see the resolved discussion in this PR).

[JBlaschke]

So we have a way of defining "global" preferences most of the time... But not all the time. Yikes!

I wanted to bump this here by providing an example: on our system (NERSC Perlmutter), it is essential that the system version of MPI and CUDA are used.

To this end we also do what @carstenbauer does and set "global" preferences via JULIA_LOAD_PATH.

Worse, the jll version of CUDA does not necessarily result in a crash, but just wrong behavior. So this leads to the frustrating situation where you develop some code, and everything works as expected (because Julia is using the JULIA_LOAD_PATH in most circumstances)... And then PKG.test() fails... With the same damned code.

What are we going to do about this? It's important that the solution is something the sysadmin can do (as we have a lot of users, most of which don't read the docs)

[staticfloat]

I think it's clear that the current behavior is not what we want, so I have just made a PR that at least fixes the fundamental problem, I think: JuliaLang/julia#56575

With that PR you should be able to have preferences that refer to packages that are not a top-level dependency in your current project (but that exist within a Manifest somewhere in your load path). This means that you can now have preferences applied to non-top-level deps, but also that you can have a LocalPreferences.toml with no accompanying Project.toml. You can even put a LocalPreferences.toml file in your stdlib/ directory for truly global preference-setting. Please feel free to leave comments on that PR about whether it works for you or not. There's likely many tests that need to be adjusted, but my Julia time is pretty limited these days, so if anyone else wants to help with fixing those tests, that'd be great!