Improve handling of preferences for sandboxes used in Pkg

[vchuravy]

Fixes #2500, when we set up the preferences for test/build we want
to propagate the preferences from either test/LocalPreferences.toml if test/Project.toml exists
or from Project.toml if the old style is being used. We gather the dependencies up and then write
them out to the temp directory.

Similarily for the .pkg/select_artifacts.jl functionality we need to make the current preferences
available so that we can augment the platform we are downloading.

Co-authored-by: @staticfloat

[vchuravy]

52a2b4c depends on JuliaLang/julia@0b7827a

We might be able to backport 78963ef to 1.7

[vchuravy]

Depends on JuliaLang/julia#43684

[KristofferC]

I don't really get why this is not just as simple as just not removing existing preferences from the Project file when creating the temp project file and copy an existing LocalPreferences.toml to the temp environment. Is it because the temp Project does not have the same UUID as the original one?

[staticfloat]

I don't really get why this is not just as simple as just not removing existing preferences from the Project file when creating the temp project file and copy an existing LocalPreferences.toml to the temp environment.

It's because Preferences merge across stacked environments. E.g. if you have MPIBackend = "foo" set in your ~/.julia/environments/v1.X/LocalPreferences.toml, that needs to be reflected in the LocalPreferences.toml you set up in the temp environment as well. The easiest way to do that is to use Base.get_preferences() to flatten everything down into a single Dict, then just write that out as a TOML.

[vchuravy]

Ok this now passes tests :)

[staticfloat]

IMO this is ready to merge. Any objections?

[timholy]

Thanks for fixing this, @staticfloat!

[staticfloat]

It's not me, it was @vchuravy! He's the one with the patience and panache to push this through, and he deserves all the credit.

[timholy]

Ahh, fooled by the "merged by" again! (Though I see you did help out.) Thanks @vchuravy!!