respect versions in sysimage #3002

KristofferC · 2022-02-23T20:23:57Z

I created a sysimage with an outdated Plots versions, loaded Julia with that sysimage and created a temp repo and added Plots:

(Pkg) pkg> activate --temp
  Activating new project at `/var/folders/26/gw1dqz6n5zsbkp72mktkhz3c0000gn/T/jl_hr2NH5`

(jl_hr2NH5) pkg> add Plots
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/private/var/folders/26/gw1dqz6n5zsbkp72mktkhz3c0000gn/T/jl_hr2NH5/Project.toml`
⌃ [91a5bcdd] + Plots v1.24.3
    Updating `/private/var/folders/26/gw1dqz6n5zsbkp72mktkhz3c0000gn/T/jl_hr2NH5/Manifest.toml`
  [79e6a3ab] + Adapt v3.3.3
  [d360d2e6] + ChainRulesCore v1.12.2

(jl_hr2NH5) pkg> st --outdated -m
Status `/private/var/folders/26/gw1dqz6n5zsbkp72mktkhz3c0000gn/T/jl_hr2NH5/Manifest.toml`
⌅ [28b8d3ca] GR v0.62.1 (<v0.64.0): Plots
⌅ [77ba4419] NaNMath v0.3.7 (<v1.0.0): Plots, RecipesPipeline
⌃ [91a5bcdd] Plots v1.24.3 (<v1.25.11)
⌅ [01d81517] RecipesPipeline v0.4.1 (<v0.5.1): Plots

(jl_hr2NH5) pkg> up
  No Changes to `/private/var/folders/26/gw1dqz6n5zsbkp72mktkhz3c0000gn/T/jl_hr2NH5/Project.toml`
  No Changes to `/private/var/folders/26/gw1dqz6n5zsbkp72mktkhz3c0000gn/T/jl_hr2NH5/Manifest.toml`

Requires JuliaLang/julia#44318

KristofferC · 2022-02-24T14:53:27Z

outdated support:

DilumAluthge · 2022-02-24T15:38:29Z

What will the error message look like if the user explicitly does ] add [email protected]?

KristofferC · 2022-02-24T15:42:44Z

(jl_3MCoqV) pkg> add [email protected]
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Plots [91a5bcdd]:
 Plots [91a5bcdd] log:
 ├─possible versions are: 1.24.3 or uninstalled (package in sysimage!)
 └─restricted to versions 1.25.11 by an explicit requirement — no versions left

KristofferC · 2022-02-24T21:36:22Z

Need to wait for a Julia nightly that includes JuliaLang/julia#44318 now.

staticfloat · 2022-02-24T22:44:15Z

How would this compose with #2916? Is it the case that any package that is in the system image is an stdlib? If so, could we get this same information by storing the current list of stdlibs (as in that PR) and using that information during resolution?

In any case, I like the UI improvements here.

KristofferC · 2022-02-26T13:47:42Z

How would this compose with #2916? Is it the case that any package that is in the system image is an stdlib?

Not really, the concept of an stdlib and a package in the sysimage is not the same. An stdlib is a package in the stdlib folder, a package in the sysimage is a package that is in the sysimage. They do not need to have any overlap, for example an stdlib does not need to be in the sysimage (like the _jll stdlibs) and if you use PackageCompiler, a package in the sysimage does not need to be an stdlib.

Also a package in the sysimage (that is not an stdlib) still gets its dependency and compat information from the registry.

KristofferC · 2022-02-26T13:50:04Z

Looking at the test failure, I'll disable the sysimage check when julia_version != VERSION because at that point you don't really care about the current Julia session.

giordano · 2022-03-01T02:25:26Z

src/Types.jl

@@ -526,6 +526,18 @@ function devpath(env::EnvCache, name::AbstractString, shared::Bool)
    return joinpath(dev_dir, name)
 end

+function error_if_in_sysimage(pkg::PackageSpec)


This breaks BinaryBuilder

You can set Pkg.respect_sysimage_versions(false) as a workaround.

Although, it would be good to figure out why it breaks BinaryBuilder.

https://github.com/JuliaPackaging/BinaryBuilderBase.jl/runs/5369016457?check_suite_focus=true

Expression: setup_dependencies(prefix, getpkg.(dependencies), platform) tried to develop or add a package by URL that is already in the sysimage Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/Types.jl:67 [2] error_if_in_sysimage(pkg::Pkg.Types.PackageSpec) @ Pkg.Types /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/Types.jl:537 [3] (::Pkg.Types.var"#47#48"{Pkg.Types.Context, Pkg.Types.PackageSpec, String})(repo::GitRepo) @ Pkg.Types /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/Types.jl:752 [4] with(f::Pkg.Types.var"#47#48"{Pkg.Types.Context, Pkg.Types.PackageSpec, String}, obj::GitRepo) @ LibGit2 /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/LibGit2/src/types.jl:1153 [5] handle_repo_add!(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec) @ Pkg.Types /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/Types.jl:709 [6] handle_repos_add!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}) @ Pkg.Types /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/Types.jl:779 [7] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Platform, kwargs::Base.Pairs{Symbol, Union{Base.DevNull, Nothing}, Tuple{Symbol, Symbol}, NamedTuple{(:io, :julia_version), Tuple{Base.DevNull, Nothing}}}) @ Pkg.API /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/API.jl:255 [8] (::BinaryBuilderBase.var"#89#95"{Bool, Prefix, Vector{Pkg.Types.PackageSpec}, Platform, Vector{String}, Vector{String}})() @ BinaryBuilderBase ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/src/Prefix.jl:631 [9] activate(f::BinaryBuilderBase.var"#89#95"{Bool, Prefix, Vector{Pkg.Types.PackageSpec}, Platform, Vector{String}, Vector{String}}, new_project::String) @ Pkg.API /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/API.jl:1700 [10] setup_dependencies(prefix::Prefix, dependencies::Vector{Pkg.Types.PackageSpec}, platform::Platform; verbose::Bool) @ BinaryBuilderBase ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/src/Prefix.jl:582 [11] setup_dependencies @ ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/src/Prefix.jl:556 [inlined] [12] (::var"#38#60"{Platform, Vector{Dependency}, Prefix})() @ Main ./none:0 [13] with_logstate(f::Function, logstate::Any) @ Base.CoreLogging ./logging.jl:511 [14] with_logger @ ./logging.jl:623 [inlined] [15] #collect_test_logs#65 @ /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Test/src/logging.jl:114 [inlined] [16] collect_test_logs @ /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Test/src/logging.jl:113 [inlined] [17] #match_logs#66 @ /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Test/src/logging.jl:260 [inlined] [18] match_logs @ /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Test/src/logging.jl:260 [inlined] [19] (::var"#37#59")(dir::String) @ Main ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:151 [20] #30 @ ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:14 [inlined] [21] activate(f::var"#30#32"{String, var"#37#59"}, new_project::String) @ Pkg.API /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Pkg/src/API.jl:1700 [22] #29 @ ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:13 [inlined] [23] mktempdir(fn::var"#29#31"{var"#37#59"}, parent::String; prefix::String) @ Base.Filesystem ./file.jl:760 [24] mktempdir (repeats 2 times) @ ./file.jl:758 [inlined] [25] with_temp_project(f::var"#37#59") @ Main ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:12 [26] macro expansion @ ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:145 [inlined] [27] macro expansion @ /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Test/src/Test.jl:1357 [inlined] [28] macro expansion @ ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:109 [inlined] [29] macro expansion @ /opt/hostedtoolcache/julia/nightly/x64/share/julia/stdlib/v1.9/Test/src/Test.jl:1357 [inlined] [30] top-level scope @ ~/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/dependencies.jl:20

We install JLL stdlibs with different version numbers

Ahhh, yeah that makes sense.

Tests still fail even with JuliaPackaging/BinaryBuilderBase.jl#227, but in a different way, so there may be something else broken now. But I don't have the time to debug now.

Alright, I believe that the remaining failure is unrelated to any changes in Pkg. Pkg.respect_sysimage_versions(false) is sufficient to be able to install packages without problems.

ericphanson · 2022-03-04T16:26:47Z

any chance this is eligible for backport to 1.8?

ViralBShah · 2022-06-03T21:17:10Z

Maybe this can go into 1.8.1?

(cherry picked from commit 31af274)

KristofferC force-pushed the kc/sysimage branch from afea2a8 to bd4bfe8 Compare February 24, 2022 14:20

KristofferC changed the title ~~wip: respect versions in sysimage~~ respect versions in sysimage Feb 24, 2022

KristofferC force-pushed the kc/sysimage branch from bd4bfe8 to 9a938e1 Compare February 24, 2022 14:31

DilumAluthge closed this Feb 24, 2022

DilumAluthge reopened this Feb 24, 2022

respect versions in sysimage

b8a61e3

KristofferC force-pushed the kc/sysimage branch from 9a938e1 to b8a61e3 Compare February 25, 2022 07:41

KristofferC closed this Feb 26, 2022

KristofferC reopened this Feb 26, 2022

KristofferC added 2 commits February 26, 2022 14:55

only respect sysimage if julia_version is not set

44b16a2

Add how to disable

8e77c68

DilumAluthge closed this Feb 26, 2022

DilumAluthge reopened this Feb 26, 2022

KristofferC merged commit 31af274 into master Feb 26, 2022

KristofferC deleted the kc/sysimage branch February 26, 2022 20:41

giordano reviewed Mar 1, 2022

View reviewed changes

KristofferC added the backport 1.8 label Jun 7, 2022

KristofferC added a commit that referenced this pull request Jun 7, 2022

respect versions in sysimage (#3002)

bab306b

(cherry picked from commit 31af274)

IanButterworth removed the backport 1.8 label Jun 17, 2022

KristofferC mentioned this pull request Sep 13, 2022

only store version of packages during load time when generating sysimage JuliaLang/julia#46738

Merged

LilithHafner mentioned this pull request Sep 19, 2023

Change semantics of respect_sysimage_versions #3627

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

respect versions in sysimage #3002

respect versions in sysimage #3002

KristofferC commented Feb 23, 2022 •

edited

Loading

KristofferC commented Feb 24, 2022

DilumAluthge commented Feb 24, 2022

KristofferC commented Feb 24, 2022

KristofferC commented Feb 24, 2022

staticfloat commented Feb 24, 2022

KristofferC commented Feb 26, 2022 •

edited

Loading

KristofferC commented Feb 26, 2022

giordano Mar 1, 2022

DilumAluthge Mar 1, 2022

giordano Mar 1, 2022

DilumAluthge Mar 1, 2022

giordano Mar 1, 2022

giordano Mar 1, 2022

ericphanson commented Mar 4, 2022

ViralBShah commented Jun 3, 2022

respect versions in sysimage #3002

respect versions in sysimage #3002

Conversation

KristofferC commented Feb 23, 2022 • edited Loading

KristofferC commented Feb 24, 2022

DilumAluthge commented Feb 24, 2022

KristofferC commented Feb 24, 2022

KristofferC commented Feb 24, 2022

staticfloat commented Feb 24, 2022

KristofferC commented Feb 26, 2022 • edited Loading

KristofferC commented Feb 26, 2022

giordano Mar 1, 2022

Choose a reason for hiding this comment

DilumAluthge Mar 1, 2022

Choose a reason for hiding this comment

giordano Mar 1, 2022

Choose a reason for hiding this comment

DilumAluthge Mar 1, 2022

Choose a reason for hiding this comment

giordano Mar 1, 2022

Choose a reason for hiding this comment

giordano Mar 1, 2022

Choose a reason for hiding this comment

ericphanson commented Mar 4, 2022

ViralBShah commented Jun 3, 2022

KristofferC commented Feb 23, 2022 •

edited

Loading

KristofferC commented Feb 26, 2022 •

edited

Loading