Invoking Pkg mode results in error

[ViralBShah]

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.550 (2023-09-25)
 _/ |\__'_|_|_|\__'_|  |  Commit 13d3efb46f (0 days old master)
|__/                   |

julia> ┌ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]
└ @ Base loading.jl:2351
ERROR: LoadError: ArgumentError: Package LibGit2 does not have LibGit2_jll in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have LibGit2 checked out for development and have
  added LibGit2_jll as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with LibGit2
Stacktrace:
 [1] macro expansion
   @ Base ./loading.jl:1777 [inlined]
 [2] macro expansion
   @ Base ./lock.jl:267 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1754
 [4] #invoke_in_world#3
   @ Base ./essentials.jl:933 [inlined]
 [5] invoke_in_world
   @ Base ./essentials.jl:930 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1747
 [7] include
   @ Base ./Base.jl:497 [inlined]
 [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
   @ Base ./loading.jl:2220
 [9] top-level scope
   @ stdin:3
in expression starting at /Users/viral/julia/usr/share/julia/stdlib/v1.11/LibGit2/src/LibGit2.jl:3
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile LibGit2 [76f85450-5226-5b5a-8eaa-529ad045b433] to "/Users/viral/.julia/compiled/v1.11/LibGit2/jl_jOMkcE".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2472
  [3] compilecache
    @ Base ./loading.jl:2338 [inlined]
  [4] (::Base.var"#998#999"{Base.PkgId})()
    @ Base ./loading.jl:1972
  [5] mkpidlock(f::Base.var"#998#999"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile ~/julia/usr/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ FileWatching.Pidfile ~/julia/usr/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile ~/julia/usr/share/julia/stdlib/v1.11/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ Base ./essentials.jl:901 [inlined]
  [9] invokelatest
    @ Base ./essentials.jl:896 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#998#999"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2984
 [11] maybe_cachefile_lock
    @ Base ./loading.jl:2981 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1968
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1813
 [14] #invoke_in_world#3
    @ Base ./essentials.jl:933 [inlined]
 [15] invoke_in_world
    @ Base ./essentials.jl:930 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1804
 [17] macro expansion
    @ Base ./loading.jl:1791 [inlined]
 [18] macro expansion
    @ Base ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1754
 [20] #invoke_in_world#3
    @ Base ./essentials.jl:933 [inlined]
 [21] invoke_in_world
    @ Base ./essentials.jl:930 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1747
 [23] include(mod::Module, _path::String)
    @ Base ./Base.jl:497
 [24] include(x::String)
    @ Pkg ~/julia/usr/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:3
 [25] top-level scope
    @ ~/julia/usr/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:58
 [26] include
    @ Base ./Base.jl:497 [inlined]
 [27] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:2220
 [28] top-level scope
    @ stdin:3
in expression starting at /Users/viral/julia/usr/share/julia/stdlib/v1.11/Pkg/src/GitTools.jl:3
in expression starting at /Users/viral/julia/usr/share/julia/stdlib/v1.11/Pkg/src/Pkg.jl:3
in expression starting at stdin:3
┌ Error: Error in the keymap
│   exception = (ErrorException("Failed to precompile Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] to \"/Users/viral/.julia/compiled/v1.11/Pkg/jl_oSLIcS\"."), Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x0000000118716647, Ptr{Nothing} @0x000000011832f61b, Ptr{Nothing} @0x00000001184880bb, Ptr{Nothing} @0x00000001054e80f7, Ptr{Nothing} @0x00000001054e81cf, Ptr{Nothing} @0x000000010551cb43, Ptr{Nothing} @0x000000010514c0fb, Ptr{Nothing} @0x000000010515404b, Ptr{Nothing} @0x000000010515409b, Ptr{Nothing} @0x000000010551cb43, Ptr{Nothing} @0x000000010552af37, Ptr{Nothing} @0x000000010514018f, Ptr{Nothing} @0x000000010551cb43, Ptr{Nothing} @0x000000010552af37, Ptr{Nothing} @0x0000000105138113, Ptr{Nothing} @0x000000010551cb43, Ptr{Nothing} @0x000000010552b68b, Ptr{Nothing} @0x000000011853b087, Ptr{Nothing} @0x00000001183d3987, Ptr{Nothing} @0x00000001183200e7, Ptr{Nothing} @0x0000000118688c4f, Ptr{Nothing} @0x000000010551cbcb, Ptr{Nothing} @0x000000010552b71b, Ptr{Nothing} @0x000000011854b80b, Ptr{Nothing} @0x0000000127862bef, Ptr{Nothing} @0x000000010551cbcb, Ptr{Nothing} @0x000000010552b68b, Ptr{Nothing} @0x0000000127862587, Ptr{Nothing} @0x00000001278cfb27, Ptr{Nothing} @0x000000010551cbcb, Ptr{Nothing} @0x0000000127807657, Ptr{Nothing} @0x00000001278dc3c7, Ptr{Nothing} @0x0000000127807fbb, Ptr{Nothing} @0x000000010551cbcb, Ptr{Nothing} @0x000000012783ccf7, Ptr{Nothing} @0x000000012779daf7, Ptr{Nothing} @0x000000012783cfaf, Ptr{Nothing} @0x000000010551cbcb, Ptr{Nothing} @0x000000010553cd23])
└ @ REPL.LineEdit /Users/viral/julia/usr/share/julia/stdlib/v1.11/REPL/src/LineEdit.jl:2809
julia> 

[ViralBShah]

Get the same error with using Pkg.

I built Julia after doing a git clean -fdx.

[KristofferC]

This would probably be solved with a pkg> resolve but the irony is of course that Pkg itself cannot be loaded.

The question is why the bundled pr compile files for Pkg are not working.
If you set ENV["JULIA_DEBUG"] = "loading" and try to load Pkg, what happens?

[vchuravy]

Try deleting you v1.11 Manifest

[vtjnash]

You can run with --compiled-modules=no or set Base.PKG_PRECOMPILE_HOOK[] = Returns(nothing) to get out of the catch-22 and run pkg> resolve

[ViralBShah]

I already wiped out my .julia, so can't try the workarounds. Just wanted to report it so it was known.

[oscardssmith]

@vtjnash neither of these work. With --compiled=modules=no I get

│   exception = (UndefVarError(:REPLMode), Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x00007f1df3f9b8c8, Ptr{Nothing} @0x00007f1df3f32e18, Ptr{Nothing} @0x00007f1de216ec82, Ptr{Nothing} @0x00007f1de26dc5ca, Ptr{Nothing} @0x00007f1de1f773b1, Ptr{Nothing} @0x00007f1df3f229dc, Ptr{Nothing} @0x00007f1de1f6fd95, Ptr{Nothing} @0x00007f1de1f6fdfc, Ptr{Nothing} @0x00007f1de1f66d5b, Ptr{Nothing} @0x00007f1de1f67a98, Ptr{Nothing} @0x00007f1de1f67f80, Ptr{Nothing} @0x00007f1de1f5ff4b, Ptr{Nothing} @0x00007f1de1f60274, Ptr{Nothing} @0x00007f1de1f602a2, Ptr{Nothing} @0x00007f1df3f38428])
└ @ REPL.LineEdit /home/oscardssmith/julia/usr/share/julia/stdlib/v1.11/REPL/src/LineEdit.jl:2809

when trying to enter pkg mode. Trying to modify PKG_PRECOMPILE_HOOK gives invalid redinition of constant.

[oscardssmith]

that said removing .julia/environments/v1.11/Manifest.toml did fix it.

[matthias314]

Removing the v1.11 manifest also fixes it for me. However, I still get error messages when testing packages:

$ ./julia -e 'using Pkg; Pkg.test("MacroTools")'
     Testing MacroTools
      Status `/tmp/jl_7LVf4U/Project.toml`
  [1914dd2f] MacroTools v0.5.11
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_7LVf4U/Manifest.toml`
  [1914dd2f] MacroTools v0.5.11
  [2a0f44e3] Base64 `@stdlib/Base64`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [ea8e919c] SHA v0.7.0 `@stdlib/SHA`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [8dfed614] Test `@stdlib/Test`
ERROR: LoadError: ArgumentError: Package REPL does not have Sockets in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have REPL checked out for development and have
  added Sockets as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with REPL

With --compiled-modules=no the error disappears.

In case it matters, I'm using custom JULIA_DEPOT_PATH and JULIA_LOAD_PATH. EDIT: This does not matter.

[KristofferC]

neither of these work. With --compiled=modules=no I get...

That seems to be a completely different error from the one this issue is about?

[matthias314]

I was assuming that it's the same underlying problem as in the OP because the error message is of the same form:

ERROR: LoadError: ArgumentError: Package XXX does not have YYY in its dependencies:

In my case, the error also happens without setting JULIA_DEPOT_PATH or JULIA_LOAD_PATH. (I should have checked this before.) If you think it has nothing to do with the OP, I can open a separate issue.

[KristofferC]

I was not replying directly to you but was quoting one of the earlier messages from this issue.

[staticfloat]

This is pretty insidious; it's impossible to re-resolve an old Manifest because when you start julia --project and then try to import Pkg, it constantly fails.

[vchuravy]

Does starting Julia in a fresh environment; then import Pkg work? From there you could activate the target environment and resolve it again. But yes we need a better solution :/ People might run into this when upgrading Julia and running older projects

x-ref: maleadt/LLVM.jl#363 (comment)

[vtjnash]

No, Pkg needs LibGit2, so it is a chicken and egg type problem right now, where Pkg cannot run because the Manifest is not valid anymore and the Manifest is not valid because Pkg hasn't been run

[staticfloat]

I'm conflicted about the Pkg changes here. On one hand, I think upgradeable stdlibs are great, but on the other hand, it really sucks that a broken Manifest is not fixable because it breaks Julia. I almost feel like we need a distinction between "user" code and Julia "system" code, which of course is what the system image was indirectly providing.

Possible solutions I see:

• We special-case pkg> REPL mode to try loading the current Pkg and if it fails, loads @stdlib/Pkg, printing out a warning if it fails.

• The pkg> REPL mode is 🌟special🌟 and always loads from @stdlib, ignoring the current Project/Manifest completely, unless you run something like REPL.load_new_pkg!().

• We create a new entrypoint for package management that is independent of the main julia executable, e.g. jlpkg or something, and tell users to use that if things break.