No method to install custom SSL certs on macOS (and Windows?)

[jmulcahy]

(accidentally sent early, sorry for the edits)

This is a new problem in Julia 1.6.

I think there is no "normal" way to add SSL certs to Pkg.jl. Setting NetworkOptions.jl's env variables to point at my certs makes download work, but breaks Pkg.jl. As example, with SSL_CERT_FILE set, attempting to install FFTW gives this error:

ERROR: failed to clone from https://github.com/JuliaBinaryWrappers/MKL_jll.jl.git, error: GitError(Code:ERROR, Class:SSL, Your Julia is built with a SSL/TLS engine that libgit2 doesn't know how to configure to use a file or directory of certificate authority roots, but your environment specifies one via the SSL_CERT_FILE variable. If you believe your system's root certificates are safe to use, you can `export JULIA_SSL_CA_ROOTS_PATH=""` in your environment to use those instead.)

The error is also slightly misleading because I believe NetworkOptions.jl only uses the system root certificates if on Linux. Otherwise, it uses a bundle that is distributed with Julia. A workaround is manually adding those certs into that bundle in Julia's install directory. It's functional, but doesn't feel great.

[StefanKarpinski]

Did you try doing export JULIA_SSL_CA_ROOTS_PATH=""?

[jmulcahy]

That doesn't work, although maybe I'm missing some other step. That just disables the other NetworkOptions.jl environment variables, right? I think on macOS and Windows that means using the bundled certs. How can I specify other certs to add?

[StefanKarpinski]

What OS and Julia version are you on? On Julia 1.6.x, both libcurl and libgit2 are built to use the system TLS engine, including certificate verification, so export JULIA_SSL_CA_ROOTS_PATH="" should at least change the error you get. You reported getting this error:

Your Julia is built with a SSL/TLS engine that libgit2 doesn't know how to configure to use a file or directory of certificate authority roots, but your environment specifies one via the SSL_CERT_FILE variable.

If you set export JULIA_SSL_CA_ROOTS_PATH="" it should override the SSL_CERT_FILE setting and tell Julia to use the system TLS engine instead of trying to use a cert file. You indicated "That doesn't work" — can you please be more specific and report exactly what happens when you do that, including any error message?

[jmulcahy]

I'm on 1.6.1, and when the issue was filed it was 1.6.0.

It doesn't appear that the system CA root is being used. As I understand it when I asked on Slack, Julia's various network stuff goes through NetworkOptions.jl. Here's what I'm seeing:

JULIA_SSL_CA_ROOTS_PATH="" julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.1 (2021-04-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using NetworkOptions

julia> ca_roots()

julia> ca_roots_path()
"/Applications/Julia-1.6.app/Contents/Resources/julia/share/julia/cert.pem"

julia> NetworkOptions.system_ca_roots()
"/Applications/Julia-1.6.app/Contents/Resources/julia/share/julia/cert.pem"

That appears to be using the bundled certs rather than the system store. This seems to be why: https://github.com/JuliaLang/NetworkOptions.jl/blob/master/src/ca_roots.jl#L87. Based on the TODO above that line and the rest of the surrounding code, it appears the system store isn't used on Windows either.

Base.download fails with SSL errors if I do nothing due to missing my custom certs. If I set some of NetworkOptions.jl's environment variables I can make Base.download work, but then libgit2 within Pkg fails with the error message you copied. The only solution I've found to bypass these errors is to copy my certs into /Applications/Julia-1.6.app/Contents/Resources/julia/share/Julia/cert.pem. That solution doesn't persist through Julia updates, and it feels like the "correct" solution is to either use the system store or allow certs to be specified via some other configuration settings.

If I am understanding all of this correctly, I can appreciate why using system stores isn't yet implemented on macOS and Windows. pip also has this problem, and I think there are no plans to fix it because the FFI linking is a problem for them. Hopefully this isn't as hard of a problem in Julia due to its excellent @ccall interface. At any rate, pip provides an escape hatch for this by manually allowing you to add certs via a configuration file. Julia appears to be missing that because NetworkOptions.jl's environment variables are incompatible with libgit2 within Pkg.

[StefanKarpinski]

I wrote that code. You may want to read the docs for ca_roots() again, noting what a return value of nothing indicates. Or just trust me that it is using your system CA root store. You can test whether I'm right by adding your firewall's cert to the macOS system CA root store and see if that fixes the problem. If you're right, that won't do anything but if I'm right it will fix the issue and we can close this.

[StefanKarpinski]

Also: asking for error messages isn't a matter of idle curiosity — it is necessary to debug what's going on. Without error messages I'm not able to debug, so there's nothing I can do here, in which case this issue isn't actionable and we can just close it.

[StefanKarpinski]

Anyway, I'm going to close this issue since there's not really any mystery here and there's nothing to fix. You need do export JULIA_SSL_CA_ROOTS_PATH="" in a shell rc file and add your firewall's cert to the system certificate authority root store. On macOS you can do this in the Keychain app.

[jmulcahy]

Here are some full stacktraces to give a better idea of what's going on. I'm on Julia 1.6.1 and macOS Catalina. I also had the same problem on 1.6.0 and High Sierra. This is with a fresh Julia install and ~/.julia directory. I've set export JULIA_SSL_CA_ROOTS_PATH="" in my configuration files.

(@v1.6) pkg> add FFTW

  Installing known registries into `~/.julia`

┌ Warning: could not download https://pkg.julialang.org/registries

└ @ Pkg.Types /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:980

┌ Warning: could not download https://pkg.julialang.org/registries

└ @ Pkg.Types /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:980

     Cloning registry from "https://github.com/JuliaRegistries/General.git"

       Added registry `General` to `~/.julia/registries/General`

  Resolving package versions...

     Cloning [621f4979-c628-5d54-868e-fcf4e3e8185c] AbstractFFTs from https://github.com/JuliaMath/AbstractFFTs.jl.git

   Installed AbstractFFTs ──── v1.0.1

     Cloning [7a1cc6ca-52ef-59f5-83cd-3a7055c09341] FFTW from https://github.com/JuliaMath/FFTW.jl.git

   Installed FFTW ──────────── v1.4.1

     Cloning [189a3867-3050-52da-a836-e630ba90ab69] Reexport from https://github.com/simonster/Reexport.jl.git

   Installed Reexport ──────── v1.0.0

     Cloning [21216c6a-2e73-6563-6e65-726566657250] Preferences from https://github.com/JuliaPackaging/Preferences.jl.git

   Installed Preferences ───── v1.2.1

     Cloning [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] JLLWrappers from https://github.com/JuliaPackaging/JLLWrappers.jl.git

   Installed JLLWrappers ───── v1.3.0

     Cloning [f5851436-0d7a-5f13-b9de-f02708fd171a] FFTW_jll from https://github.com/JuliaBinaryWrappers/FFTW_jll.jl.git

   Installed FFTW_jll ──────── v3.3.9+7

     Cloning [856f044c-d86e-5d09-b602-aeab76dc8ba7] MKL_jll from https://github.com/JuliaBinaryWrappers/MKL_jll.jl.git

   Installed MKL_jll ───────── v2021.1.1+1

     Cloning [1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0] IntelOpenMP_jll from https://github.com/JuliaBinaryWrappers/IntelOpenMP_jll.jl.git

   Installed IntelOpenMP_jll ─ v2018.0.3+2

  Downloaded artifact: FFTW

  Downloaded artifact: FFTW

ERROR: Unable to automatically install 'FFTW' from '/Users/jmulcahy/.julia/packages/FFTW_jll/VlW1Z/Artifacts.toml'

Stacktrace:

  [1] error(s::String)

    @ Base ./error.jl:33

  [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.TTY)

    @ Pkg.Artifacts /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:443

  [3] ensure_all_artifacts_installed(artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, include_lazy::Bool, verbose::Bool, quiet_download::Bool, io::Base.TTY)

    @ Pkg.Artifacts /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:508

  [4] download_artifacts(ctx::Pkg.Types.Context, pkg_roots::Vector{String}; platform::Base.BinaryPlatforms.Platform, verbose::Bool, io::Base.TTY)

    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:706

  [5] download_artifacts(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; platform::Base.BinaryPlatforms.Platform, julia_version::VersionNumber, verbose::Bool, io::Base.TTY)

    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:683

  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)

    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1237

  [7] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})

    @ Pkg.API /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:203

  [8] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})

    @ Pkg.API /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:79

  [9] add(pkgs::Vector{Pkg.Types.PackageSpec})

    @ Pkg.API /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:77

[10] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)

    @ Pkg.REPLMode /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:408

[11] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)

    @ Pkg.REPLMode /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:386

[12] do_cmd

    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]

[13] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)

    @ Pkg.REPLMode /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:550

[14] #invokelatest#2

    @ ./essentials.jl:708 [inlined]

[15] invokelatest

    @ ./essentials.jl:706 [inlined]

[16] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)

    @ REPL.LineEdit /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2441

[17] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)

    @ REPL /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126

[18] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()

    @ REPL ./task.jl:411

Note that downloading the package General registry fails, but cloning it via libgit2 does not. Downloading artifacts then fails, probably for the same reason as the registry initially. I am able to download one of the artifacts from the TOML manually via curl:

/usr/bin/curl -L -o fftw_stuff.tar.gz https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+7/FFTW.v3.3.9.x86_64-w64-mingw32.tar.gz

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100   645  100   645    0     0   1108      0 --:--:-- --:--:-- --:--:--  1106

100 2478k  100 2478k    0     0   732k      0  0:00:03  0:00:03 --:--:-- 1108k

Trying to do the same via Base.download fails:

julia> download("https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+7/FFTW.v3.3.9.x86_64-w64-mingw32.tar.gz")

ERROR: HTTP/1.1 200 Connected (SSL: certificate verification failed (result: 5)) while requesting https://github.com/JuliaBinaryWrappers/FFTW_jll.jl/releases/download/FFTW-v3.3.9+7/FFTW.v3.3.9.x86_64-w64-mingw32.tar.gz

Stacktrace:

  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)

    @ Downloads /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:356

  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)

    @ Downloads.Curl /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Curl/Curl.jl:60

  [3] #8

    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:298 [inlined]

  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::IOStream)

    @ ArgTools /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:112

  [5] #7

    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:297 [inlined]

  [6] arg_read

    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:61 [inlined]

  [7] request(url::String; input::Nothing, output::IOStream, method::Nothing, headers::Vector{Pair{String, String}}, timeout::Float64, progress::Nothing, verbose::Bool, throw::Bool, downloader::Nothing)

    @ Downloads /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:296

  [8] (::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String})(output::IOStream)

    @ Downloads /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:209

  [9] arg_write(f::Downloads.var"#3#4"{Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Nothing, String}, arg::Nothing)

    @ ArgTools /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/ArgTools/src/ArgTools.jl:101

[10] #download#2

    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:208 [inlined]

[11] download(url::String, output::Nothing)

    @ Downloads /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Downloads/src/Downloads.jl:208

[12] #invokelatest#2

    @ ./essentials.jl:708 [inlined]

[13] invokelatest

    @ ./essentials.jl:706 [inlined]

[14] do_download

    @ ./download.jl:33 [inlined]

[15] download(url::String)

    @ Base ./download.jl:29

[16] top-level scope

    @ REPL[1]:1

Some Googling suggests that result: 5 is a proxy error? That doesn't make much sense to me since unsetting JULIA_SSL_CA_ROOTS_PATH and using NetworkOptions.jl's other environment variables to specify my certs causes Base.download to succeed. Additionally, adding the certs manually to Julia's bundled cert file causes both Base.download and libgit2 to work. As far as I can tell, my certs are properly installed into Keychain, and the fact that /usr/bin/curl succeeds also suggests that is true.

[StefanKarpinski]

The situation with libraries is this:

• On Windows and macOS, the libgit2 and libcurl binaries that we ship with Julia 1.6 are compiled against the system TLS libraries rather than MbedTLS as we did previously. So by default they will both use the system CA root certs. We do this for several reasons:
  • When someone is behind a MITM firewall/proxy, it is common that a certificate for that proxy has been added to their system CA root store, in which case everything should just work.
  • It is the only way to do certificate revocation checking on these platforms (and there is no way to do certificate revocation checking on Linux at all).
• When compiled like this, libcurl can be told to use a file/directory of PEM certs instead
  • This is why download can get things when ca_roots() returns a path to a file with a certificate for your firewall in it.
• When compiled like this, libgit2 cannot be told to use a file/directory of PEM certs instead
  • This is why libgit2 can't get things when ca_roots() returns a path to a file with a certificate for your firewall in it.
  • In that situation we print the error you've seen rather than falling back to the system CA roots. We do that rather than just falling back to the system CA certificates in case you set one of those variables specifically to avoid using the system roots because they are insecure.
  • You can override that error by setting JULIA_SSL_CA_ROOTS_PATH to the empty string, as indicated in the error message.
  • There is no way to make libgit2 use certs from a PEM file. This is a limitation of the library when compiled to use the system TLS engine. You can set a PEM file if you compile libgit2 to use MbedTLS or OpenSSL, but we don't do that for the reasons explained in the first point.
  • Since libcurl is able to use the system TLS engine with a PEM file for the CA roots, it's probably possible to patch libgit2 to make this work but I don't have any plans to do this.

Additionally, adding the certs manually to Julia's bundled cert file causes both Base.download and libgit2 to work.

I have no explanation for this since this file should not be consulted by either library.

As far as I can tell, my certs are properly installed into Keychain, and the fact that /usr/bin/curl succeeds also suggests that is true.

That does suggest that. The question then is why libcurl and libgit2 aren't working by default when that's the case. It's possible that one or both are configured incorrectly and aren't using the system TLS when they should be.

[StefanKarpinski]

Regarding the original issue that "there is no method to install custom TSL certs on macOS or Windows": there is, it just isn't possible to make libgit2 honor it, which is an upstream issue that I have no intention of fixing. If you want to file an issue with libgit2 asking that they implement this capability, that would be great. Personally, I'm ready to ditch libgit2 entirely.

[jmulcahy]

Thanks for the very detailed explanation of how all these disparate pieces fit together. I think it pointed me in the right direction, and I think I've found the root cause.

download ultimately wraps the Downloads.jl stdlib. The stdlib version of Downloads.jl on 1.6.1 is missing this fix: JuliaLang/Downloads.jl@7021677. Without that fix, Downloads.jl falls back to using the bundled PEM file, which explains why manually adding my certs to that file allows download to work.

[StefanKarpinski]

That makes sense. Hopefully newer patch releases of 1.6.x will fix this for you.

[jmulcahy]

Just to confirm, this is fixed in the 1.7 beta. I'll test on the next 1.6 when that comes out, too.

[StefanKarpinski]

Great to hear! Thanks for testing and reporting back here.