Upgrade libgit2 to 1.3.0, libssh2 to 1.10.2, mbedtls to 2.28 and libcurl to 7.81.0

[nalimilan]

Goes with JuliaPackaging/Yggdrasil#3962.

Closes #42209
Closes #43249

[DilumAluthge]

Perhaps you need this patch? #43249 (comment)

[KristofferC]

Perhaps you need this patch?

That's already in here I think: https://github.com/JuliaLang/julia/pull/43250/files#diff-ba4b52130833cf4924e96d191202c63e7ddee40545d5c001cc927e77a392f762R233-R235

---

Closes #42209?

[DilumAluthge]

Hmmm. Not sure why tests are failing, then.

[DilumAluthge]

Ah. I guess the Ygg PR needs to be merged first.

[mkitti]

Just reporting in from https://github.com/conda-forge/julia-feedstock. With PR conda-forge/julia-feedstock#157, we deployed Julia 1.7.1 with LibGit2 1.3.

[ViralBShah]

mbedtls, libssh2 and libgit2 should probably all be upgraded simultaneously, since libssh2 and libgit2 are hard linked to mbedtls 2.28. And of course one hopes everything will work after that!

[nalimilan]

I've updated the PR to also upgrade libssh2 and mbedTLS, stealing commits from #42311. Please double check, as I'm quite confused by the versioning tricks from JuliaPackaging/Yggdrasil#4208. Also, what's going on with the patch at JuliaPackaging/Yggdrasil#4179? Should it be applied in Julia too?

[mkitti]

We probably do want to import the patch into JuliaLang/Julia.

The build is still looking for libmbedtls.so.13

LoadError("sysimg.jl", 19, LoadError("/buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.8/LibGit2/src/LibGit2.jl", 3, LoadError("/buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.8/LibGit2/src/utils.jl", 44, ErrorException("could not load library \"libgit2\"\nlibmbedtls.so.13: cannot open shared object file: No such file or directory"))))

[mkitti]

Looks like libssh2 needs to be updated on this branch:
https://github.com/JuliaLang/julia/blob/nl/libgit2-1.3.0/deps/libssh2.version

Currently, it says 1.10.0.

julia/deps/libssh2.version

Lines 1 to 2 in 65a5621

	LIBSSH2_BRANCH=libssh2-1.10.0
	LIBSSH2_SHA1=635caa90787220ac3773c1d5ba11f1236c22eae8

Shouldn't it be 1.10.2 https://github.com/JuliaBinaryWrappers/LibSSH2_jll.jl/releases/tag/LibSSH2-v1.10.2%2B0 ?

[nalimilan]

The build is still looking for libmbedtls.so.13

Yeah, but AFAICT this is due to libgit2.so (and then libssh2.so and libcurl.so) linking to libmbedtls.so.13. I guess they need to be rebuilt in Yggdrasil? The build works with USE_BINARYBUILDER_*=0 for these three deps.

Shouldn't it be 1.10.2 https://github.com/JuliaBinaryWrappers/LibSSH2_jll.jl/releases/tag/LibSSH2-v1.10.2%2B0 ?

This is one of the things I wondered. Since the variable says "branch", I assumed it has to refer to an existing branch in the repo? 1.10.2 doesn't actually exist upstream.

[mkitti]

This is one of the things I wondered. Since the variable says "branch", I assumed it has to refer to an existing branch in the repo? 1.10.2 doesn't actually exist upstream.

This should be the same as this:
https://github.com/JuliaPackaging/Yggdrasil/blob/dd0bfdabbaef3493841f54bce34539b270ebab8c/L/LibSSH2/build_tarballs.jl#L7-L11

[nalimilan]

OK, turns out I hadn't updated JLLs for all of these! But we'll need to update Curl for build to work: JuliaPackaging/Yggdrasil#4200

[nalimilan]

This should be the same as this: https://github.com/JuliaPackaging/Yggdrasil/blob/dd0bfdabbaef3493841f54bce34539b270ebab8c/L/LibSSH2/build_tarballs.jl#L7-L11

So AFAICT that's what this PR currently does. version is 1.10.0 at that point of the file, and the SHA256 sum of the file matches.

[giordano]

Is the issue that the ABI breakage will break everything that links to mbedtls?

Correct: if a library has embedded that it requires libmbedtls.so.13 (which is mbedtls 2.24), there is no way it can be made to work with mbedtls which has a different soname (one could suggest to use patchelf, but that's begging for troubles elsewhere, you just shift the failure from loading time to runtime).

I suppose these packages need to be looked at and rebuilt? juliahub.com/ui/Packages/MbedTLS_jll/u5NEn/2.28.0+0?page=2

The fact is that you can't "just rebuild them", if you don't change version number and julia compat version correctly you'll break users on older versions of julia.

What I'm trying to say is that there is no easy to way to retroactively change binary dependencies of julia without lots of disruption which I'm not going to deal with if there isn't a robust discussion about how to deal with it. I could understand backporting stuff like libgit2 and libssh2 only which have a stable ABI (and I'd still be wary of doing that blindly), but mbedtls is a serious mess.

[nalimilan]

It's probably better (both in terms of avoiding breakage and in terms of required work) to only backport patches that are needed to fix security issues. Note that Linux distributions have to do the same thing (notably Ubuntu LTS and CentOS, but that also applies to others over their shorter lifetimes), so with some luck we could just reuse the patches they backport. That's maybe something to keep in mind for the next LTS: using the same versions as a major LTS distribution would ensure that we can do that easily.

[nalimilan]

Merge?

[mkitti]

Why is buildbot/tester_win32 pending?

[ViralBShah]

We have the CI straight flush. Haven't seen that in a while. Good to merge?

[ViralBShah]

I've removed the backport labels, since we may want to do a very limited backport and not this whole PR.

[nalimilan]

64-bit Linux CI failed, but that doesn't seem related.

[ViralBShah]

@nalimilan Merge?

[Seelengrab]

We may want to backport the libssh2 version bump to LTS: https://cve.circl.lu/cve/CVE-2019-17498

[giordano]

That's not really doable without breaking some thousands of packages indirectly depending on mbedtls along the way

[Seelengrab]

I'm not sure I understand - is libssh2 deeply tied to mbedtls? I think I'm missing some context here, since just some messages up you mentioned that libssh2 may be possible:

I could understand backporting stuff like libgit2 and libssh2 only which have a stable ABI (and I'd still be wary of doing that blindly), but mbedtls is a serious mess.

Though I of course understand that the impact of that CVE could be considered low enough to not be an issues. I definitely don't want to imply that it has to be done!