GMP: Gracefully handle more overflows.

[maleadt]

Fixes #8286, which regressed due to a GMP upgrade in #45375.

Draft, because blocked on #51242, and needs a JLL bump (but waiting for review first).

[oscardssmith]

while we're patching, how hard would it be to make GMP not hard-code malloc and instead use GC tracked allocations?

[maleadt]

while we're patching, how hard would it be to make GMP not hard-code malloc and instead use GC tracked allocations?

Is this not sufficient?

julia/base/gmp.jl

Lines 111 to 115 in a7697fd

	ccall((:__gmp_set_memory_functions, libgmp), Cvoid,
	(Ptr{Cvoid},Ptr{Cvoid},Ptr{Cvoid}),
	cglobal(:jl_gc_counted_malloc),
	cglobal(:jl_gc_counted_realloc_with_old_size),
	cglobal(:jl_gc_counted_free_with_size))

[gbaraldi]

We do that, but those functions aren't correct if malloc fails. And I don't think they were ever correct.

[maleadt]

We do that, but those functions aren't correct if malloc fails. And I don't think they were ever correct.

Well, that's why I filed #51242: If we use those functions in place of malloc (which we do), they should behave correctly.