GH-104142: Fix _Py_RefcntAdd to respect immortality

[brandtbucher]

Also, make the existing immortality tests a bit stricter and more comprehensive.

• Issue: _Py_RefcntAdd doesn't respect immortality #104142

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @brandtbucher for commit be3b103 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[ericsnowcurrently]

CC @eduardo-elizondo

[brandtbucher]

(Not sure why buildbots aren't running...)

[corona10]

@brandtbucher Ah would you like to add the C API test in here too?

cpython/Modules/_testcapi/immortal.c

Lines 34 to 38 in 9885677

	static PyMethodDef test_methods[] = {
	{"test_immortal_builtins", test_immortal_builtins, METH_NOARGS},
	{"test_immortal_small_ints", test_immortal_small_ints, METH_NOARGS},
	{NULL},
	};

Adding a code line somewhere of here would be enough.

cpython/Modules/_testcapi/immortal.c

Lines 5 to 10 in 9885677

	assert(_Py_IsImmortal(object));
	Py_ssize_t old_count = Py_REFCNT(object);
	for (int j = 0; j < 10000; j++) {
	Py_DECREF(object);
	}
	Py_ssize_t current_count = Py_REFCNT(object);

#104143 (comment)

[corona10]

Ah sorry please ignore #104143 (comment)
it's not public API...

[eduardo-elizondo]

One quick comment here is that if this were to be in a hot path, I would recommend using the same technique that we use for Py_INCREF to minimize the number of generated instructions to do the check and add: https://github.com/python/cpython/blob/main/Include/object.h#L620-L634

Now, given that it's only in use for sq_repeat in tuple and list, I don't think this is perf sensitive so keeping the code as you have it here should be good.

[eduardo-elizondo]

Small nit: I generally prefer rolling out assertions in unit tests to make it easier to isolate by line-number in the place where the test failed. But I don't think there's a standard in the code base so I'm fine either way

[brandtbucher]

Yeah, I'd like comprehensive testing of these immortals though, and I don't want to roll out ~250 different assertions for each test.

I figured making each one a subTest in assert_immortal is a pretty good compromise.

[eduardo-elizondo]

Nice catch, I could have sworn that I had added the check to _Py_RefcntAdd as well but I somehow missed it. I added two comments but they are minor so this LGTM.

Thanks for the fix!

[sunmy2019]

LGTM, all increments to ob->ob_refcnt should be guarded now.

[JelleZijlstra]

There is a USan failure:

Include/object.h:227:12: runtime error: member access within address 0x7fff7ae78b80 with insufficient space for an object of type 'PyObject' (aka 'struct _object')
0x7fff7ae78b80: note: pointer points here
 b8 7f 00 00  00 47 a8 c2 97 55 00 00  00 b5 59 7e 78 cd 65 f5  00 8a ea c2 b8 7f 00 00  39 7a 90 c0
              ^ 
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Include/object.h:227:12 in 
make: *** [Makefile:1106: checksharedmods] Error 1

Not sure what that means, but the line is in _Py_IsImmortal (https://github.com/brandtbucher/cpython/blob/py-refcnt-add-immortal/Include/object.h#L227), so it may be real.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @JelleZijlstra for commit 1471b39 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[JelleZijlstra]

(Trying again just in case it was something flaky.)

[brandtbucher]

Grrr... it failed again.

[brandtbucher]

I honestly don't know what to do here.

It feels like a bug in USan, since I honestly don't get how the new code could cause that failure. But it's reproducible, and indeed seems to indeed be related to my change here.

[brandtbucher]

Ah, wait, the same buildbot is also failing on main: https://buildbot.python.org/all/#/builders/719/builds/2611

So it's an existing issue (but one that probably should still be looked into).

[sunmy2019]

So it's an existing issue (but one that probably should still be looked into).

See #104190

[brandtbucher]

Nice catch!