ASAN failure was detected while running test_threading

[corona10]

See: https://github.com/python/cpython/actions/runs/4971544925/jobs/8896154582
Unicodeobject looks quite related.

2023-05-14T09:18:05.5991524Z Direct leak of 1702 byte(s) in 27 object(s) allocated from:
2023-05-14T09:18:05.5991966Z     #0 0x7fb1e4894c47 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
2023-05-14T09:18:05.5992397Z     #1 0x55f4ae5903f7 in PyUnicode_New Objects/unicodeobject.c:1207
2023-05-14T09:18:05.5992778Z     #2 0x55f4ae5f3da8 in PyUnicode_New Objects/unicodeobject.c:1156
2023-05-14T09:18:05.5993182Z     #3 0x55f4ae5f3da8 in unicode_decode_utf8 Objects/unicodeobject.c:4537
2023-05-14T09:18:05.5993607Z     #4 0x55f4ae4e689a in PyDict_SetItemString Objects/dictobject.c:3914
2023-05-14T09:18:05.5994016Z     #5 0x55f4ae858ed0 in setup_confname_table Modules/posixmodule.c:13345
2023-05-14T09:18:05.5994429Z     #6 0x55f4ae858ed0 in setup_confname_tables Modules/posixmodule.c:13367
2023-05-14T09:18:05.5994830Z     #7 0x55f4ae858ed0 in posixmodule_exec Modules/posixmodule.c:16663
2023-05-14T09:18:05.5995232Z     #8 0x55f4ae510890 in PyModule_ExecDef Objects/moduleobject.c:440
2023-05-14T09:18:05.5995683Z     #9 0x55f4ae75cc7a in _imp_exec_builtin (/home/runner/work/cpython/cpython/python+0x72fc7a)
2023-05-14T09:18:05.5996126Z     #10 0x55f4ae50b4dc in cfunction_vectorcall_O Objects/methodobject.c:509
2023-05-14T09:18:05.5996568Z     #11 0x55f4ae42a4c6 in PyObject_Call (/home/runner/work/cpython/cpython/python+0x3fd4c6)
2023-05-14T09:18:05.6133164Z     #12 0x55f4ae2de55d in _PyEval_EvalFrameDefault Python/bytecodes.c:3157
2023-05-14T09:18:05.6134128Z     #13 0x55f4ae42415e in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
2023-05-14T09:18:05.6134546Z     #14 0x55f4ae42415e in object_vacall Objects/call.c:850
2023-05-14T09:18:05.6134938Z     #15 0x55f4ae427dfe in PyObject_CallMethodObjArgs Objects/call.c:911
2023-05-14T09:18:05.6135351Z     #16 0x55f4ae766927 in import_find_and_load Python/import.c:2715
2023-05-14T09:18:05.6135778Z     #17 0x55f4ae766927 in PyImport_ImportModuleLevelObject Python/import.c:2798
2023-05-14T09:18:05.6136174Z     #18 0x55f4ae2fb183 in import_name Python/ceval.c:2386
2023-05-14T09:18:05.6136645Z     #19 0x55f4ae2fb183 in _PyEval_EvalFrameDefault Python/bytecodes.c:2031
2023-05-14T09:18:05.6137070Z     #20 0x55f4ae6cf786 in _PyEval_EvalFrame Include/internal/pycore_ceval.h:87
2023-05-14T09:18:05.6137459Z     #21 0x55f4ae6cf786 in _PyEval_Vector Python/ceval.c:1611
2023-05-14T09:18:05.6137826Z     #22 0x55f4ae6cf786 in PyEval_EvalCode Python/ceval.c:567
2023-05-14T09:18:05.6138204Z     #23 0x55f4ae6c9061 in builtin_exec_impl Python/bltinmodule.c:1079
2023-05-14T09:18:05.6138615Z     #24 0x55f4ae6c9061 in builtin_exec Python/clinic/bltinmodule.c.h:586
2023-05-14T09:18:05.6139068Z     #25 0x55f4ae50c015 in cfunction_vectorcall_FASTCALL_KEYWORDS Objects/methodobject.c:438
2023-05-14T09:18:05.6139533Z     #26 0x55f4ae424f8f in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
2023-05-14T09:18:05.6139944Z     #27 0x55f4ae424f8f in PyObject_Vectorcall Objects/call.c:325
2023-05-14T09:18:05.6140349Z     #28 0x55f4ae2db394 in _PyEval_EvalFrameDefault Python/bytecodes.c:2609
2023-05-14T09:18:05.6140941Z     #29 0x55f4ae42415e in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
2023-05-14T09:18:05.6141332Z     #30 0x55f4ae42415e in object_vacall Objects/call.c:850
2023-05-14T09:18:05.6141720Z     #31 0x55f4ae427dfe in PyObject_CallMethodObjArgs Objects/call.c:911
2023-05-14T09:18:05.6142136Z     #32 0x55f4ae766927 in import_find_and_load Python/import.c:2715
2023-05-14T09:18:05.6142564Z     #33 0x55f4ae766927 in PyImport_ImportModuleLevelObject Python/import.c:2798
2023-05-14T09:18:05.6142967Z     #34 0x55f4ae2fb183 in import_name Python/ceval.c:2386
2023-05-14T09:18:05.6143360Z     #35 0x55f4ae2fb183 in _PyEval_EvalFrameDefault Python/bytecodes.c:2031
2023-05-14T09:18:05.6143799Z     #36 0x55f4ae42490f in _PyObject_VectorcallTstate Include/internal/pycore_call.h:92
2023-05-14T09:18:05.6144249Z     #37 0x55f4ae42490f in _PyObject_CallNoArgsTstate Include/internal/pycore_call.h:99
2023-05-14T09:18:05.6144663Z     #38 0x55f4ae42490f in _PyObject_CallFunctionVa Objects/call.c:535
2023-05-14T09:18:05.6145036Z     #39 0x55f4ae4262e3 in callmethod Objects/call.c:634
2023-05-14T09:18:05.6145400Z     #40 0x55f4ae4262e3 in PyObject_CallMethod Objects/call.c:653
2023-05-14T09:18:05.6145797Z     #41 0x55f4ae769ca2 in init_importlib_external Python/import.c:2265
2023-05-14T09:18:05.6146192Z     #42 0x55f4ae769ca2 in _PyImport_InitExternal Python/import.c:3186
2023-05-14T09:18:05.6146587Z     #43 0x55f4ae7b30f3 in init_interp_main Python/pylifecycle.c:1116
2023-05-14T09:18:05.6146981Z     #44 0x55f4ae7b48c4 in pyinit_main Python/pylifecycle.c:1227
2023-05-14T09:18:05.6147395Z     #45 0x55f4ae7b76af in Py_InitializeFromConfig Python/pylifecycle.c:1258
2023-05-14T09:18:05.6147773Z     #46 0x55f4ae829c19 in pymain_init Modules/main.c:67

cc @gpshead

Linked PRs

• gh-104472: Skip test_subprocess.ProcessTestCase.test_empty_env if ASAN is enabled #104667
• [3.10] gh-104472: Skip test_subprocess.ProcessTestCase.test_empty_env if ASAN is enabled (GH-104667) #104669
• [3.11] gh-104472: Skip test_subprocess.ProcessTestCase.test_empty_env if ASAN is enabled (GH-104667) #104673

[sunmy2019]

cpython leaks many things, and memory leaks are usually suppressed by environment variables:

env:
  ASAN_OPTIONS: detect_leaks=0:allocator_may_return_null=1:handle_segv=0

[sunmy2019]

Somehow the env is not set correctly in this test. May be cleared by some previous tests.

[sunmy2019]

This issue cannot be reproduced in the following build bot tests.

I think the only thing we can do here is to ensure the env is not changed. But since it's not occurring, it may not be worth doing this.

[chgnrdv]

I can reproduce this leak with current main on my machine, along with a large number of other leaks related to PyUnicode_New. I'm not competent enough, but I think that some of them may be caused by the fact that Python frees interned heap-allocated strings at interpreter finalization only if it is build with Py_DEBUG defined. It doesn't explain why ASAN failed on this particular test though.

[corona10]

cc @methane

[sunmy2019]

I can reproduce this leak with current main on my machine

even with leak_detect suppressed?

[chgnrdv]

Ah, it doesn't leak when I suppress leak detection with ASAN_OPTIONS set. @corona10 @sunmy2019, sorry for the noise.

[chgnrdv]

OK, seems like I can reproduce this by running only this particular test case:

export ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0"
./python -m unittest test.test_subprocess.ProcessTestCase.test_empty_env

[sunmy2019]

OK, seems like I can reproduce this by running only this particular test case:

Indeed. We should skip it.

from test.support import check_sanitizer


    @unittest.skipIf(check_sanitizer(address=True),
                     'AddressSanitizer cannot be runned with empty env')