Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-112536: Set up TSAN CI for free-threading #116555

Closed
wants to merge 19 commits into from
24 changes: 24 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,29 @@ jobs:
- name: Tests
run: xvfb-run make test

build_tsan_free_threading:
name: 'Thread sanitizer (free-threading)'
runs-on: ubuntu-20.04
timeout-minutes: 60
needs: check_source
if: needs.check_source.outputs.run_tests == 'true'
steps:
- uses: actions/checkout@v4
- name: Install Dependencies
run: sudo ./.github/workflows/posix-deps-apt.sh
- name: Set up GCC-10 for ASAN
uses: egor-tensin/setup-gcc@v1
with:
version: 11
Comment on lines +485 to +488
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version says "11" but the step name says "GCC-10"

- name: Configure CPython
run: ./configure --disable-gil --with-thread-sanitizer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want --with-pydebug?

- name: Build CPython
run: make -j4
- name: Display build info
run: make pythoninfo
- name: Tests
run: ./python -m test --pgo -j4 # Reduce test scope
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll want a configuration specifically for tsan. Maybe a --tsan option? We'll only want to run tests that actually use threading.

Brett had a list of tests that fail with PYTHON_GIL=0 -- those are probably a good starting point:

swtaarrs@1fe9165#diff-590c02490d066378045dd48d62e86dc85d3fb4ad934b6a2b2b0b1112e59eaefb


# CIFuzz job based on https://google.github.io/oss-fuzz/getting-started/continuous-integration/
cifuzz:
name: CIFuzz
Expand Down Expand Up @@ -561,6 +584,7 @@ jobs:
build_windows,
build_windows_free_threading,
build_asan,
build_tsan_free_threading,
'
|| ''
}}
Expand Down
1 change: 1 addition & 0 deletions Objects/obmalloc.c
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,7 @@ _PyMem_ArenaFree(void *Py_UNUSED(ctx), void *ptr,
/***************************/

static int
_Py_NO_SANITIZE_THREAD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a better experience using suppression lists rather than marking functions as _Py_NO_SANITIZE_THREAD:

  1. Add a file supressions.txt somwhere (maybe Tools/tsan/supressions.txt?)
  2. Set the environment variable TSAN_OPTIONS="suppressions=<path_to_supressions.txt>"

See https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions

set_default_allocator_unlocked(PyMemAllocatorDomain domain, int debug,
PyMemAllocatorEx *old_alloc)
{
Expand Down
7 changes: 7 additions & 0 deletions configure

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -3264,6 +3264,12 @@ AC_MSG_RESULT([no])
with_tsan="no"
])

if test "$with_tsan" = "yes"
then
AC_DEFINE([MI_TSAN], [1],
[Define MI_TSAN for thread sanitizer])
fi

# Set info about shared libraries.
AC_SUBST([SHLIB_SUFFIX])
AC_SUBST([LDSHARED])
Expand Down
3 changes: 3 additions & 0 deletions pyconfig.h.in
Original file line number Diff line number Diff line change
Expand Up @@ -1588,6 +1588,9 @@
<sysmacros.h>. */
#undef MAJOR_IN_SYSMACROS

/* Define MI_TSAN for thread sanitizer */
#undef MI_TSAN

/* Define if mvwdelch in curses.h is an expression. */
#undef MVWDELCH_IS_EXPRESSION

Expand Down
Loading