_io.TextIOWrapper.write: write during flush causes pending_bytes length mismatch

[chgnrdv]

Crash report

What happened?

Bisected to #24592.
Simple repro:

import _io

class MyIO(_io.BytesIO):
    def __init__(self):
        _io.BytesIO.__init__(self)
        self.writes = []

    def write(self, b):
        self.writes.append(b)
        tw.write("c")
        return len(b)

buf = MyIO()
tw = _io.TextIOWrapper(buf)

CHUNK_SIZE = 8192

tw.write("a" * (CHUNK_SIZE - 1))
tw.write("b" * 2)

tw.flush()

assert b''.join(tw.buffer.writes) == b"a" * (CHUNK_SIZE - 1) + b"b" * 2 + b"c"

On debug build it causes C assertion failure:

python: ./Modules/_io/textio.c:1582: _textiowrapper_writeflush: Assertion `PyUnicode_GET_LENGTH(pending) == self->pending_bytes_count' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7c84537 in __GI_abort () at abort.c:79
#2  0x00007ffff7c8440f in __assert_fail_base (fmt=0x7ffff7dfb688 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x555555a24d40 "PyUnicode_GET_LENGTH(pending) == self->pending_bytes_count", file=0x555555a25332 "./Modules/_io/textio.c", line=1582, 
    function=<optimized out>) at assert.c:92
#3  0x00007ffff7c93662 in __GI___assert_fail (assertion=assertion@entry=0x555555a24d40 "PyUnicode_GET_LENGTH(pending) == self->pending_bytes_count", 
    file=file@entry=0x555555a25332 "./Modules/_io/textio.c", line=line@entry=1582, 
    function=function@entry=0x555555a256b0 <__PRETTY_FUNCTION__.9> "_textiowrapper_writeflush") at assert.c:101
#4  0x00005555559102b9 in _textiowrapper_writeflush (self=self@entry=0x7ffff77896d0) at ./Modules/_io/textio.c:1582
#5  0x000055555591065d in _io_TextIOWrapper_flush_impl (self=0x7ffff77896d0) at ./Modules/_io/textio.c:3092
#6  0x0000555555910791 in _io_TextIOWrapper_flush (self=<optimized out>, _unused_ignored=<optimized out>) at ./Modules/_io/clinic/textio.c.h:1105
#7  0x0000555555693483 in method_vectorcall_NOARGS (func=0x7ffff7731250, args=0x7ffff7fc1070, nargsf=<optimized out>, kwnames=<optimized out>)
    at Objects/descrobject.c:447
#8  0x0000555555680d7c in _PyObject_VectorcallTstate (tstate=0x555555be4678 <_PyRuntime+294136>, callable=0x7ffff7731250, args=0x7ffff7fc1070, 
    nargsf=9223372036854775809, kwnames=0x0) at ./Include/internal/pycore_call.h:168
#9  0x0000555555680e97 in PyObject_Vectorcall (callable=callable@entry=0x7ffff7731250, args=args@entry=0x7ffff7fc1070, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at Objects/call.c:327
#10 0x000055555580876d in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555be4678 <_PyRuntime+294136>, frame=0x7ffff7fc1020, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:813
...

If _io.TextIOWrapper.write() tries to store more than self->chunk_size data in self->pending_bytes, it calls _textiowrapper_writeflush():

cpython/Modules/_io/textio.c

Lines 1726 to 1733 in b48a3db

	else if (self->pending_bytes_count + bytes_len > self->chunk_size) {
	// Prevent to concatenate more than chunk_size data.
	if (_textiowrapper_writeflush(self) < 0) {
	Py_DECREF(b);
	return NULL;
	}
	self->pending_bytes = b;
	}

_textiowrapper_writeflush() flushes self->pending_bytes contents to wrapped buffer through write() method call:

cpython/Modules/_io/textio.c

Lines 1621 to 1628 in b48a3db

	self->pending_bytes_count = 0;
	self->pending_bytes = NULL;
	Py_DECREF(pending);
	PyObject *ret;
	do {
	ret = PyObject_CallMethodOneArg(self->buffer, &_Py_ID(write), b);
	} while (ret == NULL && _PyIO_trap_eintr());

The problem is that call to write() method can cause _io.TextIOWrapper.write() call (directly, as in repro, or from other thread), which re-sets self->pending_bytes and self->pending_bytes_count values.

CPython versions tested on:

3.10, 3.11, 3.12, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a0 (heads/main:e94dbe4ed8, May 24 2024, 00:47:49) [GCC 10.2.1 20210110]

Linked PRs

• gh-119506: fix _io.TextIOWrapper.write() write during flush #119507
• [3.13] gh-119506: fix _io.TextIOWrapper.write() write during flush (GH-119507) #119964
• [3.12] gh-119506: fix _io.TextIOWrapper.write() write during flush (GH-119507) #119965
• [3.11] gh-119506: fix _io.TextIOWrapper.write() write during flush (#119507) #120314