gh-125196: Use PyUnicodeWriter in error handlers

[vstinner]

PyCodec_ReplaceErrors() and PyCodec_BackslashReplaceErrors() now use the public PyUnicodeWriter API.

• Issue: Use the public PyUnicodeWriter API #125196

[rruuaanng]

Wouldn't it be better to move this inline function into the header file? That way it can omit the declaration in the header file and integrate well. It looks like a public API.

[vstinner]

In fact, I implemented #125201 which should replace this static inline function.

[vstinner]

Benchmark:

import codecs
import pyperf

runner = pyperf.Runner()
replace = codecs.lookup_error('replace')
backslashreplace = codecs.lookup_error('backslashreplace')

for LEN in (1, 1000):
    text = "\xff" * LEN
    exc = UnicodeEncodeError("ascii", text, 0, len(text), "reason")
    runner.bench_func(f"replace encode len={LEN}", replace, exc)

    text = "\xff" * LEN
    exc = UnicodeTranslateError(text, 0, len(text), "reason")
    runner.bench_func(f"replace translate len={LEN}", replace, exc)

    data = b"\x20\x80\xff" * LEN
    exc = UnicodeDecodeError("ascii", data, 0, len(data), "reason")
    runner.bench_func(f"backslashreplace decode len={LEN}", backslashreplace, exc)

    text = "\x20\xff\u20ac\U0010ffff" * LEN
    exc = UnicodeEncodeError("ascii", text, 0, len(text), "reason")
    runner.bench_func(f"backslashreplace encode len={LEN}", backslashreplace, exc)

    text = "\x20\xff\u20ac\U0010ffff" * LEN
    exc = UnicodeTranslateError(text, 0, len(text), "reason")
    runner.bench_func(f"backslashreplace translate len={LEN}", backslashreplace, exc)

Results, Python built with gcc -O3, CPU isolation:

+-------------------------------------+---------+------------------------+
| Benchmark                           | ref     | change                 |
+=====================================+=========+========================+
| replace encode len=1                | 92.9 ns | 110 ns: 1.19x slower   |
+-------------------------------------+---------+------------------------+
| replace translate len=1             | 103 ns  | 138 ns: 1.34x slower   |
+-------------------------------------+---------+------------------------+
| backslashreplace decode len=1       | 95.6 ns | 140 ns: 1.46x slower   |
+-------------------------------------+---------+------------------------+
| backslashreplace encode len=1       | 120 ns  | 166 ns: 1.37x slower   |
+-------------------------------------+---------+------------------------+
| backslashreplace translate len=1    | 126 ns  | 169 ns: 1.35x slower   |
+-------------------------------------+---------+------------------------+
| replace encode len=1000             | 157 ns  | 2.09 us: 13.31x slower |
+-------------------------------------+---------+------------------------+
| replace translate len=1000          | 178 ns  | 2.23 us: 12.51x slower |
+-------------------------------------+---------+------------------------+
| backslashreplace decode len=1000    | 3.03 us | 27.2 us: 8.98x slower  |
+-------------------------------------+---------+------------------------+
| backslashreplace encode len=1000    | 14.4 us | 46.3 us: 3.22x slower  |
+-------------------------------------+---------+------------------------+
| backslashreplace translate len=1000 | 14.4 us | 45.9 us: 3.18x slower  |
+-------------------------------------+---------+------------------------+
| Geometric mean                      | (ref)   | 3.03x slower           |
+-------------------------------------+---------+------------------------+

It's way slower :-(

This PR has a naive PyUnicodeWriter_Fill() implementation calling PyUnicodeWriter_WriteChar() in a loop. I wrote PR gh-125201 which is way more efficient:

+-------------------------------------+---------+------------------------+-----------------------+
| Benchmark                           | ref     | change                 | fill                  |
+=====================================+=========+========================+=======================+
| replace encode len=1000             | 157 ns  | 2.09 us: 13.31x slower | 176 ns: 1.12x slower  |
+-------------------------------------+---------+------------------------+-----------------------+
| replace translate len=1000          | 178 ns  | 2.23 us: 12.51x slower | 276 ns: 1.55x slower  |
+-------------------------------------+---------+------------------------+-----------------------+

[vstinner]

For the benchmark, I had to call directly the error handler functions. Using ASCII doesn't work (it doesn't measure my change), since Objects/unicodeobject.c has a fast-path for most common error handlers such as replace and backslashreplace.

[vstinner]

Let's keep the private API for now, since it's faster.