gh-89550: Buffer GzipFile.write to reduce execution time by ~15%

[CCLDArjun]

Currently, all of this is done for every write() call:
https://github.com/python/cpython/blob/main/Lib/gzip.py#L266-L288

This pr stores GzipFile.write() data into a buffer to postpone most of the compression work to when the buffer flushes. This way, we can call into zlib and File.write() less frequently.

benchmarking with timeit:

../python -m timeit -s 'import gzip
' 'with gzip.open("./Python-3.11.1.tgz", "rb") as infile:
' '    with gzip.open("./Python-3.11.1new.tgz", "wb") as outfile:
' '        for line in infile:
' '            outfile.write(line)
' '
'
1 loop, best of 5: 6.36 sec per loop

vs

../python -m timeit -s 'import gzipold as gzip
' 'with gzip.open("./Python-3.11.1.tgz", "rb") as infile:
' '    with gzip.open("./Python-3.11.1new.tgz", "wb") as outfile:
' '        for line in infile:
' '            outfile.write(line)
' '
'
1 loop, best of 5: 7.68 sec per loop

• Issue: GzipFile.write should be buffered #89550

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[rhpvorderman]

Wow @CCLDArjun such a simple and effective solution. My solution was to split out a _GzipWriter (just like _GzipReader) and try to work from there, but I never got that to work easily and it was quite some extra code too. I am so glad you made this. I hope this will get merged soon!

[CCLDArjun]

My solution was to split out a _GzipWriter (just like _GzipReader) and try to work from there

Initially that was my idea too, where I subclassed _pyio.BufferedWriter to override its flush methods but that also wasn't working easily and was pretty messy: 943ca9c

[rhpvorderman]

I am glad that you were able to come up with a better idea. I have been hacking at the gzip module pretty thoroughly but I could not come up with this. I am in awe 😀 . I will port this to python-isal and I have recently started on making a zlib-ng module too. That way people will not have to wait for 3.12 to ship to get all this fast compression.

[orsenthil]

This looks good to me. A core developer in the expert list will review before merging.

[rhpvorderman]

@orsenthil This is still labeled "awaiting review" rather then "awaiting core review" does that matter?

[arhadthedev]

@rhpvorderman Only review messages from the Python team count.

@Yhg1s, @gpshead as zipfile experts (not quite the same but still).

[gpshead]

Overall I like this change. It's great how cleanly this can be added. :)

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[CCLDArjun]

@gpshead tests are passing! Can you re review?

[rhpvorderman]

ping

It would be great if this gets into 3.12. The measured execution difference of 15% is at gzip level 9 which is the default. At a more modest compression level such as 1, the difference can be a couple of hundreds of percents.

[CCLDArjun]

pinging @gpshead for a final review