-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement zero copy writes in StreamWriter
#9839
Conversation
CodSpeed Performance ReportMerging #9839 will improve performances by 53.25%Comparing Summary
Benchmarks breakdown
|
Well that was much better than expected. We don't have a benchmark for compressed payloads here. We should add that as well |
Codecov ReportAll modified and coverable lines are covered by tests ✅
✅ All tests successful. No failed tests found. Additional details and impacted files@@ Coverage Diff @@
## master #9839 +/- ##
=======================================
Coverage 98.70% 98.70%
=======================================
Files 118 118
Lines 36174 36261 +87
Branches 4297 4311 +14
=======================================
+ Hits 35706 35793 +87
Misses 315 315
Partials 153 153
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
StreamWriter
185a8bd
to
f337ca5
Compare
StreamWriter
StreamWriter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, many years ago transport.writelines(...)
was implemented as transport.write(b''.join(lines))
.
Now it utilizes sendmsg()
which is more efficient than memcpy()
+ send()
Still needs a test for a large compressed payload without chunked that is large enough to ensure compress and flush both output a payload |
Backport to 3.11: 💔 cherry-picking failed — conflicts found❌ Failed to cleanly apply a9a0d84 on top of patchback/backports/3.11/a9a0d849bd4ad65724710662e73f29f1e968bfe6/pr-9839 Backporting merged PR #9839 into master
🤖 @patchback |
(cherry picked from commit a9a0d84)
python/cpython#91166 made
writelines
zero copy for Python 3.12+.finishing a chunk has a lot of
memcpy
. Even if zero copy isn't implement for old python it will useb"".join()
since it has a more efficient implementation in https://github.com/python/cpython/blob/91f4908798074db6c41925b4417bee1f933aca93/Objects/stringlib/join.h#L36 vs constructing a new bytes string for every add operation #9838 shows the bytes join is also faster, just not as fast as the zero copy write