Noticeable performance downgrade from Python 3.10 to onwards versions

[gi0baro]

Hi 👋 I'm not sure wether this makes sense as an issue or should be a discussion instead: to project maintainers, feel free to move this to a discussion if you conclude that's better.

Context: in the Granian project (an HTTP server) we recently introduced some e2e benchmarks using different Python versions, which show a ~30% performance degradation for some tests when comparing Python 3.10 to all other versions onwards (PyO3 0.22, cfg pyo3_disable_reference_pool).

Now, the specific tests showing this degradation involves some relatively simple code:

• we create both a pyclass and a PyDict objects (https://github.com/emmett-framework/granian/blob/c94e73e32a4865a011a4b659ef04bbc0a96e6fd4/src/wsgi/callbacks.rs#L24-L106)
• call a python function with those arguments (https://github.com/emmett-framework/granian/blob/c94e73e32a4865a011a4b659ef04bbc0a96e6fd4/granian/wsgi.py#L48-L58)
• that will call back a method of the first pyclass object (https://github.com/emmett-framework/granian/blob/c94e73e32a4865a011a4b659ef04bbc0a96e6fd4/src/wsgi/io.rs#L49-L57)

While I understand an e2e benchmark might suffer from a lot of additional noise when compared to a smaller unit-benchmark, and there's a lot more to consider (network stack in CPython stdlib, for example), I also believe, given other protocols involving asyncio and a bunch more stuff suffer from a very smaller degradation compared to the one I referenced, there might be something going on in PyO3 <-> CPython interop. Thus I have two main questions:

• is there any well-known difference from Python 3.11 and onwards in how PyO3 interact with the Python interpreter that might explain this?
• do you have any suggestions on how to investigate this in a more fine-grained way to help enlighten any other differences between Python versions that might play a role in this?

Thanks in advance 🙏

[davidhewitt]

Thanks for the questions. I don't have an immediate answer for you; to check I understand, Python 3.11 and up got ~30% slower?

Have you tried generating flame graphs (e.g. with samply) to see if that gives a hint where the differences come from?

[gi0baro]

to check I understand, Python 3.11 and up got ~30% slower?

@davidhewitt correct, here is a more direct comparison extracted from that bench:

Python version	Total requests	RPS	avg latency	max latency
3.10	559148	55978	2.28ms	24.538ms
3.11	381549	38197	3.339ms	24.674ms
3.12	356792	35798	7.121ms	64.292ms
3.13	371324	37194	3.429ms	18.313ms

Have you tried generating flame graphs (e.g. with samply) to see if that gives a hint where the differences come from?

that makes sense. Let me plan some tests using sampling, I'll post here the findings.