Skip to content

Commit

Permalink
Update benchmarks
Browse files Browse the repository at this point in the history
  • Loading branch information
gi0baro committed Oct 29, 2024
1 parent 8ac2ef7 commit e0a64f5
Show file tree
Hide file tree
Showing 4 changed files with 129 additions and 63 deletions.
45 changes: 23 additions & 22 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@



Run at: Tue 03 Sep 2024, 21:44
Run at: Mon 28 Oct 2024, 02:09
Environment: GHA Linux x86_64 (CPUs: 4)
Python version: 3.11
Granian version: 1.6.0
Granian version: 1.6.2

## RSGI response types

Expand All @@ -14,10 +14,10 @@ The "small" response is 4 bytes, the "big" one is 80kbytes.

| Type | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| bytes small (c64) | 413639 | 41365 | 1.544ms | 4.612ms |
| str small (c128) | 433016 | 43346 | 2.944ms | 19.673ms |
| bytes big (c64) | 281576 | 28162 | 2.267ms | 5.781ms |
| str big (c64) | 288430 | 28851 | 2.211ms | 6.574ms |
| bytes small (c256) | 392764 | 39392 | 6.47ms | 61.121ms |
| str small (c512) | 424544 | 42723 | 11.921ms | 130.529ms |
| bytes big (c64) | 288745 | 28874 | 2.212ms | 6.843ms |
| str big (c64) | 276754 | 27681 | 2.305ms | 5.861ms |


## Interfaces
Expand All @@ -29,15 +29,15 @@ The "echo" request is a 4bytes POST request responding with the same body.

| Request | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| RSGI bytes (c128) | 423456 | 42379 | 3.009ms | 27.917ms |
| RSGI str (c128) | 428954 | 42942 | 2.972ms | 24.262ms |
| RSGI echo (c256) | 377133 | 37812 | 6.735ms | 74.44ms |
| ASGI bytes (c64) | 436301 | 43632 | 1.463ms | 4.525ms |
| ASGI str (c64) | 422612 | 42269 | 1.51ms | 4.182ms |
| ASGI echo (c512) | 217386 | 21864 | 23.309ms | 121.784ms |
| WSGI bytes (c128) | 367719 | 36827 | 3.463ms | 18.241ms |
| WSGI str (c64) | 373343 | 37340 | 1.71ms | 4.425ms |
| WSGI echo (c64) | 342874 | 34293 | 1.861ms | 4.515ms |
| RSGI bytes (c64) | 397815 | 39792 | 1.602ms | 4.506ms |
| RSGI str (c64) | 414448 | 41451 | 1.541ms | 4.152ms |
| RSGI echo (c256) | 366975 | 36780 | 6.93ms | 59.373ms |
| ASGI bytes (c512) | 394155 | 39596 | 12.855ms | 143.895ms |
| ASGI str (c64) | 402564 | 40258 | 1.585ms | 3.888ms |
| ASGI echo (c128) | 217107 | 21742 | 5.864ms | 31.625ms |
| WSGI bytes (c128) | 375703 | 37613 | 3.386ms | 39.83ms |
| WSGI str (c256) | 371867 | 37268 | 6.837ms | 58.755ms |
| WSGI echo (c128) | 352872 | 35325 | 3.609ms | 33.688ms |


## HTTP/2
Expand All @@ -46,10 +46,10 @@ Comparison between Granian HTTP versions on RSGI using 4bytes plain text respons

| Request | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| HTTP/1 [GET] (c128) | 409169 | 40964 | 3.11ms | 35.937ms |
| HTTP/1 [POST] (c256) | 367186 | 36816 | 6.927ms | 63.568ms |
| HTTP/2 [GET] (c64) | 429019 | 42908 | 1.489ms | 6.358ms |
| HTTP/2 [POST] (c64) | 279182 | 27921 | 2.287ms | 6.201ms |
| HTTP/1 [GET] (c64) | 402839 | 40293 | 1.584ms | 3.983ms |
| HTTP/1 [POST] (c64) | 361471 | 36152 | 1.766ms | 5.997ms |
| HTTP/2 [GET] (c128) | 411331 | 41181 | 3.097ms | 35.375ms |
| HTTP/2 [POST] (c256) | 285962 | 28684 | 8.881ms | 102.249ms |


## File responses
Expand All @@ -59,15 +59,16 @@ WSGI is not part of the benchmark since the protocol doesn't implement anything

| Request | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| RSGI (c64) | 340095 | 34014 | 1.877ms | 3.9ms |
| ASGI (c256) | 168845 | 16957 | 14.998ms | 81.803ms |
| ASGI pathsend (c64) | 299960 | 29998 | 2.129ms | 4.663ms |
| RSGI (c64) | 336910 | 33698 | 1.894ms | 9.23ms |
| ASGI (c64) | 167204 | 16723 | 3.818ms | 5.76ms |
| ASGI pathsend (c128) | 290053 | 29038 | 4.392ms | 28.584ms |


### Other benchmarks

- [Versus 3rd party servers](./vs.md)
- [Concurrency](./concurrency.md)
- [Python versions](./pyver.md)

### 3rd party benchmarks

Expand Down
53 changes: 53 additions & 0 deletions benchmarks/pyver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Granian benchmarks



## Python versions

Run at: Mon 28 Oct 2024, 02:17
Environment: GHA Linux x86_64 (CPUs: 4)
Granian version: 1.6.2

Comparison between different Python versions of Granian application protocols using 4bytes plain text response.
Bytes and string response are reported for every protocol just to report the difference with RSGI protocol.
ASGI and WSGI responses are always returned as bytes by the application.
The "echo" request is a 4bytes POST request responding with the same body.

| Python version | Request | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- | --- |
| 3.10 | RSGI bytes (c64) | 445347 | 44540 | 1.433ms | 4.16ms |
| 3.10 | RSGI str (c64) | 407154 | 40713 | 1.569ms | 4.146ms |
| 3.10 | RSGI echo (c64) | 409490 | 40952 | 1.559ms | 3.98ms |
| 3.10 | ASGI bytes (c64) | 404741 | 40475 | 1.577ms | 3.853ms |
| 3.10 | ASGI str (c64) | 443742 | 44379 | 1.439ms | 3.743ms |
| 3.10 | ASGI echo (c64) | 288973 | 28904 | 2.208ms | 4.757ms |
| 3.10 | WSGI bytes (c128) | 559148 | 55978 | 2.28ms | 24.538ms |
| 3.10 | WSGI str (c256) | 567857 | 56935 | 4.476ms | 66.155ms |
| 3.10 | WSGI echo (c512) | 501010 | 50392 | 10.098ms | 117.374ms |
| 3.11 | RSGI bytes (c64) | 408906 | 40897 | 1.56ms | 3.754ms |
| 3.11 | RSGI str (c64) | 444267 | 44427 | 1.437ms | 4.131ms |
| 3.11 | RSGI echo (c256) | 375214 | 37628 | 6.77ms | 69.911ms |
| 3.11 | ASGI bytes (c128) | 394648 | 39496 | 3.232ms | 19.781ms |
| 3.11 | ASGI str (c512) | 416410 | 41848 | 12.182ms | 104.538ms |
| 3.11 | ASGI echo (c64) | 220900 | 22093 | 2.888ms | 5.823ms |
| 3.11 | WSGI bytes (c128) | 381549 | 38197 | 3.339ms | 24.674ms |
| 3.11 | WSGI str (c128) | 377113 | 37754 | 3.377ms | 32.506ms |
| 3.11 | WSGI echo (c256) | 342484 | 34363 | 7.42ms | 66.93ms |
| 3.12 | RSGI bytes (c512) | 423560 | 42590 | 11.958ms | 141.381ms |
| 3.12 | RSGI str (c512) | 427772 | 43035 | 11.842ms | 114.818ms |
| 3.12 | RSGI echo (c512) | 361753 | 36365 | 13.995ms | 169.242ms |
| 3.12 | ASGI bytes (c128) | 419544 | 42014 | 3.032ms | 36.038ms |
| 3.12 | ASGI str (c64) | 415682 | 41567 | 1.536ms | 3.97ms |
| 3.12 | ASGI echo (c64) | 212445 | 21248 | 3.004ms | 6.795ms |
| 3.12 | WSGI bytes (c256) | 356792 | 35798 | 7.121ms | 64.292ms |
| 3.12 | WSGI str (c128) | 362003 | 36244 | 3.52ms | 20.129ms |
| 3.12 | WSGI echo (c512) | 322632 | 32432 | 15.676ms | 144.224ms |
| 3.13 | RSGI bytes (c512) | 400784 | 40264 | 12.635ms | 175.762ms |
| 3.13 | RSGI str (c64) | 407448 | 40745 | 1.568ms | 4.455ms |
| 3.13 | RSGI echo (c128) | 384481 | 38485 | 3.315ms | 23.927ms |
| 3.13 | ASGI bytes (c512) | 414131 | 41618 | 12.238ms | 147.581ms |
| 3.13 | ASGI str (c64) | 449520 | 44963 | 1.419ms | 3.919ms |
| 3.13 | ASGI echo (c128) | 225465 | 22567 | 5.648ms | 36.339ms |
| 3.13 | WSGI bytes (c128) | 371324 | 37194 | 3.429ms | 18.313ms |
| 3.13 | WSGI str (c64) | 369008 | 36910 | 1.729ms | 5.06ms |
| 3.13 | WSGI echo (c128) | 363711 | 36408 | 3.505ms | 20.864ms |
3 changes: 2 additions & 1 deletion benchmarks/templates/_vs_ws_table.tpl
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
| Server | Send throughput | Receive throughput | Throughput (total) |
| Server | Send throughput | Receive throughput | Combined throughput |
| --- | --- | --- | --- |
{{ for key, runs in _data.items(): }}
{{ max_c, run = get_max_concurrency_run(runs, "throughput", "sum") }}
{{ if run["throughput"]["sum"]: }}
| {{ =key }} (c{{ =max_c }}) | {{ =round(run["throughput"]["send"]) }} | {{ =round(run["throughput"]["recv"]) }} | {{ =round(run["throughput"]["sum"]) }} |
{{ pass }}
{{ pass }}
91 changes: 51 additions & 40 deletions benchmarks/vs.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,57 +4,57 @@

## VS 3rd party comparison

Run at: Tue 03 Sep 2024, 21:44
Run at: Mon 28 Oct 2024, 02:10
Environment: GHA Linux x86_64 (CPUs: 4)
Python version: 3.11
Granian version: 1.6.0
Granian version: 1.6.2

### ASGI

| Server | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| Granian Asgi [GET] (c128) | 453458 | 45416 | 2.806ms | 31.751ms |
| Granian Asgi [POST] (c64) | 219795 | 21983 | 2.904ms | 6.354ms |
| Uvicorn H11 [GET] (c64) | 80046 | 8006 | 7.972ms | 18.857ms |
| Uvicorn H11 [POST] (c64) | 71376 | 7139 | 8.942ms | 24.293ms |
| Uvicorn Httptools [GET] (c128) | 371460 | 37210 | 3.424ms | 32.85ms |
| Uvicorn Httptools [POST] (c128) | 342512 | 34301 | 3.718ms | 24.748ms |
| Hypercorn [GET] (c128) | 49725 | 4978 | 25.595ms | 31.829ms |
| Hypercorn [POST] (c128) | 45388 | 4547 | 28.022ms | 31.55ms |
| Granian Asgi [GET] (c64) | 412041 | 41213 | 1.549ms | 4.255ms |
| Granian Asgi [POST] (c64) | 221079 | 22111 | 2.888ms | 5.488ms |
| Uvicorn H11 [GET] (c64) | 75123 | 7513 | 8.5ms | 22.009ms |
| Uvicorn H11 [POST] (c64) | 66085 | 6609 | 9.657ms | 26.091ms |
| Uvicorn Httptools [GET] (c128) | 339905 | 34045 | 3.745ms | 23.819ms |
| Uvicorn Httptools [POST] (c128) | 314202 | 31455 | 4.055ms | 23.736ms |
| Hypercorn [GET] (c128) | 48884 | 4893 | 26.054ms | 42.096ms |
| Hypercorn [POST] (c128) | 42517 | 4257 | 29.922ms | 54.175ms |


### WSGI

| Server | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| Granian Wsgi [GET] (c256) | 380226 | 38107 | 6.691ms | 57.374ms |
| Granian Wsgi [POST] (c128) | 332496 | 33295 | 3.829ms | 33.119ms |
| Gunicorn Gthread [GET] (c64) | 36866 | 3687 | 17.304ms | 19.372ms |
| Gunicorn Gthread [POST] (c64) | 35502 | 3551 | 17.962ms | 20.207ms |
| Gunicorn Gevent [GET] (c64) | 63589 | 6361 | 6.593ms | 7589.315ms |
| Gunicorn Gevent [POST] (c256) | 60097 | 6033 | 7.233ms | 9884.01ms |
| Uwsgi [GET] (c256) | 72701 | 7286 | 33.627ms | 4735.69ms |
| Uwsgi [POST] (c256) | 72121 | 7227 | 34.111ms | 3470.737ms |
| Granian Wsgi [GET] (c256) | 381571 | 38324 | 6.644ms | 70.442ms |
| Granian Wsgi [POST] (c128) | 347832 | 34819 | 3.662ms | 34.109ms |
| Gunicorn Gthread [GET] (c64) | 36680 | 3669 | 17.386ms | 19.742ms |
| Gunicorn Gthread [POST] (c64) | 35078 | 3508 | 18.191ms | 20.25ms |
| Gunicorn Gevent [GET] (c128) | 62624 | 6270 | 7.909ms | 9876.337ms |
| Gunicorn Gevent [POST] (c128) | 57223 | 5729 | 12.683ms | 9777.452ms |
| Uwsgi [GET] (c64) | 71314 | 7131 | 8.955ms | 16.786ms |
| Uwsgi [POST] (c64) | 71214 | 7122 | 8.964ms | 13.792ms |


### HTTP/2

| Server | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| Granian Asgi [GET] (c64) | 373977 | 37408 | 1.706ms | 6.236ms |
| Granian Asgi [POST] (c256) | 197345 | 19804 | 12.862ms | 92.805ms |
| Hypercorn [GET] (c64) | 31630 | 3164 | 20.144ms | 48.95ms |
| Hypercorn [POST] (c64) | 28163 | 2817 | 22.641ms | 69.083ms |
| Granian Asgi [GET] (c128) | 346031 | 34673 | 3.673ms | 44.303ms |
| Granian Asgi [POST] (c64) | 196285 | 19633 | 3.25ms | 10.419ms |
| Hypercorn [GET] (c64) | 30507 | 3051 | 20.914ms | 50.49ms |
| Hypercorn [POST] (c64) | 27155 | 2716 | 23.491ms | 56.879ms |


### ASGI file responses

| Server | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| Granian (pathsend) (c64) | 301264 | 30123 | 2.12ms | 5.324ms |
| Uvicorn H11 (c128) | 78809 | 7893 | 16.152ms | 27.46ms |
| Uvicorn Httptools (c128) | 207081 | 20727 | 6.152ms | 27.753ms |
| Hypercorn (c128) | 49403 | 4945 | 25.77ms | 32.498ms |
| Granian (pathsend) (c64) | 296068 | 29608 | 2.157ms | 4.387ms |
| Uvicorn H11 (c128) | 74931 | 7505 | 16.979ms | 43.618ms |
| Uvicorn Httptools (c64) | 190766 | 19080 | 3.347ms | 7.536ms |
| Hypercorn (c128) | 47773 | 4782 | 26.646ms | 44.924ms |


### Long I/O
Expand All @@ -63,18 +63,29 @@ Plain text 4 bytes response comparison simulating *long* I/O waits (10ms and 100

| Server | Total requests | RPS | avg latency | max latency |
| --- | --- | --- | --- | --- |
| Granian Rsgi 10ms (c512) | 400686 | 40280 | 12.66ms | 105.734ms |
| Granian Rsgi 100ms (c512) | 50049 | 5033 | 100.68ms | 157.381ms |
| Granian Asgi 10ms (c512) | 400154 | 40203 | 12.673ms | 113.876ms |
| Granian Asgi 100ms (c512) | 50070 | 5037 | 100.641ms | 171.637ms |
| Granian Wsgi 10ms (c128) | 112522 | 11265 | 11.315ms | 31.271ms |
| Granian Wsgi 100ms (c512) | 50253 | 5054 | 100.295ms | 150.603ms |
| Uvicorn Httptools 10ms (c512) | 246558 | 24794 | 20.545ms | 104.779ms |
| Uvicorn Httptools 100ms (c512) | 50019 | 5028 | 100.811ms | 174.095ms |
| Hypercorn 10ms (c128) | 49761 | 4982 | 25.581ms | 42.662ms |
| Hypercorn 100ms (c128) | 49443 | 4949 | 25.719ms | 43.894ms |
| Gunicorn Gevent 10ms (c64) | 57483 | 5750 | 11.088ms | 25.63ms |
| Gunicorn Gevent 100ms (c512) | 48377 | 4862 | 104.317ms | 171.487ms |
| Uwsgi 10ms (c256) | 72900 | 7311 | 34.199ms | 6609.152ms |
| Uwsgi 100ms (c512) | 72643 | 7304 | 61.63ms | 7512.077ms |
| Granian Rsgi 10ms (c512) | 398741 | 40076 | 12.704ms | 122.917ms |
| Granian Rsgi 100ms (c512) | 50074 | 5035 | 100.659ms | 161.13ms |
| Granian Asgi 10ms (c512) | 368346 | 37047 | 13.753ms | 126.133ms |
| Granian Asgi 100ms (c512) | 50056 | 5031 | 100.724ms | 151.241ms |
| Granian Wsgi 10ms (c128) | 109932 | 11005 | 11.578ms | 34.242ms |
| Granian Wsgi 100ms (c512) | 50218 | 5050 | 100.365ms | 162.437ms |
| Uvicorn Httptools 10ms (c256) | 227853 | 22854 | 11.14ms | 67.594ms |
| Uvicorn Httptools 100ms (c512) | 50074 | 5028 | 100.869ms | 181.026ms |
| Hypercorn 10ms (c128) | 48782 | 4884 | 26.081ms | 46.542ms |
| Hypercorn 100ms (c128) | 49325 | 4938 | 25.812ms | 46.076ms |
| Gunicorn Gevent 10ms (c128) | 55121 | 5519 | 23.096ms | 60.147ms |
| Gunicorn Gevent 100ms (c512) | 47896 | 4812 | 105.287ms | 191.957ms |
| Uwsgi 10ms (c256) | 71141 | 7136 | 34.26ms | 3345.351ms |
| Uwsgi 100ms (c256) | 70790 | 7102 | 34.882ms | 3071.967ms |


### Websockets

Websocket broadcasting comparison with concurrent clients sending a predefined amount of messages and receiving those messages from all the connected clients. The benchmark takes the time required for the test to run and compute the relevant throughput (in messages per second).

| Server | Send throughput | Receive throughput | Combined throughput |
| --- | --- | --- | --- |
| Granian Rsgi (c16) | 962877 | 76296 | 81065 |
| Granian Asgi (c16) | 941480 | 73632 | 78234 |
| Uvicorn H11 (c8) | 456731 | 80133 | 90149 |

0 comments on commit e0a64f5

Please sign in to comment.