Skip to content

Commit

Permalink
fix: Flaky fluent-bit load test setup (#1404)
Browse files Browse the repository at this point in the history 
Co-authored-by: Nina Hingerl <[email protected]>
  • Loading branch information
@hisarbalik @NHingerl
hisarbalik and NHingerl authored Sep 10, 2024
1 parent cdeecfa commit 2f2dd9e
Show file tree
Hide file tree
Showing 3 changed files with 37 additions and 32 deletions.
30 changes: 17 additions & 13 deletions docs/contributor/benchmarks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -337,21 +337,25 @@ test results are printed out.

<div class="table-wrapper" markdown="block">

| Version/Test | Single Pipeline (ci-logs) | | | | | Multi Pipeline (ci-logs-m) | | | | | Single Pipeline Backpressure (ci-logs-b) | | | | | Multi Pipeline Backpressure (ci-logs-mb) | | | | |
|--------------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|
| | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage |
| 2.2.1 | 5165 | 8541 | 68518 | 172, 190 | 1, 1 | 2009 | 2195 | 102932 | 332, 320 | 0.9, 0.9 | 5914 | 1498 | 79247 | 184, 176 | 0.9, 1 | 1979 | 489 | 83442 | 310, 322 | 0.9, 0.9 |
| 2.2.2 | 5159 | 7811 | 75545 | 171, 170 | 1, 1 | 1910 | 2516 | 103780 | 324, 324 | 0.9, 0.9 | 5857 | 1513 | 72494 | 189, 200 | 1, 1 | 1860 | 421 | 90852 | 314, 322 | 0.9, 0.9 |
| 2.2.2 (new setup) | 5445 | 9668 | 68453 | 248, 981 | 1, 1 | 6201 | 2747 | 89291 | 544, 720 | 1, 1 | 6009 | 1723 | 58982 | 650, 682 | 1, 1 | 6111 | 385 | 108909 | 686, 931 | 0.9, 0.9 |
| 3.0.3 | 9483 | 22042 | 53251 | 366, 681 | 1, 1 | 10737 | 8785 | 115232 | 953, 568 | 0.9, 0.9 | 10425 | 4610 | 80614 | 856, 704 | 0.9, 0.9 | 10955 | 1724 | 87530 | 503, 594 | 0.9 ,0.9 |
| 3.0.4 | 4341 | 8296 | 35628 | 971, 726 | 0.1,0.1 | 1201 | 544 | 103624 | 652, 815 | 0, 0 | 932 | 297 | 37663 | 615, 726 | 0.1,0.1 | 1477 | 171 | 108885 | 530, 566 | 0, 0.1 |
| 3.0.7 (old metrics) | 4241 | 7782 | 47586 | 815,1021 | 0.7,0.1 | 3809 | 1968 | 107529 | 837,965 | 0.4,0 | 3472 | 1093 | 33818 | 792,597 | 0,0.1 | 2180 | 177 | 87052 | 708,631 | 0,0.1 |
| 3.0.7 (new metrics) | 4036 | 7173 | 31689 | 825,852 | 0.1,0.1 | 2481 | 1852 | 104689 | 747,395 | 0.1,0 | 1520 | 484 | 37907 | 561,731 | 0.1,0.1 | 807 | 58 | 94365 | 544,211 | 0,0 |
| 3.0.7 (new) | 9514 | 30273 | 30263 | 105, 113 | 1, 1 | 9027 | 23850 | 1521511 | 186, 552 | 1, 0.7 | 7285 | 8357 | 1891569 | 662, 668 | 0.8, 0.8 | 5602 | 2619 | 5249308 | 680, 713 | 0.5, 0.5 |
| 3.1.3 | 8922 | 28278 | 34609 | 105,107 | 0.8,0.9 | 4542 | 9605 | 2676743 | 601,528 | 0.4,0.4 | 3764 | 4216 | 1896390 | 620,636 | 0.4,0.4 | 3336 | 1499 | 4892724 | 678,698 | 0.3,0.3 |
| 3.1.6 | 9423 | 30652 | 37562 | 105 | 0.9 | 4522 | 9517 | 2414417 | 631 | 0.4 | 4002 | 4945 | 1897270 | 693 | 0.5 | 3224 | 1680 | 4898631 | 634 | 0.3 |
| Version/Test | Single Pipeline (ci-logs) | | | | | Multi Pipeline (ci-logs-m) | | | | | Single Pipeline Backpressure (ci-logs-b) | | | | | Multi Pipeline Backpressure (ci-logs-mb) | | | | |
|---------------------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:---------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|:----------------------------------------:|:----------------------------------------:|:-------------------------------:|:--------------------:|:-------------:|
| | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage | Input Bytes Processing Rate/sec (KByte) | Output Bytes Processing Rate/sec (KByte) | Filesystem Buffer Usage (KByte) | Pod Memory Usage(MB) | Pod CPU Usage |
| 2.2.1 | 5165 | 8541 | 68518 | 172, 190 | 1, 1 | 2009 | 2195 | 102932 | 332, 320 | 0.9, 0.9 | 5914 | 1498 | 79247 | 184, 176 | 0.9, 1 | 1979 | 489 | 83442 | 310, 322 | 0.9, 0.9 |
| 2.2.2 | 5159 | 7811 | 75545 | 171, 170 | 1, 1 | 1910 | 2516 | 103780 | 324, 324 | 0.9, 0.9 | 5857 | 1513 | 72494 | 189, 200 | 1, 1 | 1860 | 421 | 90852 | 314, 322 | 0.9, 0.9 |
| 2.2.2 (new setup) | 5445 | 9668 | 68453 | 248, 981 | 1, 1 | 6201 | 2747 | 89291 | 544, 720 | 1, 1 | 6009 | 1723 | 58982 | 650, 682 | 1, 1 | 6111 | 385 | 108909 | 686, 931 | 0.9, 0.9 |
| 3.0.3 | 9483 | 22042 | 53251 | 366, 681 | 1, 1 | 10737 | 8785 | 115232 | 953, 568 | 0.9, 0.9 | 10425 | 4610 | 80614 | 856, 704 | 0.9, 0.9 | 10955 | 1724 | 87530 | 503, 594 | 0.9, 0.9 |
| 3.0.4 | 4341 | 8296 | 35628 | 971, 726 | 0.1, 0.1 | 1201 | 544 | 103624 | 652, 815 | 0, 0 | 932 | 297 | 37663 | 615, 726 | 0.1, 0.1 | 1477 | 171 | 108885 | 530, 566 | 0, 0.1 |
| 3.0.7 (old metrics) | 4241 | 7782 | 47586 | 815,1021 | 0.7, 0.1 | 3809 | 1968 | 107529 | 837,965 | 0.4, 0 | 3472 | 1093 | 33818 | 792, 597 | 0, 0.1 | 2180 | 177 | 87052 | 708, 631 | 0, 0.1 |
| 3.0.7 (new metrics) | 4036 | 7173 | 31689 | 825,852 | 0.1, 0.1 | 2481 | 1852 | 104689 | 747,395 | 0.1, 0 | 1520 | 484 | 37907 | 561, 731 | 0.1, 0.1 | 807 | 58 | 94365 | 544, 211 | 0,0 |
| 3.0.7 (new) | 9514 | 30273 | 30263 | 105, 113 | 1, 1 | 9027 | 23850 | 1521511 | 186, 552 | 1, 0.7 | 7285 | 8357 | 1891569 | 662, 668 | 0.8, 0.8 | 5602 | 2619 | 5249308 | 680, 713 | 0.5, 0.5 |
| 3.1.3 | 8922 | 28278 | 34609 | 105,107 | 0.8, 0.9 | 4542 | 9605 | 2676743 | 601,528 | 0.4, 0.4 | 3764 | 4216 | 1896390 | 620, 636 | 0.4, 0.4 | 3336 | 1499 | 4892724 | 678, 698 | 0.3, 0.3 |
| 3.1.6 | 9423 | 30652 | 37562 | 105 | 0.9 | 4522 | 9517 | 2414417 | 631 | 0.4 | 4002 | 4945 | 1897270 | 693 | 0.5 | 3224 | 1680 | 4898631 | 634 | 0.3 |
| 3.1.3 (after optimization) | 9149 | 29694 | 37883 | 93, 98 | 1, 0.8 | 4344 | 8354 | 2893257 | 601, 587 | 0.4, 0.4 | 3715 | 4000 | 1947742 | 739, 692 | 0.4, 0.4 | 3200 | 1314 | 4950684 | 662, 693 | 0.3, 0.3 |
| 3.1.6 (after optimization) | 10183 | 33117 | 30358 | 98, 93 | 1, 1 | 4349 | 8935 | 2779129 | 539, 660 | 0.4, 0.4 | 5471 | 5840 | 1889836 | 646, 641 | 0.5, 0.6 | 3224 | 1621 | 4882502 | 686, 649 | 0.3, 0.3 |
</div>

> **NOTE:** The test results can vary up to 1000KByte/sec in the input, and up to 3000KByte/sec in the output processing rate with single pipeline tests. The multi pipeline test results can be lower than the single pipeline test results, because the test setup is shared between multiple pipelines and this may create pressure on the shared resources.
>
## Self Monitor

### Assumptions
Expand Down
7 changes: 4 additions & 3 deletions hack/load-tests/log-fluentbit-test-setup.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,15 @@ metadata:
namespace: log-load-test
data:
fluent.conf: |-
<system>
workers 2
</system>
<source>
@type http
port 9880
bind 0.0.0.0
body_size_limit 50m
add_http_headers true
<parse>
@type json
@type none
</parse>
</source>
<match **>
Expand Down
Loading

0 comments on commit 2f2dd9e

Please sign in to comment.