Support brotli compression on the server side #142334

rockdaboot · 2022-09-30T11:54:41Z

Background
While investigating a high latency in the Kibana server for the response serialization, I realized that Kibana uses Content-Encoding: gzip even if the browser supports br (brotli compression).
Since Brotli is known to often use less CPU time at even a better compression ratio then Gzip, I gave it a spin.
FYI, our data is JSON encoded.

Results from console testing
Compression of a 41492998 bytes JSON file. The size of data is quite common for generating flamegraphs.

gzip
real    0m0.786s
compressed size: 6657257 bytes

This result for gzip is roughly the same that I saw in the browser's developer console.

brotli -3
real    0m0.235s
compressed size: 4196636 bytes

Compression level 3 seemed to be a good compromise between CPU usage and compression ratio (I also tested other level).

Browser Support of brotli
All modern browsers support br Content-Encoding. Older browsers will only offer gzip and deflate.
Details on this: https://caniuse.com/?search=brotli

Results from the MVP Universal Profiling cluster
The brotli compression shaves off ~40% from the "white gap" on the right side (serialization + compression).
The compressed network payload goes down from 6.7MB to 3.9MB, ~40% less data to transfer (seen in the browser dev console under "network").

gzip

brotli

TBD

We need CPU benchmarks, I only had time for some manual tests
We need benchmarks about memory consumption of brotli

Conclusion
The support of the br Content-Encoding has the potential of strongly reducing CPU usage and moderately lowering network bandwidth for Kibana in cases were larger amounts of compressible data is transferred. This may have positive effects cloud costs.

The risk of introducing downsides seem to be relatively low.

pgayvallet · 2022-09-30T11:58:54Z

packages/core/http/core-http-server-internal/src/http_server.ts

@@ -8,6 +8,7 @@

 import { Server, Request } from '@hapi/hapi';
 import HapiStaticFiles from '@hapi/inert';
+const Brok = require('brok');


import Brok from 'brok';

The linter doesn't like it

packages/core/http/core-http-server-internal/src/http_server.ts:55:18 - error TS2307: Cannot find module 'brok' or its corresponding type declarations. 55 import Brok from 'brok';

Not sure how to fix it.

There are two things here.

each dependency of a package must be declared in it's bazel build file.

apparently this package doesn't provide the proper typings.

I'll try to push to your PR to fix that today if I can find the time

done in bf59350

…-fix'

dgieselaar · 2022-09-30T15:47:41Z

@pgayvallet do you happen to know if cloud's infrastucture (ie, everything that sits between the browser and the actual Kibana server) supports brotli?

…-ref HEAD~1..HEAD --fix'

pgayvallet · 2022-10-03T07:25:16Z

@dgieselaar no, I don't (even if given this is a content compression and not touching the payload I would assume it should work).

dgieselaar · 2022-10-03T07:27:37Z

@pgayvallet you're probably right, I'm confusing it with environments where compression happens at the edge.

…li-compression

pgayvallet

LGTM

elasticmachine · 2022-10-03T13:06:14Z

Pinging @elastic/kibana-core (Team:Core)

rudolf · 2022-10-04T08:11:38Z

Code looks good but let's wait to do a performance comparison before merging.

I've kicked-off two builds but haven't had a chance to compare the results:
kibana-load-test with Accept-encoding: gzip deflate Kibana on rockdaboot/use-brotli-compression
https://kibana-ci.elastic.co/job/elastic+kibana+load-testing/2153/
kibana-load-test with Accept-encoding: br gzip deflate Kibana on rockdaboot/use-brotli-compression
https://kibana-ci.elastic.co/job/elastic+kibana+load-testing/2154/
(for completeness sake we should probably have a test with: kibana-load-test with Accept-encoding: br gzip deflate Kibana on main but not sure if kibana-load-test tests loading static assets)

pgayvallet · 2022-10-04T09:25:57Z

@rudolf looking at the two reports, it looks like both builds have very similar results. Do you share my observations?

rockdaboot · 2022-10-04T10:10:49Z

@rudolf looking at the two reports, it looks like both builds have very similar results. Do you share my observations?

Is it possible that hapi is very picky on how the Accept-Encoding is formed ?
Not really sure, but looking at hapijs/hapi@f51c7f1 it seems, we should use gzip, deflate, br ?

rudolf · 2022-10-04T10:34:07Z

Is it possible that hapi is very picky on how the Accept-Encoding is formed ? Not really sure, but looking at hapijs/hapi@f51c7f1 it seems, we should use gzip, deflate, br ?

Good find, this seems rather brittle!

rudolf · 2022-10-04T10:40:32Z

Updated elastic/kibana-load-testing#323

and ran a new performance build based on that:
https://kibana-ci.elastic.co/job/elastic+kibana+load-testing/2159/

rudolf · 2022-10-04T12:42:12Z

Well, 50th percentile response time seems to be slightly slower while 99th percentile is slightly faster 🤷

journey	50th	99th
lensjourney	588	2060
lensjourney-brotli	708	2046
discoverjourney	1244	6833
discoverjourney-brotli	1281	6586
canvas-journey	915	6150
canvas-journey-brotli	1096	5760

rockdaboot · 2022-10-04T13:47:58Z

Well, 50th percentile response time seems to be slightly slower while 99th percentile is slightly faster shrug

This is a slightly unexpected result 🤔
Is there any details on what exactly we are measuring here ? Like number of requests/responses, payload of responses (compressed/uncompressed). And what were the response's Content-Encoding ?
Any numbers on how much CPU does response serialization take. Or what exactly the share of response serialization vs total response time is (in average) ?

dgieselaar · 2022-10-05T11:48:00Z

@rudolf do those numbers serve as just a sanity check, or are the results significant? I saw some discussion around the numbers being unstable? can we move forward with this or do we need to run additional benchmarks?

rudolf · 2022-10-05T14:56:26Z

@dgieselaar I think we can just say that these numbers show it doesn't make a significant difference, positive or negative. That means we're fine to merge this into main/8.6

rudolf · 2022-10-05T14:58:38Z

@rockdaboot These are generic performance tests and you can check out the requests made for each of the scenarios here https://github.com/elastic/kibana-load-testing/tree/main/src/test/scala/org/kibanaLoadTest/scenario

rockdaboot · 2022-10-05T16:34:30Z

Thanks @rudolf !

Just one more question about the benchmarks. Are the client(s) running inside the same region or even inside the same network as Kibana ?

rockdaboot · 2022-10-05T16:36:46Z

@elasticmachine merge upstream

rudolf · 2022-10-06T15:04:49Z

Just one more question about the benchmarks. Are the client(s) running inside the same region or even inside the same network as Kibana ?

I think these are run on the same server using dedicated hardware @dmlemeshko do we have any docs about the infrastructure?

For completeness sake I worked with @dmlemeshko to pull cpu and memory usage from these bulids. We don't (yet) have a way to distinguish between builds based on the metrics, but we used timestamps to identify the brotli build. In the graph below the brotli build is in the center and the other two builds are main without brotli. So not a huge impact, and memory and cpu seems slightly lower with brotli.

https://kibana-stats.elastic.dev/app/dashboards#/view/7b3eb890-457d-11ed-9db6-5d970c9cc752

pgayvallet · 2022-10-10T08:19:22Z

@rudolf so, from our last weekly's discussions, for 8.6 we still want to add a config option (server.compression.brotli) on front of this feature, right?

dmlemeshko · 2022-10-12T11:14:34Z

I think these are run on the same server using dedicated hardware @dmlemeshko do we have any docs about the infrastructure?

We are using bare metal machine EX-62 provided by Hezner:

Intel® Core™ i9-9900K (8 cores)
128 GB DDR4 RAM
2 TB SSD
1 GBit/s-Port connection

Both ES/Kibana and Gatling runner hosted on this machine, which obviously might have some impact (ES and Gatling can battle for resources), but for now we stick to this setup.

If you are interested about ES/Kibana server arguments, you can find it in FTR config that we use for runs.

I will add it in kibana-load-testing repo docs, and make sure it is always up-to-date

pgayvallet · 2022-10-20T08:46:20Z

@rockdaboot thanks again for the initial PR, I'm taking over to add the config flag and polish the details :)

…li-compression

afharo

LGTM! I just added some minor comments for my own education

afharo · 2022-10-24T08:13:13Z

docs/setup/settings.asciidoc

@@ -386,6 +386,11 @@ Specifies an array of trusted hostnames, such as the {kib} host, or a reverse
 proxy sitting in front of it. This determines whether HTTP compression may be used for responses, based on the request `Referer` header.
 This setting may not be used when <<server-compression, `server.compression.enabled`>> is set to `false`. *Default: `none`*

+`server.compression.brotli.enabled`::


Should we add documentation about quality as well?

Woups, I forgot to add the review comment for that one. Actually no 😅 : I added this setting with the intent of keeping it internal (undocumented) for now, as it was mostly to allow us to eventually perform perf testing on cloud environment tweaking the quality value.

afharo · 2022-10-24T08:17:25Z

test/api_integration/apis/core/compression.ts

+
+    it(`supports brotli compression`, async () => {
+      await supertest
+        .get('/app/kibana')


Nit: should we explicitly test it with an API? Just in case /app and /api are served differently (as we do today with bundles)?

Yea, all these compression.ts tests are based on the postulate that application pages and API endpoints are served the same way. Seems like a good idea to duplicate the tests against an 'API' endpoint

…li-compression

kibana-ci · 2022-10-24T12:21:41Z

💚 Build Succeeded

Buildkite Build
Commit: 4eb6900

Metrics [docs]

✅ unchanged

History

💔 Build #82052 failed a310d06
💚 Build #81948 succeeded 40fd9b6
💔 Build #81813 failed 74ecbdb
💔 Build #81778 failed 40283bf
💚 Build #81701 succeeded 5c94383

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pgayvallet

Use brotli compression

5e1fd7b

rockdaboot added enhancement New value added to drive a business result v8.5.0 labels Sep 30, 2022

rockdaboot requested a review from a team as a code owner September 30, 2022 11:54

pgayvallet reviewed Sep 30, 2022

View reviewed changes

pgayvallet added release_note:skip Skip the PR/issue when compiling release notes v8.6.0 and removed v8.5.0 labels Sep 30, 2022

kibanamachine and others added 3 commits September 30, 2022 12:27

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

0f0abfc

…-fix'

Add integration test for brotli support

127ede2

Use import instead of require()

226400e

rockdaboot and others added 2 commits September 30, 2022 18:19

Suppress build error on importing brok

9972c19

[CI] Auto-commit changed files from 'node scripts/precommit_hook.js -…

3696ef6

…-ref HEAD~1..HEAD --fix'

pgayvallet added 2 commits October 3, 2022 09:36

add brok as explicit package dep

bf59350

Merge remote-tracking branch 'upstream/main' into rockdaboot/use-brot…

23c7ad0

…li-compression

pgayvallet approved these changes Oct 3, 2022

View reviewed changes

rudolf added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Oct 3, 2022

rudolf added the performance label Oct 3, 2022

rudolf approved these changes Oct 5, 2022

View reviewed changes

rockdaboot enabled auto-merge (squash) October 5, 2022 16:36

Merge branch 'main' into rockdaboot/use-brotli-compression

a7c30a8

rockdaboot disabled auto-merge October 5, 2022 16:37

rudolf mentioned this pull request Oct 13, 2022

Accept brotli encoding elastic/kibana-load-testing#323

Merged

2 tasks

pgayvallet self-assigned this Oct 20, 2022

pgayvallet added 5 commits October 20, 2022 10:49

Merge remote-tracking branch 'upstream/main' into rockdaboot/use-brot…

5c94383

…li-compression

add server.compression.brotli config settings

9b6009c

update documentation

40283bf

fix test utils

74ecbdb

fix more test configs

40fd9b6

pgayvallet requested a review from a team October 24, 2022 06:44

afharo approved these changes Oct 24, 2022

View reviewed changes

pgayvallet added 2 commits October 24, 2022 11:25

Merge remote-tracking branch 'upstream/main' into rockdaboot/use-brot…

e1e40d0

…li-compression

add tests for endpoints too

a310d06

rudolf approved these changes Oct 24, 2022

View reviewed changes

remove against endpoint for now

4eb6900

pgayvallet merged commit 8b0145c into elastic:main Oct 24, 2022

rockdaboot mentioned this pull request Apr 15, 2024

[Profiling] Can we reduce server-side processing time for flamegraphs? #176208

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support brotli compression on the server side #142334

Support brotli compression on the server side #142334

rockdaboot commented Sep 30, 2022

pgayvallet Sep 30, 2022

rockdaboot Sep 30, 2022

pgayvallet Oct 3, 2022

pgayvallet Oct 3, 2022

dgieselaar commented Sep 30, 2022

pgayvallet commented Oct 3, 2022

dgieselaar commented Oct 3, 2022

pgayvallet left a comment

elasticmachine commented Oct 3, 2022

rudolf commented Oct 4, 2022

pgayvallet commented Oct 4, 2022

rockdaboot commented Oct 4, 2022

rudolf commented Oct 4, 2022

rudolf commented Oct 4, 2022

rudolf commented Oct 4, 2022

rockdaboot commented Oct 4, 2022

dgieselaar commented Oct 5, 2022

rudolf commented Oct 5, 2022

rudolf commented Oct 5, 2022

rockdaboot commented Oct 5, 2022

rockdaboot commented Oct 5, 2022

rudolf commented Oct 6, 2022 •

edited

Loading

pgayvallet commented Oct 10, 2022

dmlemeshko commented Oct 12, 2022

pgayvallet commented Oct 20, 2022

afharo left a comment

afharo Oct 24, 2022

pgayvallet Oct 24, 2022

afharo Oct 24, 2022

pgayvallet Oct 24, 2022

kibana-ci commented Oct 24, 2022

Support brotli compression on the server side #142334

Support brotli compression on the server side #142334

Conversation

rockdaboot commented Sep 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgieselaar commented Sep 30, 2022

pgayvallet commented Oct 3, 2022

dgieselaar commented Oct 3, 2022

pgayvallet left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 3, 2022

rudolf commented Oct 4, 2022

pgayvallet commented Oct 4, 2022

rockdaboot commented Oct 4, 2022

rudolf commented Oct 4, 2022

rudolf commented Oct 4, 2022

rudolf commented Oct 4, 2022

rockdaboot commented Oct 4, 2022

dgieselaar commented Oct 5, 2022

rudolf commented Oct 5, 2022

rudolf commented Oct 5, 2022

rockdaboot commented Oct 5, 2022

rockdaboot commented Oct 5, 2022

rudolf commented Oct 6, 2022 • edited Loading

pgayvallet commented Oct 10, 2022

dmlemeshko commented Oct 12, 2022

pgayvallet commented Oct 20, 2022

afharo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Oct 24, 2022

💚 Build Succeeded

Metrics [docs]

History

rudolf commented Oct 6, 2022 •

edited

Loading