Compress OPA response & accept compressed input #5696

AdrianArnautu · 2023-02-24T14:57:43Z

This change allows the HTTP clients to consume and send gzip compressed response and request body.
It is available for the following REST API endpoints:

GET & POST HTTP methods on /v0/data & /v1/data endpoints
POST HTTP method on /v1/compile endpoint.

HTTP clients can optionally:

send Accept-Encoding: gzip header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.gzip_min_length (defaulted on 1024 bytes). If the size is below the threshold, the body is not compressed
send Content-Encoding: gzip header and a gzip compressed body and expect the server to correctly interpret the request

Reducing the network load in our setup is the reason for this change.

partial evaluation is almost always returns a large response
/v1/data endpoint might need to consume a large input because otherwise storing it in the bundle puts pressure on memory

High level implementation details

new internal HTTP handler captures the request and watches for Accept-Encoding: gzip request header and responds with a gzip body. The implementation is independent of gorilla/mux HTTP router
the handler will decide if the response is compressed or not if the body is larger than server.gzip_min_length configuration
data & compile endpoints watches for Content-Encoding: gzip request header and is decompressing the input before consuming it
both the input and the response it is logged decompressed to the console regardless of the client's compress headers
the compression threshold can be changed from the configuration file, the key is server.gzip_min_length
there's also server.gzip_compression_level defaulted to value 9 (best compression). See the gzip library

Fixes #5310
Signed-off-by: Adrian Arnautu [email protected]

netlify · 2023-02-24T14:58:52Z

✅ Deploy Preview for openpolicyagent ready!

Name	Link
🔨 Latest commit	`840946e`
🔍 Latest deploy log	https://app.netlify.com/sites/openpolicyagent/deploys/640860e831323700081d9af6
😎 Deploy Preview	https://deploy-preview-5696--openpolicyagent.netlify.app/docs/edge/management-status
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

anderseknert · 2023-02-24T19:19:29Z

This is great! We'll take a closer look next week. I wonder if this is something to consider?

AdrianArnautu · 2023-02-24T19:37:10Z

@anderseknert thank you
A threshold payload size to compress the body seems sensible, I'll do an addendum.

anderseknert · 2023-02-27T12:19:20Z

Setting status to waiting-for-response as it is the closest thing we have :) Will be ready for review once the payload threshold has been taken into consideration.

AdrianArnautu · 2023-03-01T10:12:42Z

Setting status to waiting-for-response as it is the closest thing we have :) Will be ready for review once the payload threshold has been taken into consideration.

I'm planning to push the change this week, worst case, next one.

anderseknert · 2023-03-01T13:08:19Z

Thanks for the update, @AdrianArnautu 👍

AdrianArnautu · 2023-03-05T21:53:00Z

@anderseknert I've pushed the latest changes, this PR is ready for the review.
There's a new config key for changing the threshold value as it is fit based on one's setup. Defaults to 1024.

Let me know if changes are needed, either naming/structure, features or something I've grossly overlooked.

anderseknert

This is a great contribution! Code looks good to me, and a really extensive set of tests 👏

Only a few questions, really. Perhaps @johanfylling or @ashutosh-narkar could take a look as well, but this is a great improvement 👍

docs/content/configuration.md

plugins/server/config.go

AdrianArnautu · 2023-03-06T11:58:49Z

I've done the changes and rebased the branch.

anderseknert

LGTM!

srenatus

Nice work! Some minor questions from my side 👇

runtime/logging.go

runtime/logging_test.go

server/handlers/compress.go

AdrianArnautu · 2023-03-06T15:53:50Z

Ready for review @anderseknert & @srenatus

runtime/logging.go

plugins/server/config.go

docs/content/configuration.md

server/server.go

srenatus

Thanks, that's an impressive first contribution.

srenatus · 2023-03-08T14:34:10Z

runtime/logging.go

@@ -73,8 +74,26 @@ func (h *LoggingHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 				bs, r.Body, err = readBody(r.Body)
 			}
 			if err == nil {
-				fields["req_body"] = string(bs)
-			} else {
+				if gzipReceived(r.Header) {


It's somewhat unfortunate that we have to decompress what we've ourselves compressed before just for logging it. I wonder if there's a different way of chaining handlers possible that would let us avoid that...

ashutosh-narkar · 2023-03-08T22:07:54Z

@AdrianArnautu thanks for this contribution. Can you please update the commit message explaining why the changes were made? Here are some guidelines. We can then merge this.

…ed response and request body. It is available for the following REST API endpoints: - GET & POST HTTP methods on /v0/data & /v1/data endpoints - POST HTTP method on /v1/compile endpoint HTTP clients can optionally: - send 'Accept-Encoding: gzip' header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.encoding.gzip.min_length (default value is 1024). If the size is below the threshold, the body is not compressed - send 'Content-Encoding: gzip' header and a gzip compressed body and expect the server to correctly interpret the request Fixes open-policy-agent#5310 Signed-off-by: aarnautu <[email protected]>

AdrianArnautu · 2023-03-09T08:37:36Z

@anderseknert , @srenatus & @ashutosh-narkar - thank you.

anderseknert · 2023-03-09T08:38:53Z

Thank YOU @AdrianArnautu 👏

AdrianArnautu changed the title ~~#5310 Compress OPA response & accept compressed input~~ Compress OPA response & accept compressed input Feb 24, 2023

AdrianArnautu force-pushed the compression-handler-data-and-compile branch from eb07d30 to da62d5c Compare February 24, 2023 15:19

anderseknert added the waiting-for-response label Feb 27, 2023

AdrianArnautu force-pushed the compression-handler-data-and-compile branch 2 times, most recently from 543ae95 to 285fce5 Compare March 5, 2023 21:17

anderseknert reviewed Mar 6, 2023

View reviewed changes

docs/content/configuration.md Outdated Show resolved Hide resolved

plugins/server/config.go Outdated Show resolved Hide resolved

AdrianArnautu force-pushed the compression-handler-data-and-compile branch from 285fce5 to e1a6c76 Compare March 6, 2023 11:49

AdrianArnautu requested a review from anderseknert March 6, 2023 11:58

anderseknert previously approved these changes Mar 6, 2023

View reviewed changes

anderseknert removed the waiting-for-response label Mar 6, 2023

srenatus reviewed Mar 6, 2023

View reviewed changes

runtime/logging.go Outdated Show resolved Hide resolved

runtime/logging_test.go Outdated Show resolved Hide resolved

server/handlers/compress.go Show resolved Hide resolved

AdrianArnautu dismissed anderseknert’s stale review via 2bf6b5d March 6, 2023 15:31

AdrianArnautu force-pushed the compression-handler-data-and-compile branch 2 times, most recently from 2bf6b5d to 74394a8 Compare March 6, 2023 15:40

AdrianArnautu requested review from srenatus and anderseknert and removed request for srenatus and anderseknert March 6, 2023 15:52

anderseknert previously approved these changes Mar 6, 2023

View reviewed changes

srenatus reviewed Mar 6, 2023

View reviewed changes

runtime/logging.go Outdated Show resolved Hide resolved

ashutosh-narkar reviewed Mar 6, 2023

View reviewed changes

plugins/server/config.go Outdated Show resolved Hide resolved

docs/content/configuration.md Outdated Show resolved Hide resolved

AdrianArnautu dismissed anderseknert’s stale review via 9597f39 March 7, 2023 07:44

AdrianArnautu force-pushed the compression-handler-data-and-compile branch from 74394a8 to 9597f39 Compare March 7, 2023 07:44

AdrianArnautu requested review from srenatus and anderseknert and removed request for srenatus and anderseknert March 7, 2023 08:09

srenatus reviewed Mar 7, 2023

View reviewed changes

server/server.go Outdated Show resolved Hide resolved

AdrianArnautu force-pushed the compression-handler-data-and-compile branch from 9597f39 to f9839d6 Compare March 7, 2023 12:05

AdrianArnautu requested a review from srenatus March 7, 2023 12:26

AdrianArnautu force-pushed the compression-handler-data-and-compile branch 2 times, most recently from efbcc7e to 840946e Compare March 8, 2023 10:18

srenatus previously approved these changes Mar 8, 2023

View reviewed changes

AdrianArnautu dismissed srenatus’s stale review via 911e794 March 9, 2023 08:10

AdrianArnautu force-pushed the compression-handler-data-and-compile branch from 840946e to 911e794 Compare March 9, 2023 08:10

anderseknert force-pushed the compression-handler-data-and-compile branch from 911e794 to 727b33e Compare March 9, 2023 08:23

anderseknert approved these changes Mar 9, 2023

View reviewed changes

anderseknert merged commit 9e97f98 into open-policy-agent:main Mar 9, 2023

philipaconrad mentioned this pull request Jun 26, 2024

server/authorizer: Fix gzip payload handling. #6825

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compress OPA response & accept compressed input #5696

Compress OPA response & accept compressed input #5696

AdrianArnautu commented Feb 24, 2023 •

edited

Loading

netlify bot commented Feb 24, 2023 •

edited

Loading

anderseknert commented Feb 24, 2023

AdrianArnautu commented Feb 24, 2023

anderseknert commented Feb 27, 2023

AdrianArnautu commented Mar 1, 2023

anderseknert commented Mar 1, 2023

AdrianArnautu commented Mar 5, 2023

anderseknert left a comment

AdrianArnautu commented Mar 6, 2023

anderseknert left a comment

srenatus left a comment

AdrianArnautu commented Mar 6, 2023

srenatus left a comment

srenatus Mar 8, 2023

ashutosh-narkar commented Mar 8, 2023

AdrianArnautu commented Mar 9, 2023

anderseknert commented Mar 9, 2023

Compress OPA response & accept compressed input #5696

Compress OPA response & accept compressed input #5696

Conversation

AdrianArnautu commented Feb 24, 2023 • edited Loading

netlify bot commented Feb 24, 2023 • edited Loading

✅ Deploy Preview for openpolicyagent ready!

anderseknert commented Feb 24, 2023

AdrianArnautu commented Feb 24, 2023

anderseknert commented Feb 27, 2023

AdrianArnautu commented Mar 1, 2023

anderseknert commented Mar 1, 2023

AdrianArnautu commented Mar 5, 2023

anderseknert left a comment

Choose a reason for hiding this comment

AdrianArnautu commented Mar 6, 2023

anderseknert left a comment

Choose a reason for hiding this comment

srenatus left a comment

Choose a reason for hiding this comment

AdrianArnautu commented Mar 6, 2023

srenatus left a comment

Choose a reason for hiding this comment

srenatus Mar 8, 2023

Choose a reason for hiding this comment

ashutosh-narkar commented Mar 8, 2023

AdrianArnautu commented Mar 9, 2023

anderseknert commented Mar 9, 2023

AdrianArnautu commented Feb 24, 2023 •

edited

Loading

netlify bot commented Feb 24, 2023 •

edited

Loading