Skip to content

Commit

Permalink
This change allows the HTTP clients to consume and send gzip compress…
Browse files Browse the repository at this point in the history
…ed response and request body.

It is available for the following REST API endpoints:
- GET & POST HTTP methods on /v0/data & /v1/data endpoints
- POST HTTP method on /v1/compile endpoint

HTTP clients can optionally:
- send 'Accept-Encoding: gzip' header and expect a gzip compressed body and a Content-Encoding: gzip response header. The server will send the content encoded as gzip only after a threshold defined by server.encoding.gzip.min_length (default value is 1024). If the size is below the threshold, the body is not compressed
- send 'Content-Encoding: gzip' header and a gzip compressed body and expect the server to correctly interpret the request

Fixes #5310

Signed-off-by: aarnautu <[email protected]>
  • Loading branch information
aarnautu authored and anderseknert committed Mar 9, 2023
1 parent dc37446 commit 727b33e
Show file tree
Hide file tree
Showing 12 changed files with 1,296 additions and 21 deletions.
5 changes: 4 additions & 1 deletion config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,10 @@ type Config struct {
NDBuiltinCache bool `json:"nd_builtin_cache,omitempty"`
PersistenceDirectory *string `json:"persistence_directory,omitempty"`
DistributedTracing json.RawMessage `json:"distributed_tracing,omitempty"`
Storage *struct {
Server *struct {
Encoding json.RawMessage `json:"encoding,omitempty"`
} `json:"server,omitempty"`
Storage *struct {
Disk json.RawMessage `json:"disk,omitempty"`
} `json:"storage,omitempty"`
}
Expand Down
16 changes: 16 additions & 0 deletions config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,14 @@ func TestActiveConfig(t *testing.T) {
"plugins": {
"some-plugin": {}
},
"server": {
"encoding": {
"gzip": {
"min_length": 1024,
"compression_level": 1
}
}
},
"discovery": {"name": "config"}`

serviceObj := `"services": {
Expand Down Expand Up @@ -249,6 +257,14 @@ func TestActiveConfig(t *testing.T) {
"plugins": {
"some-plugin": {}
},
"server": {
"encoding": {
"gzip": {
"min_length": 1024,
"compression_level": 1
}
}
},
"default_authorization_decision": "/system/authz/allow",
"default_decision": "/system/main",
"discovery": {"name": "config"}`, version.Version)
Expand Down
16 changes: 16 additions & 0 deletions docs/content/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,12 @@ distributed_tracing:
service_name: opa
sample_percentage: 50
encryption: "off"

server:
encoding:
gzip:
min_length: 1024,
compression_level: 9
```
#### Environment Variable Substitution
Expand Down Expand Up @@ -925,3 +931,13 @@ with data put into the configured `directory`.
| `storage.disk.badger` | `string` | No (default: empty) | "Superflags" passed to Badger allowing to modify advanced options. |

See [the docs on disk storage](../misc-disk/) for details about the settings.

### Server

The `server` configuration sets the gzip compression settings for `/v0/data`, `/v1/data` and `/v1/compile` HTTP `POST` endpoints
The gzip compression settings are used when the client sends `Accept-Encoding: gzip`

| Field | Type | Required | Description |
|------------------------------------------|-------|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `server.encoding.gzip.min_length` | `int` | No, (default: 1024) | Specifies the minimum length of the response to compress |
| `server.encoding.gzip.compression_level` | `int` | No, (default: 9) | Specifies the compression level. Accepted values: a value of either 0 (no compression), 1 (best speed, lowest compression) or 9 (slowest, best compression). See https://pkg.go.dev/compress/flate#pkg-constants |
13 changes: 13 additions & 0 deletions docs/content/rest-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -732,6 +732,10 @@ The path separator is used to access values inside object and array documents. I
- **instrument** - Instrument query evaluation and return a superset of performance metrics in addition to result. See [Performance Metrics](#performance-metrics) for more detail.
- **strict-builtin-errors** - Treat built-in function call errors as fatal and return an error immediately.

#### Request Headers

- **Accept-Encoding: gzip**: Indicates the server should respond with a gzip encoded body. The server will send the compressed response only if its length is above `server.encoding.gzip.min_length` value. See the configuration section

#### Status Codes

- **200** - no error
Expand Down Expand Up @@ -820,6 +824,8 @@ The request body contains an object that specifies a value for [The input Docume
#### Request Headers

- **Content-Type: application/x-yaml**: Indicates the request body is a YAML encoded object.
- **Content-Encoding: gzip**: Indicates the request body is a gzip encoded object.
- **Accept-Encoding: gzip**: Indicates the server should respond with a gzip encoded body. The server will send the compressed response only if its length is above `server.encoding.gzip.min_length` value. See the configuration section

#### Query Parameters

Expand Down Expand Up @@ -939,6 +945,8 @@ array documents.
#### Request Headers

- **Content-Type: application/x-yaml**: Indicates the request body is a YAML encoded object.
- **Content-Encoding: gzip**: Indicates the request body is a gzip encoded object.
- **Accept-Encoding: gzip**: Indicates the server should respond with a gzip encoded body. The server will send the compressed response only if its length is above `server.encoding.gzip.min_length` value. See the configuration section

#### Query Parameters

Expand Down Expand Up @@ -1290,6 +1298,11 @@ Compile API requests contain the following fields:
| `options` | `object[string, any]` | No | Additional options to use during partial evaluation. Only `disableInlining` option is supported. (default: undefined). |
| `unknowns` | `array[string]` | No | The terms to treat as unknown during partial evaluation (default: `["input"]`]). |

### Request Headers

- **Content-Encoding: gzip**: Indicates the request body is a gzip encoded object.
- **Accept-Encoding: gzip**: Indicates the server should respond with a gzip encoded body. The server will send the compressed response only if its length is above `server.encoding.gzip.min_length` value

#### Query Parameters

- **pretty** - If parameter is `true`, response will formatted for humans.
Expand Down
91 changes: 91 additions & 0 deletions plugins/server/encoding/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package encoding

import (
"compress/gzip"
"fmt"

"github.com/open-policy-agent/opa/util"
)

var defaultGzipMinLength = 1024
var defaultGzipCompressionLevel = gzip.BestCompression

// Config represents the configuration for the Server.Encoding settings
type Config struct {
Gzip *Gzip `json:"gzip,omitempty"`
}

// Gzip represents the configuration for the Server.Encoding.Gzip settings
type Gzip struct {
MinLength *int `json:"min_length,omitempty"` // the minimum length of a response that will be gzipped
CompressionLevel *int `json:"compression_level,omitempty"` // the compression level for gzip
}

// ConfigBuilder assists in the construction of the plugin configuration.
type ConfigBuilder struct {
raw []byte
}

// NewConfigBuilder returns a new ConfigBuilder to build and parse the server config
func NewConfigBuilder() *ConfigBuilder {
return &ConfigBuilder{}
}

// WithBytes sets the raw server config
func (b *ConfigBuilder) WithBytes(config []byte) *ConfigBuilder {
b.raw = config
return b
}

// Parse returns a valid Config object with defaults injected.
func (b *ConfigBuilder) Parse() (*Config, error) {
if b.raw == nil {
defaultConfig := &Config{
Gzip: &Gzip{
MinLength: &defaultGzipMinLength,
CompressionLevel: &defaultGzipCompressionLevel,
},
}
return defaultConfig, nil
}

var result Config

if err := util.Unmarshal(b.raw, &result); err != nil {
return nil, err
}

return &result, result.validateAndInjectDefaults()
}

func (c *Config) validateAndInjectDefaults() error {
if c.Gzip == nil {
c.Gzip = &Gzip{
MinLength: &defaultGzipMinLength,
CompressionLevel: &defaultGzipCompressionLevel,
}
}
if c.Gzip.MinLength == nil {
c.Gzip.MinLength = &defaultGzipMinLength
}

if c.Gzip.CompressionLevel == nil {
c.Gzip.CompressionLevel = &defaultGzipCompressionLevel
}

if *c.Gzip.MinLength <= 0 {
return fmt.Errorf("invalid value for server.encoding.gzip.min_length field, should be a positive number")
}

acceptedCompressionLevels := map[int]bool{
gzip.NoCompression: true,
gzip.BestSpeed: true,
gzip.BestCompression: true,
}
_, compressionLevelAccepted := acceptedCompressionLevels[*c.Gzip.CompressionLevel]
if !compressionLevelAccepted {
return fmt.Errorf("invalid value for server.encoding.gzip.compression_level field, accepted values are 0, 1 or 9")
}

return nil
}
105 changes: 105 additions & 0 deletions plugins/server/encoding/config_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
package encoding

import (
"fmt"
"testing"
)

func TestConfigValidation(t *testing.T) {
tests := []struct {
input string
wantErr bool
}{
{
input: `{}`,
wantErr: false,
},
{
input: `{"gzip": {"min_length": "not-a-number"}}`,
wantErr: true,
},
{
input: `{"gzip": {min_length": 42}}`,
wantErr: false,
},
{
input: `{"gzip":{"min_length": "42"}}`,
wantErr: true,
},
{
input: `{"gzip":{"min_length": 0}}`,
wantErr: true,
},
{
input: `{"gzip":{"min_length": -10}}`,
wantErr: true,
},
{
input: `{"gzip":{"random_key": 0}}`,
wantErr: false,
},
{
input: `{"gzip": {"min_length": -10, "compression_level": 13}}`,
wantErr: true,
},
{
input: `{"gzip":{"compression_level": "not-an-number"}}`,
wantErr: true,
},
{
input: `{"gzip":{"compression_level": 1}}`,
wantErr: false,
},
{
input: `{"gzip":{"compression_level": 13}}`,
wantErr: true,
},
{
input: `{"gzip":{"min_length": 42, "compression_level": 9}}`,
wantErr: false,
},
}

for i, test := range tests {
t.Run(fmt.Sprintf("TestConfigValidation_case_%d", i), func(t *testing.T) {
_, err := NewConfigBuilder().WithBytes([]byte(test.input)).Parse()
if err != nil && !test.wantErr {
t.Fail()
}
if err == nil && test.wantErr {
t.Fail()
}
})
}
}

func TestConfigValue(t *testing.T) {
tests := []struct {
input string
minLengthExpectedValue int
compressionLevelExpectedValue int
}{
{
input: `{}`,
minLengthExpectedValue: 1024,
compressionLevelExpectedValue: 9,
},
{
input: `{"gzip":{"min_length": 42, "compression_level": 1}}`,
minLengthExpectedValue: 42,
compressionLevelExpectedValue: 1,
},
}

for i, test := range tests {
t.Run(fmt.Sprintf("TestConfigValue_case_%d", i), func(t *testing.T) {
config, err := NewConfigBuilder().WithBytes([]byte(test.input)).Parse()
if err != nil {
t.Fail()
}
if *config.Gzip.MinLength != test.minLengthExpectedValue || *config.Gzip.CompressionLevel != test.compressionLevelExpectedValue {
t.Fail()
}
})
}
}
Loading

0 comments on commit 727b33e

Please sign in to comment.