aws: support for Unsigned Payload or provided content sha256 in AWS s…

…igning (#6581) To support uses cases where OPA is used for signing s3 requests whose payload is not known upfront or payload is big enough (big file upload) to be sent over wire, this PR adds support for unsigned payloads. AWS signer has configurable option to use unsigned payload where the x-amz-content-sha256 is set to "UNSIGNED-PAYLOAD" and is included as part of signing process. This PR provides an option for unsigned payload if aws_config.disable_payload_signing is set to true. If payload signing is disabled, SignV4 method will not compute the content sha from the request body but instead use "UNSIGNED-PAYLOAD" string literal for x-amz-content-sha256 header during signature computation. References: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html Signed-off-by: Prasanth Jayachandran <[email protected]>
open-policy-agent · Mar 8, 2024 · 37c2da8 · 37c2da8
1 parent 40bdc61
commit 37c2da8
Show file tree

Hide file tree

Showing 7 changed files with 318 additions and 19 deletions.
diff --git a/docs/content/policy-reference.md b/docs/content/policy-reference.md
@@ -968,9 +968,10 @@ The table below shows examples of calling `http.send`:
 
 The AWS Request Signing builtin in OPA implements the header-based auth,
 single-chunk method described in the [AWS SigV4 docs](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html).
-It will always sign the payload when present, and will sign most user-provided
+It will default to signing the payload when present, configurable via `aws_config`, and will sign most user-provided
 headers for the request, to ensure their integrity.
 
+
 {{< info >}}
 Note that the `authorization`, `user-agent`, and `x-amzn-trace-id` headers,
 are commonly modified by proxy systems, and as such are ignored by OPA
@@ -986,17 +987,18 @@ The following fields will have effects on the output `Authorization` header sign
 | `method` | yes | `string` | HTTP method to specify in request. Used in the signature. |
 | `body` | no | `any` | HTTP message body. The JSON serialized version of this value will be used for the payload portion of the signature if present. |
 | `raw_body` | no | `string` | HTTP message body. This will be used for the payload portion of the signature if present. |
-| `headers` | no | `object` | HTTP headers to include in the request. These will be added to the list of headers to sign. |
+| `headers` | no | `object` | HTTP headers to include in the request. These will be added to the list of headers to sign.|
 
 The `aws_config` object parameter may contain the following fields:
 
-| Field | Required | Type | Description |
-| --- | --- | --- | --- |
-| `aws_access_key` | yes | `string` | AWS access key. |
-| `aws_secret_access_key` | yes | `string` | AWS secret access key. Used in generating the signing key for the request. |
-| `aws_service` | yes | `string` | AWS service the request will be valid for. (e.g. `"s3"`) |
-| `aws_region` | yes | `string` | AWS region for the request. (e.g. `"us-east-1"`) |
-| `aws_session_token` | no | `string` | AWS security token. Used for the `x-amz-security-token` request header. |
+| Field | Required | Type | Description                                                                                                                                                                                                                    |
+| --- | --- | --- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `aws_access_key` | yes | `string` | AWS access key.                                                                                                                                                                                                                |
+| `aws_secret_access_key` | yes | `string` | AWS secret access key. Used in generating the signing key for the request.                                                                                                                                                     |
+| `aws_service` | yes | `string` | AWS service the request will be valid for. (e.g. `"s3"`)                                                                                                                                                                       |
+| `aws_region` | yes | `string` | AWS region for the request. (e.g. `"us-east-1"`)                                                                                                                                                                               |
+| `aws_session_token` | no | `string` | AWS security token. Used for the `x-amz-security-token` request header.                                                                                                                                                        |
+| `disable_payload_signing` | no | `boolean` | When `true` an `UNSIGNED-PAYLOAD` value will be used for calculating the `x-amz-content-sha256` header during signing, and will be returned in the response. Applicable only for `s3` and `glacier` service. Default: `false`. |
 
 #### AWS Request Signing Examples
 
@@ -1024,6 +1026,26 @@ example_verify_resource {
 }
 ```
 
+##### Unsigned Payload Request Signing Example
+The [AWS S3 request signing API](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html)
+supports unsigned payload signing option. This example below shows s3 request signing with payload signing disabled. 
+
+```live:providers/aws/sign_req_unsigned:module
+req := {"method": "get", "url": "https://examplebucket.s3.amazonaws.com/data"}
+aws_config := {
+    "aws_access_key": "MYAWSACCESSKEYGOESHERE",
+    "aws_secret_access_key": "MYAWSSECRETACCESSKEYGOESHERE",
+    "aws_service": "s3",
+    "aws_region": "us-east-1",
+    "disable_payload_signing": true,
+}
+
+example_verify_resource {
+    resp := http.send(providers.aws.sign_req(req, aws_config, time.now_ns()))
+    # process response from AWS ...
+}
+```
+
 ##### Pre-Signed Request Example
 The [AWS S3 request signing API](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html)
 supports pre-signing requests, so that they will only be valid at a future date.

diff --git a/internal/providers/aws/signing_v4.go b/internal/providers/aws/signing_v4.go
@@ -17,6 +17,8 @@ import (
 	"strings"
 	"time"
 
+	v4 "github.com/open-policy-agent/opa/internal/providers/aws/v4"
+
 	"github.com/open-policy-agent/opa/ast"
 )
 
@@ -104,7 +106,7 @@ func SignRequest(req *http.Request, service string, creds Credentials, theTime t
 		signedHeaders := SignV4a(req.Header, req.Method, req.URL, body, service, creds, now)
 		req.Header = signedHeaders
 	} else {
-		authHeader, awsHeaders := SignV4(req.Header, req.Method, req.URL, body, service, creds, now)
+		authHeader, awsHeaders := SignV4(req.Header, req.Method, req.URL, body, service, creds, now, false)
 		req.Header.Set("Authorization", authHeader)
 		for k, v := range awsHeaders {
 			req.Header.Add(k, v)
@@ -115,14 +117,16 @@ func SignRequest(req *http.Request, service string, creds Credentials, theTime t
 }
 
 // SignV4 modifies a map[string][]string of headers to generate an AWS V4 signature + headers based on the config/credentials provided.
-func SignV4(headers map[string][]string, method string, theURL *url.URL, body []byte, service string, awsCreds Credentials, theTime time.Time) (string, map[string]string) {
+func SignV4(headers map[string][]string, method string, theURL *url.URL, body []byte, service string,
+	awsCreds Credentials, theTime time.Time, disablePayloadSigning bool) (string, map[string]string) {
 	// General ref. https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
 	// S3 ref. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
 	// APIGateway ref. https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/
-	bodyHexHash := fmt.Sprintf("%x", sha256.Sum256(body))
 
 	now := theTime.UTC()
 
+	contentSha256 := getContentHash(disablePayloadSigning, body)
+
 	// V4 signing has specific ideas of how it wants to see dates/times encoded
 	dateNow := now.Format("20060102")
 	iso8601Now := now.Format("20060102T150405Z")
@@ -134,7 +138,7 @@ func SignV4(headers map[string][]string, method string, theURL *url.URL, body []
 
 	// s3 and glacier require the extra x-amz-content-sha256 header. other services do not.
 	if service == "s3" || service == "glacier" {
-		awsHeaders["x-amz-content-sha256"] = bodyHexHash
+		awsHeaders[amzContentSha256Key] = contentSha256
 	}
 
 	// the security token header is necessary for ephemeral credentials, e.g. from
@@ -173,7 +177,7 @@ func SignV4(headers map[string][]string, method string, theURL *url.URL, body []
 	// include the list of the signed headers
 	headerList := strings.Join(orderedKeys, ";")
 	canonicalReq += headerList + "\n"
-	canonicalReq += bodyHexHash
+	canonicalReq += contentSha256
 
 	// the "string to sign" is a time-bounded, scoped request token which
 	// is linked to the "canonical request" by inclusion of its SHA-256 hash
@@ -202,3 +206,11 @@ func SignV4(headers map[string][]string, method string, theURL *url.URL, body []
 
 	return authHeader, awsHeaders
 }
+
+// getContentHash returns UNSIGNED-PAYLOAD if payload signing is disabled else will compute sha256 from body
+func getContentHash(disablePayloadSigning bool, body []byte) string {
+	if disablePayloadSigning {
+		return v4.UnsignedPayload
+	}
+	return fmt.Sprintf("%x", sha256.Sum256(body))
+}
diff --git a/internal/providers/aws/signing_v4a.go b/internal/providers/aws/signing_v4a.go
@@ -32,6 +32,7 @@ const (
 	amzSecurityTokenKey = v4Internal.AmzSecurityTokenKey
 	amzDateKey          = v4Internal.AmzDateKey
 	authorizationHeader = "Authorization"
+	amzContentSha256Key = "x-amz-content-sha256"
 
 	signingAlgorithm = "AWS4-ECDSA-P256-SHA256"
 
@@ -192,7 +193,7 @@ func (s *httpSigner) Build() (signedRequest, error) {
 
 	// seemingly required by S3/MRAP -- 403 Forbidden otherwise
 	headers.Set("host", req.URL.Host)
-	headers.Set("x-amz-content-sha256", s.PayloadHash)
+	headers.Set(amzContentSha256Key, s.PayloadHash)
 
 	s.setRequiredSigningFields(headers, query)
 
@@ -381,8 +382,7 @@ type signedRequest struct {
 
 // SignV4a returns a map[string][]string of headers, including an added AWS V4a signature based on the config/credentials provided.
 func SignV4a(headers map[string][]string, method string, theURL *url.URL, body []byte, service string, awsCreds Credentials, theTime time.Time) map[string][]string {
-	bodyHexHash := fmt.Sprintf("%x", sha256.Sum256(body))
-
+	contentSha256 := getContentHash(false, body)
 	key, err := retrievePrivateKey(awsCreds)
 	if err != nil {
 		return map[string][]string{}
@@ -394,7 +394,7 @@ func SignV4a(headers map[string][]string, method string, theURL *url.URL, body [
 
 	signer := &httpSigner{
 		Request:     req,
-		PayloadHash: bodyHexHash,
+		PayloadHash: contentSha256,
 		ServiceName: service,
 		RegionSet:   []string{"*"},
 		Credentials: key,

diff --git a/plugins/rest/aws_test.go b/plugins/rest/aws_test.go
@@ -5,6 +5,7 @@
 package rest
 
 import (
+	"bytes"
 	"context"
 	"crypto/rand"
 	"encoding/json"
@@ -582,6 +583,84 @@ func TestV4Signing(t *testing.T) {
 	}
 }
 
+func TestV4SigningUnsignedPayload(t *testing.T) {
+	ts := ec2CredTestServer{}
+	ts.start()
+	defer ts.stop()
+
+	// happy path: sign correctly
+	cs := &awsMetadataCredentialService{
+		RoleName:        "my_iam_role", // not present
+		RegionName:      "us-east-1",
+		credServicePath: ts.server.URL + "/latest/meta-data/iam/security-credentials/",
+		tokenPath:       ts.server.URL + "/latest/api/token",
+		logger:          logging.Get(),
+	}
+	ts.payload = metadataPayload{
+		AccessKeyID:     "MYAWSACCESSKEYGOESHERE",
+		SecretAccessKey: "MYAWSSECRETACCESSKEYGOESHERE",
+		Code:            "Success",
+		Token:           "MYAWSSECURITYTOKENGOESHERE",
+		Expiration:      time.Now().UTC().Add(time.Minute * 2)}
+
+	// force a non-random source so that we can predict the v4a signing key and, thus, signature
+	myReader := strings.NewReader("000000000000000000000000000000000")
+	aws.SetRandomSource(myReader)
+	defer func() { aws.SetRandomSource(rand.Reader) }()
+
+	tests := []struct {
+		disablePayloadSigning bool
+		expectedAuthorization []string
+		expectedShaHeaderVal  string
+	}{
+		{
+			disablePayloadSigning: true,
+			expectedAuthorization: []string{
+				"AWS4-HMAC-SHA256 Credential=MYAWSACCESSKEYGOESHERE/20190424/us-east-1/s3/aws4_request," +
+					"SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token," +
+					"Signature=3682e55f6d86d3372003b3d28c74aa960f076d91fce833b129ae76415a12e5e4",
+			},
+			expectedShaHeaderVal: "UNSIGNED-PAYLOAD",
+		},
+		{
+			disablePayloadSigning: false,
+			expectedAuthorization: []string{
+				"AWS4-HMAC-SHA256 Credential=MYAWSACCESSKEYGOESHERE/20190424/us-east-1/s3/aws4_request," +
+					"SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token," +
+					"Signature=d3f0561abae5e35d9ee2c15e678bb7acacc4b4743707a8f7fbcbfdb519078990",
+			},
+			expectedShaHeaderVal: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+		},
+	}
+	for _, test := range tests {
+		creds, err := cs.credentials(context.Background())
+		if err != nil {
+			t.Fatal("unexpected error getting credentials")
+		}
+		req, _ := http.NewRequest("GET", "https://mybucket.s3.amazonaws.com/bundle.tar.gz", strings.NewReader(""))
+		var body []byte
+		if req.Body == nil {
+			body = []byte("")
+		} else {
+			body, _ = io.ReadAll(req.Body)
+			req.Body = io.NopCloser(bytes.NewReader(body))
+		}
+
+		authHeader, awsHeaders := aws.SignV4(req.Header, req.Method, req.URL, body, "s3", creds, time.Unix(1556129697, 0).UTC(), test.disablePayloadSigning)
+		req.Header.Set("Authorization", authHeader)
+		for k, v := range awsHeaders {
+			req.Header.Add(k, v)
+		}
+
+		// expect mandatory headers
+		assertEq("mybucket.s3.amazonaws.com", req.Header.Get("Host"), t)
+		assertIn(test.expectedAuthorization, req.Header.Get("Authorization"), t)
+		assertEq(test.expectedShaHeaderVal, req.Header.Get("X-Amz-Content-Sha256"), t)
+		assertEq("20190424T181457Z", req.Header.Get("X-Amz-Date"), t)
+		assertEq("MYAWSSECURITYTOKENGOESHERE", req.Header.Get("X-Amz-Security-Token"), t)
+	}
+}
+
 func TestV4SigningForApiGateway(t *testing.T) {
 	ts := ec2CredTestServer{}
 	ts.start()

diff --git a/test/cases/testdata/providers-aws/aws-sign_req-errors.yaml b/test/cases/testdata/providers-aws/aws-sign_req-errors.yaml
@@ -147,3 +147,29 @@ cases:
     want_error_code: eval_type_error
     want_error: "providers.aws.sign_req: operand 3 could not convert time_ns value into a unix timestamp"
     strict_error: true
+  # boolean type error for disable_payload_signing key
+  - data:
+    modules:
+      - |
+        package test
+        req := {"method": "get", "url": "https://example.com", "headers": {"foo": "bar"}}
+        aws_config := {"aws_access_key": "MYAWSACCESSKEYGOESHERE", "aws_secret_access_key": "MYAWSSECRETACCESSKEYGOESHERE", "aws_service": "s3", "aws_region": "us-east-1", "disable_payload_signing": "false"}
+        expected := {
+          "headers": {
+            "Authorization": "AWS4-HMAC-SHA256 Credential=MYAWSACCESSKEYGOESHERE/20151228/us-east-1/s3/aws4_request,SignedHeaders=foo;host;x-amz-content-sha256;x-amz-date,Signature=8f1dc7c9b9978356a0d0989fd26a95307f4f8a4aa264d8220647b7097d839952",
+            "foo": "bar",
+            "host": "example.com",
+            "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+            "x-amz-date": "20151228T140825Z"
+          },
+          "method": "get",
+          "url": "https://example.com"
+        }
+        p {
+            providers.aws.sign_req(req, aws_config, 1451311705000000000) == expected
+        }
+    note: providers-aws-sign_req/failure-simple-bad-type-payload-signing-config
+    query: data.test.p = x
+    want_error_code: eval_type_error
+    want_error: "providers.aws.sign_req: operand 2 invalid value for 'disable_payload_signing' in AWS config"
+    strict_error: true