ReferenceGrant from Gateway to Secret (#791)

Problem: NKG does not support cross-namespace Secret references on Gateway.

Solution: Add support for ReferenceGrants that permit Gateways to reference Secrets
in different namespaces. NKG now processes ReferenceGrants and verifies that 
Gateways with references to Secrets in different Namespaces have a corresponding 
ReferenceGrant. If no ReferenceGrant exists, the RefNotPermitted reason is used in 
all the listener conditions (Accepted, Programmed, and ResolvedRefs), and the 
listener is marked invalid. Secrets will only be resolved if the reference is permitted.
No additional validation is needed for ReferenceGrant as it does not correspond to 
any nginx config. We treat every upsert/delete of a ReferenceGrant as a change. 
This means we will regenerate nginx config every time a ReferenceGrant is created, 
updated (generation must change), or deleted, even if it does not apply to the 
accepted Gateway.

conformance/Makefile

@@ -2,7 +2,6 @@ NKG_TAG = edge
 NKG_PREFIX = nginx-kubernetes-gateway
 GATEWAY_CLASS = nginx
 SUPPORTED_FEATURES = HTTPRoute,HTTPRouteQueryParamMatching,HTTPRouteMethodMatching,HTTPRoutePortRedirect,HTTPRouteSchemeRedirect
-EXEMPT_FEATURES = ReferenceGrant
 KIND_KUBE_CONFIG_FOLDER = $${HOME}/.kube/kind
 TAG = latest
 PREFIX = conformance-test-runner
@@ -62,7 +61,7 @@ run-conformance-tests: ## Run conformance tests
 		--image=$(PREFIX):$(TAG) --image-pull-policy=Never \
 		--overrides='{ "spec": { "serviceAccountName": "conformance" }  }' \
 		--restart=Never -- go test -v . -tags conformance -args --gateway-class=$(GATEWAY_CLASS) --debug \
-						        --supported-features=$(SUPPORTED_FEATURES) --exempt-features=$(EXEMPT_FEATURES)
+						        --supported-features=$(SUPPORTED_FEATURES)
 
 .PHONY: cleanup-conformance-tests
 cleanup-conformance-tests: ## Clean up conformance tests fixtures

conformance/tests/conformance-rbac.yaml

@@ -30,18 +30,13 @@ rules:
   - delete
   - get
   - list
-- apiGroups:
-  - gateway.networking.k8s.io
-  resources:
-  - gatewayclasses
-  verbs:
-  - get
-  - list
 - apiGroups:
   - gateway.networking.k8s.io
   resources:
   - gateways
   - httproutes
+  - referencegrants
+  - gatewayclasses
   verbs:
   - create
   - delete

deploy/manifests/rbac.yaml

@@ -38,6 +38,7 @@ rules:
   - gatewayclasses
   - gateways
   - httproutes
+  - referencegrants
   verbs:
   - list
   - watch

docs/gateway-api-compatibility.md

@@ -12,7 +12,7 @@ This document describes which Gateway API resources NGINX Kubernetes Gateway sup
 | [TLSRoute](#tlsroute) | Not supported |
 | [TCPRoute](#tcproute) | Not supported |
 | [UDPRoute](#udproute) | Not supported |
-| [ReferenceGrant](#referencegrant) |  Not supported |
+| [ReferenceGrant](#referencegrant) |  Partially supported |
 | [Custom policies](#custom-policies) | Not supported |
 
 ## Terminology
@@ -148,7 +148,20 @@ Fields:
 
 ### ReferenceGrant
 
-> Status: Not supported.
+> Status: Partially supported.
+
+NKG only supports ReferenceGrants that permit Gateways to reference Secrets. 
+
+Fields:
+* `spec`
+  * `to`
+    * `group` - supported.
+    * `kind` - partially supported. Only `Secret`.
+    * `name`- supported.
+  * `from`
+    * `group` - supported.
+    * `kind` - partially supported. Only `Gateway`.
+    * `namespace`- supported.
 
 ### Custom Policies
 

examples/https-termination/README.md

@@ -1,6 +1,8 @@
 # HTTPS Termination Example
 
-In this example, we expand on the simple [cafe-example](../cafe-example) by adding HTTPS termination to our routes and an HTTPS redirect from port 80 to 443.
+In this example, we expand on the simple [cafe-example](../cafe-example) by adding HTTPS termination to our routes and
+an HTTPS redirect from port 80 to 443. We will also show how you can use a ReferenceGrant to permit your Gateway to
+reference a Secret in a different Namespace.
 
 ## Running the Example
 
@@ -40,37 +42,50 @@ In this example, we expand on the simple [cafe-example](../cafe-example) by addi
 
 ## 3. Configure HTTPS Termination and Routing
 
-1. Create a Secret with a TLS certificate and key:
+1. Create the Namespace `certificate` and a Secret with a TLS certificate and key:
    ```
-   kubectl apply -f cafe-secret.yaml
+   kubectl apply -f certificate-ns-and-cafe-secret.yaml
    ```
 
    The TLS certificate and key in this Secret are used to terminate the TLS connections for the cafe application.
-   **Important**: This certificate and key are for demo purposes only.
+   > **Important**: This certificate and key are for demo purposes only.
+
+1. Create the `ReferenceGrant`:
+   ```
+   kubectl apply -f reference-grant.yaml
+   ```
+
+   This ReferenceGrant allows all Gateways in the `default` namespace to reference the `cafe-secret` Secret in
+   the `certificate` namespace.
 
 1. Create the `Gateway` resource:
    ```
    kubectl apply -f gateway.yaml
    ```
 
    This [Gateway](./gateway.yaml) configures:
-   * `http` listener for HTTP traffic
-   * `https` listener for HTTPS traffic. It terminates TLS connections using the `cafe-secret` we created in step 1.
+    * `http` listener for HTTP traffic
+    * `https` listener for HTTPS traffic. It terminates TLS connections using the `cafe-secret` we created in step 1.
 
 1. Create the `HTTPRoute` resources:
    ```
    kubectl apply -f cafe-routes.yaml
    ```
 
-   To configure HTTPS termination for our cafe application, we will bind our `coffee` and `tea` HTTPRoutes to the `https` listener in [cafe-routes.yaml](./cafe-routes.yaml) using the [`parentReference`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.ParentReference) field:
+   To configure HTTPS termination for our cafe application, we will bind our `coffee` and `tea` HTTPRoutes to
+   the `https` listener in [cafe-routes.yaml](./cafe-routes.yaml) using
+   the [`parentReference`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.ParentReference)
+   field:
 
    ```yaml
    parentRefs:
    - name: gateway
      sectionName: https
    ```
 
-   To configure an HTTPS redirect from port 80 to 443, we will bind the special `cafe-tls-redirect` HTTPRoute with a [`HTTPRequestRedirectFilter`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRequestRedirectFilter) to the `http` listener:
+   To configure an HTTPS redirect from port 80 to 443, we will bind the special `cafe-tls-redirect` HTTPRoute with
+   a [`HTTPRequestRedirectFilter`](https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io/v1beta1.HTTPRequestRedirectFilter)
+   to the `http` listener:
 
    ```yaml
    parentRefs:
@@ -80,13 +95,16 @@ In this example, we expand on the simple [cafe-example](../cafe-example) by addi
 
 ## 4. Test the Application
 
-To access the application, we will use `curl` to send requests to the `coffee` and `tea` Services. First, we will access the application over HTTP to test that the HTTPS redirect works. Then we will use HTTPS.
+To access the application, we will use `curl` to send requests to the `coffee` and `tea` Services. First, we will access
+the application over HTTP to test that the HTTPS redirect works. Then we will use HTTPS.
 
 ### 4.1 Test HTTPS Redirect
 
-To test that NGINX sends an HTTPS redirect, we will send requests to the `coffee` and `tea` Services on HTTP port. We will use curl's `--include` option to print the response headers (we are interested in the `Location` header).
+To test that NGINX sends an HTTPS redirect, we will send requests to the `coffee` and `tea` Services on HTTP port. We
+will use curl's `--include` option to print the response headers (we are interested in the `Location` header).
 
 To get a redirect for coffee:
+
 ```
 curl --resolve cafe.example.com:$GW_HTTP_PORT:$GW_IP http://cafe.example.com:$GW_HTTP_PORT/coffee --include
 HTTP/1.1 302 Moved Temporarily
@@ -96,6 +114,7 @@ Location: https://cafe.example.com:443/coffee
 ```
 
 To get a redirect for tea:
+
 ```
 curl --resolve cafe.example.com:$GW_HTTP_PORT:$GW_IP http://cafe.example.com:$GW_HTTP_PORT/tea --include
 HTTP/1.1 302 Moved Temporarily
@@ -104,9 +123,10 @@ Location: https://cafe.example.com:443/tea
 ...
 ```
 
-### 4.2 Access Coffee and Tea 
+### 4.2 Access Coffee and Tea
 
-Now we will access the application over HTTPS. Since our certificate is self-signed, we will use curl's `--insecure` option to turn off certificate verification.
+Now we will access the application over HTTPS. Since our certificate is self-signed, we will use curl's `--insecure`
+option to turn off certificate verification.
 
 To get coffee:
 
@@ -123,3 +143,45 @@ curl --resolve cafe.example.com:$GW_HTTPS_PORT:$GW_IP https://cafe.example.com:$
 Server address: 10.12.0.19:80
 Server name: tea-7cd44fcb4d-xfw2x
 ```
+
+### 4.3 Remove the ReferenceGrant
+
+To restrict access to the `cafe-secret` in the `certificate` Namespace, we can delete the ReferenceGrant we created in
+Step 3:
+
+```
+kubectl delete -f reference-grant.yaml
+```
+
+Now, if we try to access the application over HTTPS, we will get a connection refused error:
+```
+curl --resolve cafe.example.com:$GW_HTTPS_PORT:$GW_IP https://cafe.example.com:$GW_HTTPS_PORT/coffee --insecure -vvv
+...
+curl: (7) Failed to connect to cafe.example.com port 443 after 0 ms: Connection refused
+```
+
+
+You can also check the conditions of the Gateway `https` Listener to verify the that the reference is not permitted:
+
+```
+ Name:                    https
+ Conditions:
+   Last Transition Time:  2023-06-26T20:23:56Z
+   Message:               Certificate ref to secret certificate/cafe-secret not permitted by any ReferenceGrant
+   Observed Generation:   2
+   Reason:                RefNotPermitted
+   Status:                False
+   Type:                  Accepted
+   Last Transition Time:  2023-06-26T20:23:56Z
+   Message:               Certificate ref to secret certificate/cafe-secret not permitted by any ReferenceGrant
+   Observed Generation:   2
+   Reason:                RefNotPermitted
+   Status:                False
+   Type:                  ResolvedRefs
+   Last Transition Time:  2023-06-26T20:23:56Z
+   Message:               Certificate ref to secret certificate/cafe-secret not permitted by any ReferenceGrant
+   Observed Generation:   2
+   Reason:                Invalid
+   Status:                False
+   Type:                  Programmed
+```

examples/https-termination/cafe-secret.yaml → examples/https-termination/certificate-ns-and-cafe-secret.yaml

@@ -1,7 +1,13 @@
 apiVersion: v1
+kind: Namespace
+metadata:
+  name: certificate
+---
+apiVersion: v1
 kind: Secret
 metadata:
   name: cafe-secret
+  namespace: certificate
 type: kubernetes.io/tls
 data:
   tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNzakNDQVpvQ0NRQzdCdVdXdWRtRkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBYk1Sa3dGd1lEVlFRRERCQmoKWVdabExtVjRZVzF3YkdVdVkyOXRNQjRYRFRJeU1EY3hOREl4TlRJek9Wb1hEVEl6TURjeE5ESXhOVEl6T1ZvdwpHekVaTUJjR0ExVUVBd3dRWTJGbVpTNWxlR0Z0Y0d4bExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTHFZMnRHNFc5aStFYzJhdnV4Q2prb2tnUUx1ek10U1Rnc1RNaEhuK3ZRUmxIam8KVzFLRnMvQVdlS25UUStyTWVKVWNseis4M3QwRGtyRThwUisxR2NKSE50WlNMb0NEYUlRN0Nhck5nY1daS0o4Qgo1WDNnVS9YeVJHZjI2c1REd2xzU3NkSEQ1U2U3K2Vab3NPcTdHTVF3K25HR2NVZ0VtL1Q1UEMvY05PWE0zZWxGClRPL051MStoMzROVG9BbDNQdTF2QlpMcDNQVERtQ0thaEROV0NWbUJQUWpNNFI4VERsbFhhMHQ5Z1o1MTRSRzUKWHlZWTNtdzZpUzIrR1dYVXllMjFuWVV4UEhZbDV4RHY0c0FXaGRXbElweHlZQlNCRURjczN6QlI2bFF1OWkxZAp0R1k4dGJ3blVmcUVUR3NZdWxzc05qcU95V1VEcFdJelhibHhJZVVDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBcjkrZWJ0U1dzSnhLTGtLZlRkek1ISFhOd2Y5ZXFVbHNtTXZmMGdBdWVKTUpUR215dG1iWjlpbXQKL2RnWlpYVE9hTElHUG9oZ3BpS0l5eVVRZVdGQ2F0NHRxWkNPVWRhbUloOGk0Q1h6QVJYVHNvcUNOenNNLzZMRQphM25XbFZyS2lmZHYrWkxyRi8vblc0VVNvOEoxaCtQeDljY0tpRDZZU0RVUERDRGh1RUtFWXcvbHpoUDJVOXNmCnl6cEJKVGQ4enFyM3paTjNGWWlITmgzYlRhQS82di9jU2lyamNTK1EwQXg4RWpzQzYxRjRVMTc4QzdWNWRCKzQKcmtPTy9QNlA0UFlWNTRZZHMvRjE2WkZJTHFBNENCYnExRExuYWRxamxyN3NPbzl2ZzNnWFNMYXBVVkdtZ2todAp6VlZPWG1mU0Z4OS90MDBHUi95bUdPbERJbWlXMGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

examples/https-termination/gateway.yaml

@@ -18,4 +18,4 @@ spec:
       certificateRefs:
       - kind: Secret
         name: cafe-secret
-        namespace: default
+        namespace: certificate

examples/https-termination/reference-grant.yaml

@@ -0,0 +1,14 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+  name: allow-default-to-cafe-secret
+  namespace: certificate
+spec:
+  to:
+  - group: ""
+    kind: Secret
+    name: cafe-secret # if you omit this name, then Gateways in default ns can access all Secrets in the certificate ns
+  from:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    namespace: default

internal/events/handler.go

@@ -126,6 +126,8 @@ func (h *EventHandlerImpl) propagateUpsert(e *UpsertEvent) {
 		h.cfg.Processor.CaptureUpsertChange(r)
 	case *v1beta1.HTTPRoute:
 		h.cfg.Processor.CaptureUpsertChange(r)
+	case *v1beta1.ReferenceGrant:
+		h.cfg.Processor.CaptureUpsertChange(r)
 	case *apiv1.Service:
 		h.cfg.Processor.CaptureUpsertChange(r)
 	case *apiv1.Namespace:
@@ -149,6 +151,8 @@ func (h *EventHandlerImpl) propagateDelete(e *DeleteEvent) {
 		h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
 	case *v1beta1.HTTPRoute:
 		h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
+	case *v1beta1.ReferenceGrant:
+		h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
 	case *apiv1.Service:
 		h.cfg.Processor.CaptureDeleteChange(e.Type, e.NamespacedName)
 	case *apiv1.Namespace:

internal/manager/manager.go

@@ -121,6 +121,9 @@ func Start(cfg config.Config) error {
 				controller.WithK8sPredicate(k8spredicate.LabelChangedPredicate{}),
 			},
 		},
+		{
+			objectType: &gatewayv1beta1.ReferenceGrant{},
+		},
 	}
 
 	ctx := ctlr.SetupSignalHandler()
@@ -207,6 +210,7 @@ func prepareFirstEventBatchPreparerArgs(
 		&apiv1.NamespaceList{},
 		&discoveryV1.EndpointSliceList{},
 		&gatewayv1beta1.HTTPRouteList{},
+		&gatewayv1beta1.ReferenceGrantList{},
 	}
 
 	if gwNsName == nil {

internal/manager/manager_test.go

@@ -34,6 +34,7 @@ func TestPrepareFirstEventBatchPreparerArgs(t *testing.T) {
 				&discoveryV1.EndpointSliceList{},
 				&gatewayv1beta1.HTTPRouteList{},
 				&gatewayv1beta1.GatewayList{},
+				&gatewayv1beta1.ReferenceGrantList{},
 			},
 		},
 		{
@@ -52,6 +53,7 @@ func TestPrepareFirstEventBatchPreparerArgs(t *testing.T) {
 				&apiv1.NamespaceList{},
 				&discoveryV1.EndpointSliceList{},
 				&gatewayv1beta1.HTTPRouteList{},
+				&gatewayv1beta1.ReferenceGrantList{},
 			},
 		},
 	}

internal/state/change_processor.go

@@ -85,11 +85,12 @@ type ChangeProcessorImpl struct {
 // NewChangeProcessorImpl creates a new ChangeProcessorImpl for the Gateway resource with the configured namespace name.
 func NewChangeProcessorImpl(cfg ChangeProcessorConfig) *ChangeProcessorImpl {
 	clusterStore := graph.ClusterState{
-		GatewayClasses: make(map[types.NamespacedName]*v1beta1.GatewayClass),
-		Gateways:       make(map[types.NamespacedName]*v1beta1.Gateway),
-		HTTPRoutes:     make(map[types.NamespacedName]*v1beta1.HTTPRoute),
-		Services:       make(map[types.NamespacedName]*apiv1.Service),
-		Namespaces:     make(map[types.NamespacedName]*apiv1.Namespace),
+		GatewayClasses:  make(map[types.NamespacedName]*v1beta1.GatewayClass),
+		Gateways:        make(map[types.NamespacedName]*v1beta1.Gateway),
+		HTTPRoutes:      make(map[types.NamespacedName]*v1beta1.HTTPRoute),
+		Services:        make(map[types.NamespacedName]*apiv1.Service),
+		Namespaces:      make(map[types.NamespacedName]*apiv1.Namespace),
+		ReferenceGrants: make(map[types.NamespacedName]*v1beta1.ReferenceGrant),
 	}
 
 	extractGVK := func(obj client.Object) schema.GroupVersionKind {
@@ -119,6 +120,11 @@ func NewChangeProcessorImpl(cfg ChangeProcessorConfig) *ChangeProcessorImpl {
 				store:             newObjectStoreMapAdapter(clusterStore.HTTPRoutes),
 				trackUpsertDelete: true,
 			},
+			{
+				gvk:               extractGVK(&v1beta1.ReferenceGrant{}),
+				store:             newObjectStoreMapAdapter(clusterStore.ReferenceGrants),
+				trackUpsertDelete: true,
+			},
 			{
 				gvk:               extractGVK(&apiv1.Namespace{}),
 				store:             newObjectStoreMapAdapter(clusterStore.Namespaces),
@@ -145,8 +151,8 @@ func NewChangeProcessorImpl(cfg ChangeProcessorConfig) *ChangeProcessorImpl {
 
 			var err error
 			switch o := obj.(type) {
-			// We don't validate GatewayClass, because as of 0.7.1, the webhook doesn't validate it (it only
-			// validates an update that requires the previous version of the resource,
+			// We don't validate GatewayClass or ReferenceGrant, because as of 0.7.1, the webhook doesn't validate them.
+			// It only validates a GatewayClass update that requires the previous version of the resource,
 			// which NKG cannot reliably provide - for example, after NKG restarts).
 			// https://github.com/kubernetes-sigs/gateway-api/blob/v0.7.1/apis/v1beta1/validation/gatewayclass.go#L28
 			case *v1beta1.Gateway: