Gateway Settings Enhancement Proposal

Problem: We want a design for GatewayClass level settings that applies to all Gateways, such as `otel_exporter`.

Solution: Add enhancement proposal introducing `GatewaySettings`.

Enhancement Proposal: nginxinc#1775

docs/proposals/gateway-settings.md

@@ -0,0 +1,175 @@
+# Enhancement Proposal-1630: Gateway Settings
+
+- Issue: https://github.com/nginxinc/nginx-gateway-fabric/issues/1775
+- Status: Implementable
+
+## Summary
+
+This Enhancement Proposal introduces the `GatewaySettings` API, which allows Cluster Operators to define configuration at the GatewayClass level for all Gateways under that Class. This configuration is attached via the GatewayClass `parametersRef` field.
+
+## Goals
+
+- Define the Gateway settings.
+- Define an API for the Gateway settings.
+
+## Non-Goals
+
+- Provide implementation details for implementing the Gateway settings.
+
+## Introduction
+
+### Gateway Settings
+
+Gateway Settings are NGINX directives or configuration attached at the GatewayClass level that should be solely controlled by the Cluster Operator and should not be changed by the Application Developers. All Gateways attached to this GatewayClass will inherit these settings.
+
+These settings apply to the `main`, `http`, and/or `stream` contexts of the NGINX config.
+
+To begin, the Gateway Settings config will include the following NGINX directives (focusing on OpenTelemetry tracing):
+
+- [`otel_exporter`](https://nginx.org/en/docs/ngx_otel_module.html#otel_exporter)
+- [`otel_service_name`](https://nginx.org/en/docs/ngx_otel_module.html#otel_service_name)
+- [`otel_span_attr`](https://nginx.org/en/docs/ngx_otel_module.html#otel_span_attr): set global span attributes that will be merged with the span attributes set in the [Observability extension](nginx-extensions.md#gateway-settings).
+
+In the future, this config will be extended to support other directives, such as those defined in the [NGINX Extensions Proposal](nginx-extensions.md#gateway-settings).
+
+## API, Customer Driven Interfaces, and User Experience
+
+The `GatewaySettings` API is a CRD that is a part of the `gateway.nginx.org` Group. It will be referenced in the `parametersRef` field of a GatewayClass. It will live at the cluster scope.
+
+For example, a `GatewaySettings` named `gw-settings` would be referenced as follows:
+
+```yaml
+kind: GatewayClass
+metadata:
+    name: nginx
+spec:
+    controllerName: gateway.nginx.org/nginx-gateway-controller
+    parametersRef:
+        group: gateway.nginx.org/v1alpha1
+        kind: GatewaySettings
+        name: gw-settings
+```
+
+Below is the Golang API for the `GatewaySettings` API:
+
+### Go
+
+```go
+package v1alpha1
+
+import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+type GatewaySettings struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// Spec defines the desired state of the GatewaySettings.
+	Spec GatewaySettingsSpec `json:"spec"`
+
+	// Status defines the state of the GatewaySettings.
+	Status GatewaySettingsStatus `json:"status,omitempty"`
+}
+
+type GatewaySettingsSpec struct {
+	// OtelExporter specifies OpenTelemetry export parameters.
+	// +optional
+	OtelExporter *OtelExporter `json:"otelExporter,omitempty"`
+
+	// OtelServiceName is the "service.name" attribute of the Otel resource.
+	// +optional
+	OtelServiceName *string `json:"otelServiceName,omitempty"`
+
+	// OtelSpanAttributes are custom key/value attributes that are added to each span.
+	// Variables can be included in the values.
+	// +optional
+	OtelSpanAttributes map[string]string `json:"otelSpanAttributes,omitempty"`
+}
+
+type GatewaySettingsStatus struct {
+	// Conditions describe the current conditions of the GatewaySettings.
+	// +optional
+	Conditions []metav1.Condition `json:"conditions,omitempty"`
+}
+
+// OtelExporter specifies OpenTelemetry export parameters.
+type OtelExporter struct {
+	// Interval is the maximum interval between two exports, by default is 5 seconds.
+	// +optional
+	Interval *Duration `json:"interval,omitempty"`
+
+	// BatchSize is the maximum number of spans to be sent in one batch per worker, by default is 512.
+	// +optional
+	BatchSize *int32 `json:"batchSize,omitempty"`
+
+	// BatchCount is the number of pending batches per worker, spans exceeding the limit are dropped,
+	// by default is 4.
+	// +optional
+	BatchCount *int32 `json:"batchCount,omitempty"`
+
+	// Endpoint is the address of OTLP/gRPC endpoint that will accept telemetry data.
+	Endpoint string `json:"endpoint"`
+}
+
+// Duration is a string value representing a duration in time.
+// The format is a subset of the syntax parsed by Golang time.ParseDuration.
+// Examples: 1h, 12m, 30s, 150ms.
+type Duration string
+```
+
+### Status
+
+#### Conditions
+
+According to the [Policy and Metaresources GEP](https://gateway-api.sigs.k8s.io/geps/gep-713/), the `GatewaySettings` CRD must include a `status` stanza with a slice of Conditions.
+
+The `Accepted` Condition must be populated on the `GatewaySettings` CRD using the reasons defined in the [PolicyCondition API](https://github.com/kubernetes-sigs/gateway-api/blob/main/apis/v1alpha2/policy_types.go). If these reasons are not sufficient, we can add implementation-specific reasons.
+
+#### GatewayClass Status
+
+While this status does not officially exist, in order to be consistent with other resources with object refs we should add a custom `ResolvedRefs` Condition to the `GatewayClass` when the `parametersRef` is in use.
+
+NGINX Gateway Fabric must set this Condition on the GatewayClass affected by a `GatewaySettings`.
+Below is an example of what this Condition may look like:
+
+```yaml
+Conditions:
+  Type:                  gateway.nginx.org/ResolvedRefs
+  Message:               All references are resolved
+  Observed Generation:   1
+  Reason:                ResolvedRefs
+  Status:                True
+```
+
+Some additional rules:
+
+- This Condition should be added when the affected object starts being affected by a `GatewaySettings`.
+- When the `GatewaySettings` affecting that object is removed, the Condition should be removed.
+- The Observed Generation is the generation of the GatewayClass, not the generation of the `GatewaySettings`.
+
+## Use Cases
+
+- As a Cluster Operator, I want to set global settings that will apply to all Gateways that are a part of a GatewayClass. These settings should not be overriden at a lower level.
+
+## Testing
+
+- Unit tests
+- Functional tests that verify the attachment of the CRD to the GatewayClass, and that NGINX behaves properly based on the configuration.
+
+## Security Considerations
+
+Validating all fields in the `GatewaySettings` is critical to ensuring that the NGINX config generated by NGINX Gateway Fabric is correct and secure.
+
+All fields in the `GatewaySettings` will be validated with Open API Schema. If the Open API Schema validation rules are not sufficient, we will use [CEL](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules).
+
+RBAC via the Kubernetes API server will ensure that only authorized users can update the CRD containing Gateway Settings.
+
+## Alternatives
+
+- ParametersRef with ConfigMap: A ConfigMap is another resource type where a user can provide configuration options. However, unlike CRDs, ConfigMaps do not have built-in schema validation, versioning, or conversion webhooks.
+- Direct Policy: A Direct Policy may also work for Gateway Settings. It can be attached to a Gateway and scoped to Cluster Operators through RBAC. It would allow Cluster Operators to apply settings for specific Gateways, instead of all Gateways.
+
+## References
+
+- [NGINX Extensions Enhancement Proposal](nginx-extensions.md)
+- [Attaching Policy to GatewayClass](https://gateway-api.sigs.k8s.io/geps/gep-713/#attaching-policy-to-gatewayclass)
+- [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md)