Skip to content

Commit

Permalink
Merge branch 'main' of https://github.com/solo-io/gloo into fix-cli-d…
Browse files Browse the repository at this point in the history 
…ebug-tests
  • Loading branch information
@arianaw66
arianaw66 committed Dec 16, 2024
2 parents 3cde0c6 + 1bed380 commit 5904523
Show file tree
Hide file tree
Showing 15 changed files with 1,160 additions and 15 deletions.
5 changes: 5 additions & 0 deletions changelog/v1.19.0-beta3/data-plane-validation.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
changelog:
- type: NON_USER_FACING
description: >-
Adding docs for full Envoy validation.
skipCI-kube-tests:true
20 changes: 20 additions & 0 deletions changelog/v1.19.0-beta3/fix-sp-7315.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
changelog:
- type: FIX
issueLink: https://github.com/solo-io/solo-projects/issues/7315
resolvesIssue: false
description: |
gateway2/delegation: enable inherited policy overrides
Adds the ability to override inherited policy fields when
explicitly permitted by a parent route using the annotation
delegation.gateway.solo.io/enable-policy-overrides.
It supports a wildcard value "*" or a comma separated list
of field names such as "faults,timeouts,retries,headermanipulation".
Functionally, a child RouteOption may only override the RouteOptions
derived from its parent if the above annotation exists on the parent
route. This is required to make the override behavior safe to use.
Testing done:
- Translator tests for the new scenarios.
164 changes: 164 additions & 0 deletions ...content/guides/traffic_management/configuration_validation/admission_control.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ For more information about how resource configuration validation works in Gloo G

Configure the validating admission webhook to reject invalid Gloo custom resources before they are applied in the cluster.


1. Enable strict resource validation by updating your Gloo Gateway installation and set the following Helm values.
```bash
--set gateway.validation.alwaysAcceptResources=false
Expand Down Expand Up @@ -61,7 +62,170 @@ Configure the validating admission webhook to reject invalid Gloo custom resourc
{{< notice tip >}}
You can also use the validating admission webhook by running the <code>kubectl apply --dry-run=server</code> command to test your Gloo configuration before you apply it to your cluster. For more information, see <a href="#test-resource-configurations">Test resource configurations</a>.
{{< /notice >}}

## Enable full Envoy validation (beta) {#envoy-validation}

In addition to strict resource validation, you can enable full Envoy validation in your Gloo Gateway setup. The full Envoy validation adds another validation layer to the validation webhook by converting the translated xDS snapshot into static bootstrap configuration that can be fed into Envoy. This way, you can validate configuration that is typically accepted by Gloo Gateway, but later rejected by Envoy. For example, you might have a transformation policy in your VirtualService that uses an invalid Inja template. Gloo Gateway cannot validate the Inja template and therefore accepts the configuration. However, with the full Envoy validation enabled, this configuration is checked against Envoy and rejected if Envoy detects invalid configuration.

{{% notice note %}}
The full Envoy validation is a beta feature.
{{% /notice %}}
{{% notice warning %}}
Enabling full Envoy validation is a resource-intensive operation that can have a negative performance impact on your environment, especially if the environment has a lot of resources.
{{% /notice %}}

1. Follow the [Hello World guide]({{% versioned_link_path fromRoot="/guides/traffic_management/hello_world/" %}}) to set up the hello world app and expose it with a VirtualService.
2. Edit the Settings to enable strict resource validation without the full Envoy validation. Make sure to also set `disableTransformationValidation: true` to disable the transformation validation. To persist this setting between Gloo Gateway upgrades, add this setting to your Helm values file instead.
```sh
kubectl edit settings default -n gloo-system
```

Enter the following values:
```yaml
...
spec:
gateway:
validation:
allowWarnings: false
alwaysAccept: false
disableTransformationValidation: true
fullEnvoyValidation: false
...
```

3. Add a transformation policy to the hello world VirtualService that uses an invalid Inja template. The following example does not close the else statement. Verify that this configuration is accepted by Gloo Gateway.
{{< highlight yaml "hl_lines=23" >}}
kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: default
namespace: gloo-system
spec:
virtualHost:
domains:
- '*'
routes:
- matchers:
- exact: /all-pets
options:
prefixRewrite: /api/pets
stagedTransformations:
regular:
responseTransforms:
- responseTransformation:
transformationTemplate:
headers:
test_header:
text: '{% if default(data.error.message, "") != %}400{% else '
routeAction:
single:
upstream:
name: default-petstore-8080
namespace: gloo-system
EOF
{{< /highlight >}}

4. Review the logs of the `gateway-proxy` pod. Verify that although Gloo Gateway accepted the configuration, a warning regarding the malformatted Inja template is reported by Envoy.
```sh
kubectl logs -f -n gloo-system -l gateway-proxy
```

Example output:
```
[2024-12-13 18:43:12.090][1][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected: Failed to parse response template on response matcher: Failed to parse header template 'test_header': [inja.exception.parser_error] (at 1:42) too few arguments
```

5. Restore the old VirtualService again.
```yaml
kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: default
namespace: gloo-system
spec:
virtualHost:
domains:
- '*'
routes:
- matchers:
- exact: /all-pets
options:
prefixRewrite: /api/pets
routeAction:
single:
upstream:
name: default-petstore-8080
namespace: gloo-system
EOF
```

5. Edit the Settings resource to enable full Envoy validation. Note that you can leave `disableTransformationValidation: true`, because the transformation validation is included in the full Envoy validation.
```sh
kubectl edit settings default -n gloo-system
```

Enter the following values:
```yaml
...
spec:
gateway:
validation:
allowWarnings: false
alwaysAccept: false
disableTransformationValidation: true
fullEnvoyValidation: true
...
```

6. Try to apply the invalid VirtualService again. Verify that the resource is now rejected and the Envoy error about the invalid Inja template is surfaced to you.
{{< highlight yaml "hl_lines=23" >}}
kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: default
namespace: gloo-system
spec:
virtualHost:
domains:
- '*'
routes:
- matchers:
- exact: /all-pets
options:
prefixRewrite: /api/pets
stagedTransformations:
regular:
responseTransforms:
- responseTransformation:
transformationTemplate:
headers:
test_header:
text: '{% if default(data.error.message, "") != %}400{% else '
routeAction:
single:
upstream:
name: default-petstore-8080
namespace: gloo-system
EOF
{{< /highlight >}}

Example output:
```
Error from server: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"gateway.solo.io/v1\",\"kind\":\"VirtualService\",\"metadata\":{\"annotations\":{},\"name\":\"default\",\"namespace\":\"gloo-system\"},\"spec\":{\"virtualHost\":{\"domains\":[\"*\"],\"routes\":[{\"matchers\":[{\"exact\":\"/all-pets\"}],\"options\":{\"prefixRewrite\":\"/api/pets\",\"stagedTransformations\":{\"regular\":{\"responseTransforms\":[{\"responseTransformation\":{\"transformationTemplate\":{\"headers\":{\"test_header\":{\"text\":\"{% if default(data.error.message, \\\"\\\") != %}400{% else \"}}}}}]}}},\"routeAction\":{\"single\":{\"upstream\":{\"name\":\"default-petstore-8080\",\"namespace\":\"gloo-system\"}}}}]}}}\n"}},"spec":{"virtualHost":{"routes":[{"matchers":[{"exact":"/all-pets"}],"options":{"prefixRewrite":"/api/pets","stagedTransformations":{"regular":{"responseTransforms":[{"responseTransformation":{"transformationTemplate":{"headers":{"test_header":{"text":"{% if default(data.error.message, \"\") != %}400{% else "}}}}}]}}},"routeAction":{"single":{"upstream":{"name":"default-petstore-8080","namespace":"gloo-system"}}}}]}}}
to:
Resource: "gateway.solo.io/v1, Resource=virtualservices", GroupVersionKind: "gateway.solo.io/v1, Kind=VirtualService"
Name: "default", Namespace: "gloo-system"
for: "STDIN": error when patching "STDIN": admission webhook "gloo.gloo-system.svc" denied the request: resource incompatible with current Gloo snapshot: [Validating *v1.VirtualService failed: 1 error occurred:
* Validating *v1.VirtualService failed: validating *v1.VirtualService name:"default" namespace:"gloo-system": 1 error occurred:
* failed gloo validation resource reports: 2 errors occurred:
* invalid resource gloo-system.gateway-proxy
* envoy validation mode output: error initializing configuration '/dev/fd/0': Failed to parse response template on response matcher: Failed to parse header template 'test_header': [inja.exception.parser_error] (at 1:42) too few arguments
, error: command ""/usr/local/bin/envoy" "--mode" "validate" "--config-path" "/dev/fd/0" "-l" "critical" "--log-format" "%v"" failed with error: exit status 1
```

## View the current validating admission webhook configuration

Expand Down
2 changes: 1 addition & 1 deletion docs/content/guides/traffic_management/hello_world/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -377,7 +377,7 @@ At this point we have a Virtual Service with a routing rule sending traffic on t
Let’s test the route rule by retrieving the URL of Gloo Gateway, and sending a web request to the `/all-pets` path of the URL using curl.

```shell
curl $(glooctl proxy url)/all-pets
curl $(glooctl proxy url --name gateway-proxy)/all-pets
```

```json
Expand Down
3 changes: 3 additions & 0 deletions projects/gateway2/translator/gateway_translator_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -326,4 +326,7 @@ var _ = DescribeTable("Route Delegation translator",
Entry("Child route matcher does not match parent", "bug-6621.yaml"),
// https://github.com/k8sgateway/k8sgateway/issues/10379
Entry("Multi-level multiple parents delegation", "bug-10379.yaml"),
Entry("RouteOptions prefer child override when allowed", "route_options_inheritance_child_override_allow.yaml"),
Entry("RouteOptions multi level inheritance with child override when allowed", "route_options_multi_level_inheritance_override_allow.yaml"),
Entry("RouteOptions multi level inheritance with partial child override", "route_options_multi_level_inheritance_override_partial.yaml"),
)
79 changes: 70 additions & 9 deletions projects/gateway2/translator/plugins/routeoptions/route_options_plugin.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"strings"

"github.com/hashicorp/go-multierror"
"github.com/rotisserie/eris"
Expand All @@ -15,6 +16,7 @@ import (
"istio.io/istio/pkg/kube/krt"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/sets"
"sigs.k8s.io/controller-runtime/pkg/client"
gwv1 "sigs.k8s.io/gateway-api/apis/v1"

Expand All @@ -31,6 +33,15 @@ import (
glooutils "github.com/solo-io/gloo/projects/gloo/pkg/utils"
)

const (
// policyOverrideAnnotation can be set by parent routes to allow child routes to override
// all (wildcard *) or specific fields (comma separated field names) in RouteOptions inherited from the parent route.
policyOverrideAnnotation = "delegation.gateway.solo.io/enable-policy-overrides"

// wildcardField is used to enable overriding all fields in RouteOptions inherited from the parent route.
wildcardField = "*"
)

var (
_ plugins.RoutePlugin = &plugin{}
_ plugins.StatusPlugin = &plugin{}
Expand Down Expand Up @@ -83,7 +94,7 @@ func (p *plugin) ApplyRoutePlugin(
routeCtx *plugins.RouteContext,
outputRoute *gloov1.Route,
) error {
// check for RouteOptions applied to full Route
// check for RouteOptions applied to the given routeCtx
routeOptions, _, sources, err := p.handleAttachment(ctx, routeCtx)
if err != nil {
return err
Expand All @@ -92,22 +103,59 @@ func (p *plugin) ApplyRoutePlugin(
return nil
}

// If the route already has options set, we should override them.
// This is important because for delegated routes, the plugin will
// be invoked on the child routes multiple times for each parent route
// that may override them.
merged, usedExistingSources := glooutils.ShallowMergeRouteOptions(routeOptions, outputRoute.GetOptions())
merged, OptionsMergeResult := mergeOptionsForRoute(ctx, routeCtx.HTTPRoute, routeOptions, outputRoute.GetOptions())
if OptionsMergeResult == glooutils.OptionsMergedNone {
// No existing options merged into 'sources', so set the 'sources' on the outputRoute
routeutils.SetRouteSources(outputRoute, sources)
} else if OptionsMergeResult == glooutils.OptionsMergedPartial {
// Some existing options merged into 'sources', so append the 'sources' on the outputRoute
routeutils.AppendRouteSources(outputRoute, sources)
} // In case OptionsMergedFull, the correct sources are already set on the outputRoute

// Set the merged RouteOptions on the outputRoute
outputRoute.Options = merged

// Track the RouteOption policy sources that are used so we can report status on it
routeutils.AppendSourceToRoute(outputRoute, sources, usedExistingSources)
// Track that we used this RouteOption is our status cache
// we do this so we can persist status later for all attached RouteOptions
p.trackAcceptedRouteOptions(sources)
p.trackAcceptedRouteOptions(outputRoute.GetMetadataStatic().GetSources())

return nil
}

func mergeOptionsForRoute(
ctx context.Context,
route *gwv1.HTTPRoute,
dst, src *gloov1.RouteOptions,
) (*gloov1.RouteOptions, glooutils.OptionsMergeResult) {
// By default, lower priority options cannot override higher priority ones
// and can only augment them during a merge such that fields unset in the higher
// priority options can be merged in from the lower priority options.
// In the case of delegated routes, a parent route can enable child routes to override
// all (wildcard *) or specific fields using the policyOverrideAnnotation.
fieldsAllowedToOverride := sets.New[string]()

// If the route already has options set, we should override/augment them.
// This is important because for delegated routes, the plugin will
// be invoked on the child routes multiple times for each parent route
// that may override/augment them.
//
// By default, parent options (routeOptions) are preferred, unless the parent explicitly
// enabled child routes (outputRoute.Options) to override parent options.
fieldsStr, delegatedPolicyOverride := route.Annotations[policyOverrideAnnotation]
if delegatedPolicyOverride {
delegatedFieldsToOverride := parseDelegationFieldOverrides(fieldsStr)
if delegatedFieldsToOverride.Len() == 0 {
// Invalid annotation value, so log an error but enforce the default behavior of preferring the parent options.
contextutils.LoggerFrom(ctx).Errorf("invalid value %q for annotation %s on route %s; must be %s or a comma-separated list of field names",
fieldsStr, policyOverrideAnnotation, client.ObjectKeyFromObject(route), wildcardField)
} else {
fieldsAllowedToOverride = delegatedFieldsToOverride
}
}

return glooutils.MergeRouteOptionsWithOverrides(dst, src, fieldsAllowedToOverride)
}

func (p *plugin) InitStatusPlugin(ctx context.Context, statusCtx *plugins.StatusContext) error {
for _, proxyWithReport := range statusCtx.ProxiesWithReports {
// now that we translate proxies one by one, we can't assume ApplyRoutePlugin is called before ApplyStatusPlugin for all proxies
Expand Down Expand Up @@ -300,3 +348,16 @@ func extractRouteOptionSourceKeys(routeErr *validation.RouteReport_Error) (types

return types.NamespacedName{}, false
}

func parseDelegationFieldOverrides(val string) sets.Set[string] {
if val == wildcardField {
return sets.New(wildcardField)
}

set := sets.New[string]()
parts := strings.Split(val, ",")
for _, part := range parts {
set.Insert(strings.ToLower(strings.TrimSpace(part)))
}
return set
}
Loading

0 comments on commit 5904523

Please sign in to comment.