Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/dynamic upstreams #221

Merged
merged 1 commit into from
Sep 30, 2022
Merged

Feature/dynamic upstreams #221

merged 1 commit into from
Sep 30, 2022

Conversation

kate-osborn
Copy link
Contributor

@kate-osborn kate-osborn commented Sep 2, 2022 •
edited
Loading

Adds support for dynamic upstreams.

Previously, NKG used the cluster IP of the Service to route traffic to the backend Services specified by HTTPRoutes. With this PR, NKG will use the endpoints of the Pods corresponding to a Service as the upstream servers for a backend Service.

This change adds the following components:

  • EndpointSlice controller. This controller subscribes to EndpointSlice CRUD events and indexes EndpointSlices by their Service owners. This allows us to perform O(1) lookups of EndpointSlices by Service owners.
  • Relationship.Capturer. This component is responsible for tracking relationships between Gateway API resources (e.g., HTTPRoutes) and non-Gateway API resources (e.g., Services). This allows us to determine when a Service or EndpointSlice update requires an nginx reload.
  • ServiceResolver. This component replaces the ServiceStore and is responsible for resolving a Service and Port to a list of endpoints.

This PR also adds upstreams to the nginx config generator. One upstream is generated for each unique and valid Service:Port backend ref. If a backendRef cannot be resolved, an upstream is generated that returns a 502. In addition, a default upstream that returns a 502 is generated to handle the case where a backend ref is invalid.

Known limitations:

  • Traffic cannot be routed to Headless Services that do not have a defined port. While this is a valid Kubernetes Service configuration, we do not currently understand the use case for it. I will do some research and see what I can find out.
  • If a user manually creates an EndpointSlice, they will need to populate set the kubernetes.io/service-name label, and set the ready condition of the endpoints to true. Otherwise, NKG will fail to resolve the Service endpoints.

pleshakov
pleshakov suggested changes Sep 7, 2022
View reviewed changes
Copy link
Contributor

@pleshakov pleshakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kate-osborn please see my review

A few more things:

(1)
Could we update the api compatibility doc? The part about backendRefs in HTTPRoute

NGINX Kubernetes Gateway will use the IP of the Service as a backend, not the IPs of the corresponding Pods. Watching for Service updates is not supported.

(2) I noticed a bug:

(a) deploy cafe example
(b) confirm everything is deployed and working
(c) change tea svc port from 80 to 7777 in the service manifest
bug: config doesn’t get regenerated - keeps using the old endpoints.

internal/implementations/endpointslice/endpointslice.go Outdated Show resolved Hide resolved
internal/nginx/config/upstreams_template.go Outdated Show resolved Hide resolved
internal/events/handler.go Outdated Show resolved Hide resolved
internal/nginx/config/generator.go Outdated Show resolved Hide resolved
internal/nginx/config/upstreams_template.go Outdated Show resolved Hide resolved
internal/nginx/config/generator_test.go Outdated Show resolved Hide resolved
internal/state/change_processor.go Outdated Show resolved Hide resolved
internal/state/change_processor.go Outdated Show resolved Hide resolved
internal/state/graph.go Show resolved Hide resolved
internal/state/graph_test.go Outdated Show resolved Hide resolved
@kate-osborn kate-osborn force-pushed the feature/dynamic-upstreams branch from 9fcbfac to 2936171 Compare September 15, 2022 17:28
@kate-osborn kate-osborn marked this pull request as ready for review September 15, 2022 17:28
@kate-osborn
Copy link
Contributor Author

@kate-osborn please see my review

A few more things:

(1) Could we update the api compatibility doc? The part about backendRefs in HTTPRoute

NGINX Kubernetes Gateway will use the IP of the Service as a backend, not the IPs of the corresponding Pods. Watching for Service updates is not supported.

(2) I noticed a bug:

(a) deploy cafe example (b) confirm everything is deployed and working (c) change tea svc port from 80 to 7777 in the service manifest bug: config doesn’t get regenerated - keeps using the old endpoints.

Thanks @pleshakov

I fixed the bug you found and updated the docs.

@kate-osborn kate-osborn mentioned this pull request Sep 19, 2022
Closed
pleshakov
pleshakov reviewed Sep 19, 2022
View reviewed changes
internal/state/relationship/capturer.go Outdated Show resolved Hide resolved
internal/state/relationship/capturer.go Outdated Show resolved Hide resolved
internal/state/change_processor.go Outdated Show resolved Hide resolved
pkg/sdk/endpointslice_controller.go Outdated Show resolved Hide resolved
pkg/sdk/endpointslice_controller.go Outdated Show resolved Hide resolved
internal/state/graph_test.go Outdated Show resolved Hide resolved
internal/state/relationship/capturer_test.go Show resolved Hide resolved
internal/state/resolver/service_resolver_test.go Outdated Show resolved Hide resolved
internal/state/relationship/capturer.go Outdated Show resolved Hide resolved
@github-actions github-actions bot added documentation Improvements or additions to documentation enhancement New feature or request labels Sep 22, 2022
pleshakov
pleshakov reviewed Sep 22, 2022
View reviewed changes
internal/state/graph.go Outdated Show resolved Hide resolved
internal/state/relationship/capturer.go Outdated Show resolved Hide resolved
internal/state/resolver/resolver.go Outdated Show resolved Hide resolved
pleshakov
pleshakov approved these changes Sep 27, 2022
View reviewed changes
Copy link
Contributor

@pleshakov pleshakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@kate-osborn
Add support for dynamic upstreams
231937f 
Previously, NKG used the cluster IP of the Service to route traffic to the
backend Services specified by HTTPRoutes. With this commit, NKG will use the
endpoints of the Pods corresponding to a Service as the upstream servers for a
backend Service.

This change adds the following components:
* EndpointSlice controller for caching and listing EndpointSlices.
* Relationship.Capturer for tracking and reporting on relationships between
  Gateway API resources and non-Gateway API resources (e.g. Services).
* ServiceResolver replaces the ServiceStore and resolves Service:Port to a list
* of endpoints.

This commit also adds upstreams to the nginx config generator. One upstream is
generated for each unique and valid Service:Port BackendRef. If a BackendRef
cannot be resolved, a 502 is returned.

Known Limitations:
* Traffic cannot be routed to Headless Services that do not have a defined port.
* If a user manually creates and EndpointSlice, they will need to populate the
  "kubernetes.io/service-name" label and set the ready condition of the
   endpoints to true. Otherwise, NKG will fail to resolve the Service endpoints.
@kate-osborn kate-osborn force-pushed the feature/dynamic-upstreams branch from 52127a5 to 231937f Compare September 30, 2022 17:23
@kate-osborn kate-osborn merged commit c4a0727 into main Sep 30, 2022
@kate-osborn kate-osborn deleted the feature/dynamic-upstreams branch September 30, 2022 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pleshakov pleshakov pleshakov approved these changes

@f5yacobucci f5yacobucci Awaiting requested review from f5yacobucci

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kate-osborn @pleshakov @f5yacobucci