Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming lists documentation #39993

Merged
merged 1 commit into from
Mar 27, 2023
Merged

Streaming lists documentation #39993

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions content/en/docs/reference/command-line-tools-reference/feature-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ For a reference to old feature gates that are removed, please refer to
| `UserNamespacesStatelessPodsSupport` | `false` | Alpha | 1.25 | |
| `ValidatingAdmissionPolicy` | `false` | Alpha | 1.26 | |
| `VolumeCapacityPriority` | `false` | Alpha | 1.21 | - |
| `WatchList` | false | Alpha | 1.27 | |
| `WinDSR` | `false` | Alpha | 1.14 | |
| `WinOverlay` | `false` | Alpha | 1.14 | 1.19 |
| `WinOverlay` | `true` | Beta | 1.20 | |
Expand Down Expand Up @@ -729,6 +730,7 @@ Each feature gate is designed for enabling/disabling a specific feature:
- `VolumeCapacityPriority`: Enable support for prioritizing nodes in different
topologies based on available PV capacity.
- `WatchBookmark`: Enable support for watch bookmark events.
- `WatchList` : Enable support for [streaming initial state of objects in watch requests](/docs/reference/using-api/api-concepts/#streaming-lists).
- `WinDSR`: Allows kube-proxy to create DSR loadbalancers for Windows.
- `WinOverlay`: Allows kube-proxy to run in overlay mode for Windows.
- `WindowsHostNetwork`: Enables support for joining Windows containers to a hosts' network namespace.
Expand Down
60 changes: 59 additions & 1 deletion content/en/docs/reference/using-api/api-concepts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ For subscribing to collections, Kubernetes client libraries typically offer some
of standard tool for this **list**-then-**watch** logic. (In the Go client library,
this is called a `Reflector` and is located in the `k8s.io/client-go/tools/cache` package.)

### Watch bookmarks
### Watch bookmarks {#watch-bookmarks}

To mitigate the impact of short history window, the Kubernetes API provides a watch
event named `BOOKMARK`. It is a special kind of event to mark that all changes up
Expand Down Expand Up @@ -226,6 +226,64 @@ As a client, you can request `BOOKMARK` events by setting the
assume bookmarks are returned at any specific interval, nor can clients assume that
the API server will send any `BOOKMARK` event even when requested.

## Streaming lists

{{< feature-state for_k8s_version="v1.27" state="alpha" >}}

On large clusters, retrieving the collection of some resource types may result in
a significant increase of resource usage (primarily RAM) on the control plane.
In order to alleviate its impact and simplify the user experience of the **list** + **watch**
pattern, Kubernetes v1.27 introduces as an alpha feature the support
for requesting the initial state (previously requested via the **list** request) as part of
the **watch** request.
Comment on lines +236 to +238
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pattern, Kubernetes v1.27 introduces as an alpha feature the support
for requesting the initial state (previously requested via the **list** request) as part of
the **watch** request.
pattern, Kubernetes {{< skew currentVersion >}} allows you to fetch the contents of a collection
as part of a **watch** request. See [Efficient detection of changes](#efficient-detection-of-changes)
to learn more about watches in Kubernetes.
As a client, you can either continue the **watch** to subscribe to further event updates, or you can
use this mechanism purely as a more efficient alternative to **list** a collection.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pattern, Kubernetes {{< skew currentVersion >}}

Please figure it out - I got the opposite comment from @tengqm - I'm with this project for 8y+, but it can be extremely annoying if different reviewers are giving contradicting comments.

Also - as I mentioend in the other comment I don't want to endorse this as a more efficient alternative to LIST yet - we're not ready for that. So let's not try to recommend things we're not ready for.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the recommend vs. not, feel free to frame this as experimental rather than as more efficient. It's good if a reader can still see under what conditions they might try it out.

We don't have a clear style guideline for when to mention the current version or not. Our overall guidance is to aim for timeless documentation.
Sorry about the mixed messages. I know I've got a particular way of writing for maintainability, and because this is text not code, there's no single right way to do it.

Here's a test: imagine that what you write is still present in Kubernetes 1.42 and the feature is still alpha. Does it make sense and feel current for a 1.42 release? If it clearly does, it's probably fine.


Provided that the `WatchList` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
is enabled, this can be achieved by specifying `sendInitialEvents=true` as query string parameter
Comment on lines +240 to +241
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Provided that the `WatchList` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
is enabled, this can be achieved by specifying `sendInitialEvents=true` as query string parameter
Provided your cluster is running at least Kubernetes v1.27 and you have enabled the
the `WatchList` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/), you can
perform a streaming list by specifying `sendInitialEvents=true` as query string parameter

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 1.27 version is mentioned 2 lines above (in addition to the header of the section). I don't think it makes sense to constantly repeat it - for me as a reader it would be more annoying than useful.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm proposing that this (and the feature state shortcode, which lots of readers don't fully notice) is where we make the requirements clear.

That way, we're documenting in one place both:

  • the version requirement
  • the need to explicitly enable a feature gate

My experience suggests putting these details in 2 places is likely to lead to more GitHub issues telling us the docs are wrong (even when the docs are strictly correct and have been misunderstood).

in a **watch** request. If set, the API server starts the watch stream with synthetic init
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

synthetic init events

Are these always type: ADDED? Maybe add a note about that here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes - added

events (of type `ADDED`) to build the whole state of all existing objects followed by a
[`BOOKMARK` event](/docs/reference/using-api/api-concepts/#watch-bookmarks)
(if requested via `allowWatchBookmarks=true` option). The bookmark event includes the resource version
to which is synced. After sending the bookmark event, the API server continues as for any other **watch**
request.

When you set `sendInitialEvents=true` in the query string, Kubernetes also requires that you set
`resourceVersionMatch` to `NotOlderThan` value.
If you provided `resourceVersion` in the query string without providing a value or don't provide
it at all, this is interpreted as a request for _consistent read_;
the bookmark event is sent when the state is synced at least to the moment of a consistent read
from when the request started to be processed. If you specify `resourceVersion` (in the query string),
the bookmark event is sent when the state is synced at least to the provided resource version.

### Example {#example-streaming-lists}

An example: you want to watch a collection of Pods. For that collection, the current resource version
is 10245 and there are two pods: `foo` and `bar`. Then sending the following request (explicitly requesting
_consistent read_ by setting empty resource version using `resourceVersion=`) could result
in the following sequence of events:

```console
GET /api/v1/namespaces/test/pods?watch=1&sendInitialEvents=true&allowWatchBookmarks=true&resourceVersion=&resourceVersionMatch=NotOlderThan
sftim marked this conversation as resolved.
Show resolved Hide resolved
---
200 OK
Transfer-Encoding: chunked
Content-Type: application/json

{
"type": "ADDED",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(aside) Did you consider adding a new event type for the synthetic init events? I could imagine a case where a controller might want to use WatchList to initialize it's cache, but also takes a special action when resources are added that shouldn't be performed on the initialization events.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - it's not a good idea.
Because technically watch with RV=0 was supported from k8s 1.0 version and that also had ADDED events. Changing that would introduce more confusion or incompatibility than it's worth it.

And the bookmark give you exactly what you need in case you want to handle them differently (which btw isn't clear - we changed reflector to it (I mean optionally now) and having ADDED is good enough).

"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "8467", "name": "foo"}, ...}
}
{
"type": "ADDED",
"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "5726", "name": "bar"}, ...}
}
{
"type": "BOOKMARK",
"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10245"} }
}
...
<followed by regular watch stream starting from resourceVersion="10245">
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't explain why watch is set to 1 but sendInitialEvent and allowWatchBookmarks are set to true. Does the API let us use a consistent value (eg true) for all three?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can set all to true.

I was copy-pasting from other calls in this page - I think it would be even more confusing to have watch=1 in some places and watch=true in other places in this doc.

I'm all for unifying it, but let's do that in a separate PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.

## Retrieving large results sets in chunks

{{< feature-state for_k8s_version="v1.9" state="beta" >}}
Expand Down