diff --git a/keps/168-pending-workloads-visibility/README.md b/keps/168-pending-workloads-visibility/README.md new file mode 100644 index 0000000000..cbc3cea7dd --- /dev/null +++ b/keps/168-pending-workloads-visibility/README.md @@ -0,0 +1,349 @@ +# KEP-168: Pending workloads visibility + + + + + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) + - [Time and memory consuming computation of top pending workloads](#time-and-memory-consuming-computation-of-top-pending-workloads) + - [Too large local queue object](#too-large-local-queue-object) + - [Increased number of localqueue updates](#increased-number-of-localqueue-updates) +- [Design Details](#design-details) + - [Local Queue API](#local-queue-api) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit Tests](#unit-tests) + - [Integration tests](#integration-tests) + - [Graduation Criteria](#graduation-criteria) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + + +## Summary + +The enhancement extends the API of LocalQueue and ClusterQueue to expose the +information about the order of their pending workloads. + +## Motivation + +Currently, there is no visibility to users of the contests of the queue by a +user. +. + + + +### Goals + +- expose the order of workloads in the LocalQueue and ClusterQueue + + + +### Non-Goals + +- expose the information about workload position for each workload individually + + + +## Proposal + +The proposal is to extend the APIs for the status of LocalQueue and ClusterQueue +to expose the order to workloads. The order will be only exposed up to some +configurable depth. + +The approach of keeping the information in the status of +the queues will allow to expose the information without an extra cost of new +API requests, as the statuses are already updated on changes for the number of +pending workloads. + +In order to keep the size of the information constrained only the head of the +queue of pending workloads will be exposed. + + + +### User Stories (Optional) + + + +#### Story 1 + +As a user of Kueue with LocalQueue visibility I would like to know the +position of my workload in the local queue. Knowing the position would allow me +to compute the estimate arrival time (ETA) of my workload. + +I would like to be able to get this information by inspecting the local queue +status. + + + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + +#### Time and memory consuming computation of top pending workloads + +Currently, we organize the cluster queue as a heap, thus only the top of the +heap is readily available. If we want to get to know the top N pending workloads +we may need to copy the heap, sort and select N, which is time and memory +consuming. + +In order to mitigate this risk we may need to migrate from using a heap to +red-black trees. + +#### Too large local queue object + +As the number of pending workloads is arbitrarily large there is a risk that the +status information about the workloads may exceed the etcd limit of 1.5Mi on +object size. In order to allow feature extensions of the structure we +should assume not more that 500Ki is used. + +In order to mitigate this risk we put a constraint on the number of exposed +pending workloads. We limit the number to 1000. + +#### Increased number of localqueue updates + +As we put the global cluster queue position into the list of top pending workloads +changes to one local queue may trigger updates in another local queue connected +to the same cluster queue, due to status changes in the structure. + +In order to mitigate this risk we need to consider: +1. batching updates to local queues, by a batch period, similarly as Job status + updates are batched in Kubernetes. +2. allow to control the number of N workloads which expose their position in the + cluster queue. Setting low value for the limit would reduce the change of + changes in one local queue to trigger an update in another local queue. + + + + +## Design Details + +### Local Queue API + +```golang +// PendingWorkload contains the information identifying a pending workload in +// the local queue. +type PendingWorkload struct { + // Name indicates the name of the pending workload. + Name string + + // Position indicates the position of the workload among all pending + // workloads in the cluster queue. + Position *int32 +} + +type PendingWorkloadsStatus struct { + // TopList contains the list of top pending workloads. + // +listType=map + // +listMapKey=name + // +listMapKey=namespace + // +optional + TopList []PendingWorkload +} + +// LocalQueueStatus defines the observed state of LocalQueue +type LocalQueueStatus struct { +... + // PendingWorkloadsStatus contains the information exposed about the current + // status of the queue of pending workloads. + // +optional + PendingWorkloadsStatus *PendingWorkloadsStatus +... +} +``` + +The returned information is controlled by the global Kueue configuration: + +```golang +// Configuration is the Schema for the kueueconfigurations API +type Configuration struct { +... + // PendingWorkloadsVisibility is configuration to expose the information + // about the top pending workloads in the local queue. + PendingWorkloadsVisibility *PendingWorkloadsVisibility `json:"pendingWorkloadsVisibility,omitempty"` +} + +type PendingWorkloadsVisibility struct { + // MaxPendingWorkloadsInStatus indicates the maximal number of pending + // workloads for which their local queue order is exposed. + // Defaults to 100. + MaxTopPendingWorkloads *int32 +} +``` + + + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +#### Unit Tests + + + + + +- ``: `` - `` + +#### Integration tests + + + +### Graduation Criteria + + + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + diff --git a/keps/168-pending-workloads-visibility/kep.yaml b/keps/168-pending-workloads-visibility/kep.yaml new file mode 100644 index 0000000000..293e239481 --- /dev/null +++ b/keps/168-pending-workloads-visibility/kep.yaml @@ -0,0 +1,20 @@ +title: Pending workloads visibility +kep-number: 168 +authors: + - "@mimowo" +status: provisional +creation-date: 2023-07-14 +reviewers: + - "@ahg-g" + - "@alculquicondor" + - "@mwielgus" +approvers: + - "@ahg-g" + - "@alculquicondor" + +stage: stable + +latest-milestone: "v0.5" +milestone: + stable: "v0.5" +disable-supported: false