Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add simple queue control example to python docs #967

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 62 additions & 1 deletion site/content/en/docs/tasks/run_python_jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ The following examples demonstrate different use cases for using Kueue in Python
This example demonstrates installing Kueue to an existing cluster. You can save this
script to your local machine as `install-kueue-queues.py`.

{{% include "python/sample-job.py" "python" %}}
{{% include "python/install-kueue-queues.py" "python" %}}

And then run as follows:

Expand Down Expand Up @@ -98,3 +98,64 @@ Use:

You can also change the container image with `--image` and args with `--args`.
For more customization, you can edit the example script.

### Interact with Queues and Jobs

If you are developing an application that submits jobs and needs to interact
with and check on them, you likely want to interact with queues or jobs directly.
After running the example above, you can test the following example to interact
with the results. Write the following to a script called `sample-queue-control.py`.

{{% include "python/sample-queue-control.py" "python" %}}

To make the output more interesting, we can run a few random jobs first:

```bash
python sample-job.py
python sample-job.py
python sample-job.py --job-name tacos
```

And then run the script to see your queue and sample job that you submit previously.

```bash
python sample-queue-control.py
```
```console
⛑️ Local Queues
Found queue user-queue
Admitted workloads: 3
Pending workloads: 0
Flavor default-flavor has resources [{'name': 'cpu', 'total': '3'}, {'name': 'memory', 'total': '600Mi'}]

💼️ Jobs
Found job sample-job-8n5sb
Succeeded: 3
Ready: 0
Found job sample-job-gnxtl
Succeeded: 1
Ready: 0
Found job tacos46bqw
Succeeded: 1
Ready: 1
```

If you wanted to filter jobs to a specific queue, you can do this via the job labels
under `job["metadata"]["labels"]["kueue.x-k8s.io/queue-name"]'. To list a specific job by
name, you can do:

```python
from kubernetes import client, config

# Interact with batch
config.load_kube_config()
batch_api = client.BatchV1Api()

# This is providing the name, and namespace
job = batch_api.read_namespaced_job("tacos46bqw", "default")
print(job)
```

See the [BatchV1](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/BatchV1Api.md)
API documentation for more calls.

20 changes: 16 additions & 4 deletions site/static/examples/python/install-kueue-queues.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import argparse

# install-kueue-queues.py will:
# 1. install queue from the latest on GitHub
# 1. install queue from the latest or a specific version on GitHub
# This example will demonstrate installing Kueue and applying a YAML file (local) to install Kueue

# Make sure your cluster is running!
Expand All @@ -22,8 +22,8 @@ def get_parser():
)
parser.add_argument(
"--version",
help="Version of Kueue to install",
default="0.4.0",
help="Version of Kueue to install (if undefined, will install from master branch)",
default=None,
)
return parser

Expand All @@ -39,12 +39,24 @@ def main():
install_kueue(args.version)


def get_install_url(version):
"""
Get the install version.

If a version is specified, use it. Otherwise install
from the main branch.
"""
if version is not None:
return f"https://github.com/kubernetes-sigs/kueue/releases/download/v{version}/manifests.yaml"
return "https://github.com/kubernetes-sigs/kueue/config/default?ref=main"


def install_kueue(version):
"""
Install Kueue of a particular version.
"""
print("⭐️ Installing Kueue...")
url = f"https://github.com/kubernetes-sigs/kueue/releases/download/v{version}/manifests.yaml"
url = get_install_url(version)
with tempfile.NamedTemporaryFile(delete=True) as install_yaml:
res = requests.get(url)
assert res.status_code == 200
Expand Down
94 changes: 94 additions & 0 deletions site/static/examples/python/sample-queue-control.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
#!/usr/bin/env python3

import argparse
from kubernetes import config, client

# sample-queue-control.py
# This will show how to interact with queues

# Make sure your cluster is running!
config.load_kube_config()
crd_api = client.CustomObjectsApi()
api_client = crd_api.api_client


def get_parser():
parser = argparse.ArgumentParser(
description="Interact with Queues e",
formatter_class=argparse.RawTextHelpFormatter,
)
parser.add_argument(
"--namespace",
help="namespace to list for",
default="default",
)
return parser


def main():
"""
Get a listing of jobs in the queue
"""
parser = get_parser()
args, _ = parser.parse_known_args()

listing = crd_api.list_namespaced_custom_object(
group="kueue.x-k8s.io",
version="v1beta1",
namespace=args.namespace,
plural="localqueues",
)
list_queues(listing)

listing = crd_api.list_namespaced_custom_object(
group="batch",
version="v1",
namespace=args.namespace,
plural="jobs",
)
list_jobs(listing)


def list_jobs(listing):
"""
Iterate and show job metadata.
"""
if not listing:
print("💼️ There are no jobs.")
return

print("\n💼️ Jobs")
for job in listing["items"]:
jobname = job["metadata"]["name"]
status = (
"TBA" if "succeeded" not in job["status"] else job["status"]["succeeded"]
)
ready = job["status"]["ready"]
print(f"Found job {jobname}")
print(f" Succeeded: {status}")
print(f" Ready: {ready}")


def list_queues(listing):
"""
Helper function to iterate over and list queues.
"""
if not listing:
print("⛑️ There are no queues.")
return

print("\n⛑️ Local Queues")

# This is listing queues
for q in listing["items"]:
print(f'Found queue {q["metadata"]["name"]}')
print(f" Admitted workloads: {q['status']['admittedWorkloads']}")
print(f" Pending workloads: {q['status']['pendingWorkloads']}")

# And flavors with resources
for f in q["status"]["flavorUsage"]:
print(f' Flavor {f["name"]} has resources {f["resources"]}')


if __name__ == "__main__":
main()