-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add install instruction, micro synopsis to README (#109)
* Add install instruction, micro synopsis to README Currently non-functional, until the final PyPI names have been decided. Emphasizes the split between the two packages, and mentions which user persona likely needs which package. * Remove server installation, add a usage example * Add contributions and licensing sections Incomplete, since the PR containing the two is not yet merged. * Fix outstanding TODOs with links to material
- Loading branch information
1 parent
39c282a
commit 1a1f92e
Showing
1 changed file
with
91 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,96 @@ | ||
# Jobq: Multi-backend Job definition and Management from Python and the CLI | ||
# jobq: Packaging, scheduling, and monitoring Python workloads as k8s jobs | ||
|
||
Jobq is a compute job definition, workflow management and scheduling tool that supports Kubernetes, Kueue, and Ray backends. | ||
jobq is a tool to define, schedule, and observe compute jobs as Python functions in Kubernetes. | ||
It supports Docker (standard Kubernetes), Kueue, and Ray backends for workload execution. | ||
|
||
The `jobq` Python package itself is split into a client and backend part. | ||
The client package is used to define jobs with the `@job` decorator in your Python files, and (optionally) to interact with an existing API server through a CLI. | ||
The backend contains a FastAPI-based server application that acts as an intermediary between the user and the target Kubernetes cluster, and can be easily packaged into a Docker container or a virtual machine for deployment. | ||
|
||
## Installation | ||
|
||
Depending on what your role is, you will likely require only some parts of the `jobq` package. | ||
|
||
As a data scientist, you only require the client part to define and dispatch jobs. | ||
To install the client package, simply use `pip`: | ||
|
||
```shell | ||
pip install --upgrade "aai-jobq[cli]" | ||
``` | ||
|
||
This instruction includes the command-line interface to interact with an existing Kubernetes cluster. | ||
If that functionality is not needed, simply omit the bracketed extra from the install command. | ||
|
||
## Usage | ||
|
||
Define jobs in your `.py` files by using the `@job` decorator on the function that contains your compute logic. | ||
Within the decorator you define the desired resources and link the environment configuration. We support Dockerfiles or a declarative `.yaml` file specification. | ||
Using the `jobq` CLI you can execute jobs locally or submit them to one of the supported backends. | ||
|
||
```python | ||
import time | ||
|
||
from jobq import job | ||
|
||
@job | ||
def hello_world(): | ||
"""A simple job printing a log message ten times.""" | ||
for idx in range(10): | ||
print(f"Hello, World!, {idx=}") | ||
time.sleep(2) | ||
``` | ||
|
||
Within the decorator, you define the desired resources and link the environment configuration. We support Dockerfiles or a declarative `.yaml` file specification. | ||
|
||
```python | ||
# The same example, but with configuration options for a Kueue admission. | ||
import time | ||
from pathlib import Path | ||
|
||
from jobq import ImageOptions, JobOptions, ResourceOptions, SchedulingOptions, job | ||
|
||
@job( | ||
options=JobOptions( | ||
labels={"type": "hello-world@dev", "x-jobq.io/key": "value"}, | ||
resources=ResourceOptions(memory="1Gi", cpu="1"), | ||
scheduling=SchedulingOptions( | ||
priority_class="background", queue_name="user-queue" | ||
), | ||
), | ||
image=ImageOptions( | ||
spec=Path("examples/example-hello.yaml"), | ||
name="localhost:5000/hello-world-dev", | ||
tag="latest", | ||
), | ||
) | ||
def hello_world(): | ||
"""A simple job printing a log message ten times.""" | ||
for idx in range(10): | ||
print(f"Hello, World!, {idx=}") | ||
time.sleep(2) | ||
``` | ||
|
||
Using the `jobq` CLI, you can execute jobs locally or submit them to one of the supported backends. | ||
|
||
```shell | ||
# Requires the file name in which the job is defined, and the execution mode. | ||
jobq submit --mode=kueue example.py | ||
``` | ||
|
||
Additionally, you can query for a job status and fetch logs. | ||
|
||
Jobq is an early stage project, so expect rough edges. We are happy for feedback and may accomodate your feature requests, so don't hesitate to get in touch. | ||
```shell | ||
# Below is a typical unique identifier of a workload inside a cluster. | ||
jobq logs 4c0d4cbb-13a2-4d05-afde-8784e22bb940 | ||
``` | ||
|
||
jobq is an early stage project, so expect rough edges. We are happy for feedback and may accomodate your feature requests, so don't hesitate to get in touch. | ||
|
||
## Contributions | ||
|
||
If you'd like to contribute to jobq, the best way to do so is to open an [issue](https://github.com/aai-institute/jobq/issues) for bug reports and feature requests, | ||
or start a [discussion](https://github.com/aai-institute/jobq/discussions) for topics around the project, and features that might need prior feedback from the core developers. | ||
|
||
For more information on how to contribute, refer to the [contribution guide](CONTRIBUTING.md). | ||
|
||
## License | ||
|
||
jobq is distributed under the [Apache-2 license](LICENSE). |