From 3f937bc8e740cf73205653931f79649548402e16 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Wed, 7 Feb 2024 15:27:18 +0000 Subject: [PATCH 01/54] Write a DEP for background workers --- draft/0000-background-workers.rst | 299 ++++++++++++++++++++++++++++++ 1 file changed, 299 insertions(+) create mode 100644 draft/0000-background-workers.rst diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst new file mode 100644 index 00000000..ea17b838 --- /dev/null +++ b/draft/0000-background-workers.rst @@ -0,0 +1,299 @@ +============================= +DEP XXXX: Background workers +============================= + +:DEP: XXXX +:Author: Jake Howard +:Implementation Team: Jake Howard +:Shepherd: Carlton Gibson +:Status: Draft +:Type: Feature +:Created: 2024-02-07 +:Last-Modified: 2024-02-09 + +.. contents:: Table of Contents + :depth: 3 + :local: + +Abstract +======== + +Django doesn't have a first-party solution for long-running tasks, however the ecosystem is filled with incredibly popular frameworks, all of which interact with Django in slightly different ways. Other frameworks such as Laravel have background workers built-in, allowing them to push tasks into the background to be processed at a later date, without requiring the end user to wait for them to occur. + +Library maintainers must implement support for any possible task backend separately, should they wish to offload functionality to the background. This includes smaller libraries, but also larger meta-frameworks with their own package ecosystem such as `Wagtail `_. + +Specification +============= + +The proposed implementation will be in the form of an application wide "task backend" interface. This backend will be what connects Django to the task runners with a single pattern. The task backend will provide an interface for either third-party libraries, or application developers to specify how tasks should be created and pushed into the background + +Backends +-------- + +A backend will be a class which extends a Django-defined base class, and provides the common interface between Django and the underlying task runner. + +.. code:: python + + from datetime import datetime + from typing import Callable, Dict, List + + from django.contrib.tasks import BaseTask + from django.contrib.tasks.backends.base import BaseTaskBackend + + + class MyBackend(BaseTaskbackend): + def __init__(self, options: Dict): + """ + Any connections which need to be setup can be done here + """ + super().__init__(options) + + def enqueue(self, func: Callable, priority: int | None, args: List, kwargs: Dict) -> BaseTask: + """ + Queue up a task function to be executed + """ + ... + + def defer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: + """ + Add a task to be completed at a specific time + """ + ... + + async def aenqueue(self, func: Callable, priority: int | None, args: List, kwargs: Dict) -> BaseTask: + """ + Queue up a task function (or coroutine) to be executed + """ + ... + + async def adefer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: + """ + Add a task function (or coroutine) to be completed at a specific time + """ + ... + + def close(self) -> None: + """ + Close any connections opened as part of the constructor + """ + ... + +If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_async`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. + +Django will ship with 2 implementations: + +ImmediateBackend + This backend runs the tasks immediately, rather than offloading to a background process. This is useful both for a graceful transition towards background workers, but without impacting existing functionality. + +DatabaseBackend + This backend uses the Django ORM as a task store. This backend will support all features, and should be considered production-grade. + +Tasks +----- + +A ``Task`` is used as a handle to the running task, and contains useful information the application may need when referencing the task. + +.. code:: python + + from datetime import datetime + from typing import Any + + from django.contrib.tasks import BaseTask, TaskStatus + + class MyBackendTask(BaseTask): + id: str + """A unique identifier for the task""" + + status: TaskStatus + """The status of the task""" + + result: Any + """The return value from the task function""" + + queued_at: datetime + """When the task was added to the queue""" + + completed_at: datetime | None + """When the task was completed""" + + raw: Any | None + """Return the underlying runner's task handle""" + + def __init__(self, **kwargs): + """ + Unpacking the raw response from the backend and storing it here for future use + """ + super().__init__(**kwargs) + + def refresh(self) -> None: + """ + Reload the cached task data from the task store + """ + ... + + async def arefresh(self) -> None: + """ + Reload the cached task data from the task store + """ + ... + + +A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``arefresh`` to reload the values from the task store. + +To enable a ``Task`` to be backend-agnostic, statuses must include a set of known values. Additional fields may be added if the backend supports them, but these attributes must be supported: + +:New: The task has been created, but hasn't started running yet +:Running: The task is currently running +:Failed: The task failed +:Complete: The task is complete, and the result is accessible + +Running tasks +------------- + +Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on the default task backend: + +.. code:: python + + from django.contrib.tasks import enqueue + + def do_a_task(*args, **kwargs): + pass + + # Submit the task function to be run + task = enqueue(do_a_task) + + # Optionally, provide arguments + task = enqueue(do_a_task, args=[], kwargs={})] + +Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. When multiple task backends are configured, each can be obtained from a global ``tasks`` connection handler: + +.. code:: python + + from django.contrib.tasks import tasks + + def do_a_task(*args, **kwargs): + pass + + # Submit the task function to be run + task = tasks["special"].enqueue(do_a_task) + + # Optionally, provide arguments + task = tasks["special"].enqueue(do_a_task, args=[], kwargs={}) + +When enqueueing tasks, ``args`` and ``kwargs`` are intentionally their own dedicated arguments to make the API simpler and backwards-compatible should other attributes be added in future. + +Here, ``do_a_task`` can either be a regular function or coroutine. It will be up to the backend implementor to determine whether coroutines are supported. + +Deferring tasks +--------------- + +Tasks may also be "deferred" to run at a specific time in the future: + +.. code:: python + + from django.utils import timezone + from datetime import timedelta + from django.contrib.tasks import defer + + task = defer(do_a_task, when=timezone.now() + timedelta(minutes=5)) + +When scheduling a task, it may not be **exactly** that time a task is executed, however it should be accurate to within a few seconds. This will depend on the current state of the queue and task runners, and is out of the control of Django. + +Sending emails +-------------- + +One of the easiest and most common places that offloading work to the background can be performed is sending emails. Sending an email requires communicating with an external, potentially third-party service, which adds additional latency and risk to web requests. These can be easily offloaded to the background. + +Django will ship with an additional task-based SMTP email backend, configured identically to the existing SMTP backend. The other backends included with Django don't benefit from being moved to the background. + +Async tasks +----------- + +Where the underlying task runner supports it, backends may also provide an ``async``-compatible interface for task running, using ``a``-prefixed methods: + +.. code:: python + + from django.contrib.tasks import aenqueue + + await aenqueue(do_a_task) + +Settings +--------- + +.. code:: python + + TASKS = { + "default": { + "BACKEND": "django.contrib.tasks.backends.ImmediateBackend", + "OPTIONS": {} + } + } + +``OPTIONS`` is passed as-is to the backend's constructor. + +Motivation +========== + +Having a first-party interface for background workers poses 2 main benefits: + +Firstly, it lowers the barrier to entry for offloading computation to the background. Currently, a user needs to research different worker technologies, follow their integration tutorial, and modify how their tasks are called. Instead, a developer simply needs to install the dependencies, and work out how to *run* the background worker. Similarly, a developer can start determining which actions should run in the background before implementing a true background worker, and avoid refactoring should the backend change over time. + +Secondly, it allows third-party libraries to offload some of their execution. Currently, library maintainers need to either accept their code will run inside the request-response lifecycle, or provide hooks for application developers to offload actions themselves. This can be particularly helpful when offloading certain expensive signals. + +One of the key benefits behind background workers is removing the requirement for the user to wait for tasks they don't need to, moving computation and complexity out of the request-response cycle, towards dedicated background worker processes. Moving certain actions to be run in the background not improves performance of web requests, but also allows those actions to run on specialised hardware, potentially scaled differently to the web servers. This presents an opportunity to greatly decrease the percieved execution time of certain common actions performed by Django projects. + +But what about *X*? +------------------- + +The most obvious alternative to this DEP would be to standardise on a task implementation and vendor it in to Django. The Django ecosystem is already full of background worker libraries, eg Celery and RQ. Writing a production-ready task runner is a complex and nuanced undertaking, and discarding the work already done is a waste. + +This proposal doesn't seek to replace existing tools, nor add yet another option for developers to consider. The primary motivation is creating a shared API contract between worker libaries and developers. It does however provide a simple way to get started, with a solution suitable for most sizes of projects (``DatabaseBackend``). + +Rationale +========= + +This proposed implementation specifically doesn't assume anything about the user's setup. This not only reduces the chances of Django conflicting with existing task systems a user may be using (eg Celery, RQ), but also allows it to work with almost any hosting environment a user might be using. + +This proposal started out as `Wagtail RFC 72 `_, as it was becoming clear a unified interface for background tasks was required, without imposing on a developer's decisions for how the tasks are executed. Wagtail is run in many different forms at many differnt scales, so it needed to be possible to allow developers to choose the backend they're comfortable with, in a way which Wagtail and its associated packages can execute tasks without assuming anything of the environment it's running in. + +The global task connection ``tasks`` is used to access the configured backends, with global versions of those methods available for the default backend. This contradicts the pattern already used for storage and caches. A "task" is already used in a number of places to refer to an executed task, so using it to refer to the default backend is confusing and may lead to it being overridden in the current scope: + +.. code:: python + + from django.contrib.tasks import task + + # Later... + task = task.enqueue(do_a_thing) + + # Clearer + thing_task = task.enqueue(do_a_thing) + +Backwards Compatibility +======================= + +So that library maintainers can use this integration without concern as to whether a Django project has configured background workers, the default configuration will use the ``ImmediateBackend``. Developers on older versions of Django but who need libraries which assume tasks are available can use the reference implementation. + +Reference Implementation +======================== + +The reference implementation is currently being developed alongside this DEP process. This implementation will serve both as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django core, but also as a backport for users of supported Django versions prior to this work being released. + +Once code is available, it will be referenced here. + +Future iterations +================= + +The field of background tasks is vast, and attempting to implement everything supported by existing tools in the first iteration is futile. The following functionality has been considered, and deemed explicitly out of scope of the first pass, but still worthy of future development: + +- Completion hooks, to run subsequent tasks automatically +- Bulk queueing +- Automated task retrying +- A generic way of executing task runners. This will remain the repsonsibility of the underlying implementation, and the user to execute correctly. +- Observability into task queues, including monitoring and reporting +- Cron-based scheduling. For now, this can be achieved by triggering the same task once it finishes. + +Copyright +========= + +This document has been placed in the public domain per the Creative Commons +CC0 1.0 Universal license (http://creativecommons.org/publicdomain/zero/1.0/deed). From c5d836c8505d81e903fb17f4ab2b92ff206e08aa Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 9 Feb 2024 15:12:12 +0000 Subject: [PATCH 02/54] Move tasks from contrib to `django` root --- draft/0000-background-workers.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index ea17b838..3a57f0d3 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -37,8 +37,8 @@ A backend will be a class which extends a Django-defined base class, and provide from datetime import datetime from typing import Callable, Dict, List - from django.contrib.tasks import BaseTask - from django.contrib.tasks.backends.base import BaseTaskBackend + from django.tasks import BaseTask + from django.tasks.backends.base import BaseTaskBackend class MyBackend(BaseTaskbackend): @@ -98,7 +98,7 @@ A ``Task`` is used as a handle to the running task, and contains useful informat from datetime import datetime from typing import Any - from django.contrib.tasks import BaseTask, TaskStatus + from django.tasks import BaseTask, TaskStatus class MyBackendTask(BaseTask): id: str @@ -154,7 +154,7 @@ Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on .. code:: python - from django.contrib.tasks import enqueue + from django.tasks import enqueue def do_a_task(*args, **kwargs): pass @@ -169,7 +169,7 @@ Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. W .. code:: python - from django.contrib.tasks import tasks + from django.tasks import tasks def do_a_task(*args, **kwargs): pass @@ -193,7 +193,7 @@ Tasks may also be "deferred" to run at a specific time in the future: from django.utils import timezone from datetime import timedelta - from django.contrib.tasks import defer + from django.tasks import defer task = defer(do_a_task, when=timezone.now() + timedelta(minutes=5)) @@ -213,7 +213,7 @@ Where the underlying task runner supports it, backends may also provide an ``asy .. code:: python - from django.contrib.tasks import aenqueue + from django.tasks import aenqueue await aenqueue(do_a_task) @@ -224,7 +224,7 @@ Settings TASKS = { "default": { - "BACKEND": "django.contrib.tasks.backends.ImmediateBackend", + "BACKEND": "django.tasks.backends.ImmediateBackend", "OPTIONS": {} } } @@ -260,7 +260,7 @@ The global task connection ``tasks`` is used to access the configured backends, .. code:: python - from django.contrib.tasks import task + from django.tasks import task # Later... task = task.enqueue(do_a_thing) From 96305e84389c4f914451899f3af5ee097dc21361 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 9 Feb 2024 16:01:50 +0000 Subject: [PATCH 03/54] Provide default async implementations --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 3a57f0d3..5b52133d 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -78,7 +78,7 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... -If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_async`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. +If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide stubs which expose ``enqueue`` and ``defer`` wrapped with ``asgiref.sync_to_async``. Django will ship with 2 implementations: From c9c76d9397f936bde804f8dabdbbfe426de05450 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 9 Feb 2024 16:06:55 +0000 Subject: [PATCH 04/54] Explicitly mention the potential for future development --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 5b52133d..3d1629b3 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -247,7 +247,7 @@ But what about *X*? The most obvious alternative to this DEP would be to standardise on a task implementation and vendor it in to Django. The Django ecosystem is already full of background worker libraries, eg Celery and RQ. Writing a production-ready task runner is a complex and nuanced undertaking, and discarding the work already done is a waste. -This proposal doesn't seek to replace existing tools, nor add yet another option for developers to consider. The primary motivation is creating a shared API contract between worker libaries and developers. It does however provide a simple way to get started, with a solution suitable for most sizes of projects (``DatabaseBackend``). +This proposal doesn't seek to replace existing tools, nor add yet another option for developers to consider. The primary motivation is creating a shared API contract between worker libaries and developers. It does however provide a simple way to get started, with a solution suitable for most sizes of projects (``DatabaseBackend``). Slowly increasing features, adding more built-in storage backends and a first-party task runner aren't out of the question for the future, but must be done with careful planning and consideration. Rationale ========= From 815ac4919f291a8c5bd21689fcaaf47e957297ad Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 9 Feb 2024 16:15:07 +0000 Subject: [PATCH 05/54] Link out to example code --- draft/0000-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 3d1629b3..e18a9012 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -276,9 +276,9 @@ So that library maintainers can use this integration without concern as to wheth Reference Implementation ======================== -The reference implementation is currently being developed alongside this DEP process. This implementation will serve both as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django core, but also as a backport for users of supported Django versions prior to this work being released. +The reference implementation will be developed alongside this DEP process. This implementation will serve both as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django core, but also as a backport for users of supported Django versions prior to this work being released. -Once code is available, it will be referenced here. +A more complete implementation picture can be found at https://github.com/RealOrangeOne/django-core-tasks, however it should not be considered final. Future iterations ================= From bc8170bf9e2daec60ea6474722640e94dad8ba5b Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 9 Feb 2024 16:40:10 +0000 Subject: [PATCH 06/54] Add the ability to get an existing task --- draft/0000-background-workers.rst | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index e18a9012..12656462 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -72,6 +72,18 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... + def get_task(self, task_id: str) -> BaseTask: + """ + Retrieve a task by its id (if one exists) + """ + ... + + async def aget_task(self, task_id: str) -> BaseTask: + """ + Retrieve a task by its id (if one exists) + """ + ... + def close(self) -> None: """ Close any connections opened as part of the constructor @@ -137,6 +149,7 @@ A ``Task`` is used as a handle to the running task, and contains useful informat """ ... +A ``Task`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``arefresh`` to reload the values from the task store. From f0c6b3ae236628f591074a1fa239cd751d996272 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 12 Feb 2024 09:36:43 +0000 Subject: [PATCH 07/54] Fix typos --- draft/0000-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 12656462..5cc42a13 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -25,7 +25,7 @@ Library maintainers must implement support for any possible task backend separat Specification ============= -The proposed implementation will be in the form of an application wide "task backend" interface. This backend will be what connects Django to the task runners with a single pattern. The task backend will provide an interface for either third-party libraries, or application developers to specify how tasks should be created and pushed into the background +The proposed implementation will be in the form of an application wide "task backend" interface. This backend will be what connects Django to the task runners with a single pattern. The task backend will provide an interface for either third-party libraries, or application developers to specify how tasks should be created and pushed into the background. Backends -------- @@ -176,7 +176,7 @@ Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on task = enqueue(do_a_task) # Optionally, provide arguments - task = enqueue(do_a_task, args=[], kwargs={})] + task = enqueue(do_a_task, args=[], kwargs={}) Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. When multiple task backends are configured, each can be obtained from a global ``tasks`` connection handler: From 018af5c5d2f5b3edc4440ddc2dc5f6566cb2b569 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 12 Feb 2024 09:38:21 +0000 Subject: [PATCH 08/54] Remove suggestion for how to implement cron-based scheduling This suffered from a bootstrapping problem. Instead, people can use whatever techniques they're using at the moment, and task-ify them as needed. --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 5cc42a13..68840e84 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -303,7 +303,7 @@ The field of background tasks is vast, and attempting to implement everything su - Automated task retrying - A generic way of executing task runners. This will remain the repsonsibility of the underlying implementation, and the user to execute correctly. - Observability into task queues, including monitoring and reporting -- Cron-based scheduling. For now, this can be achieved by triggering the same task once it finishes. +- Cron-based scheduling Copyright ========= From d9a10c7a6e8349d2d69aa8a8290c0d3b3ad3f3ed Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 12 Feb 2024 09:41:18 +0000 Subject: [PATCH 09/54] Add thought process behind DatabaseBackend to DEP --- draft/0000-background-workers.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 68840e84..85f8195b 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -255,6 +255,8 @@ Secondly, it allows third-party libraries to offload some of their execution. Cu One of the key benefits behind background workers is removing the requirement for the user to wait for tasks they don't need to, moving computation and complexity out of the request-response cycle, towards dedicated background worker processes. Moving certain actions to be run in the background not improves performance of web requests, but also allows those actions to run on specialised hardware, potentially scaled differently to the web servers. This presents an opportunity to greatly decrease the percieved execution time of certain common actions performed by Django projects. +The target audience for ``DatabaseBackend`` and a SQL-based queue are likely fairly well aligned with those who may choose something like PostgreSQL FTS over something like ElasticSearch. ElasticSearch is probably better for those 10% of users who really need it, but doesn't mean the other 90% won't be perfectly happy with PostgreSQL, and probably wouldn't benefit from ElasticSearch anyway. + But what about *X*? ------------------- From 28b5b84a53562b7e91650f88e6afe52955b92779 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 13 Feb 2024 12:53:39 +0000 Subject: [PATCH 10/54] Ensure a task keeps track of the parameters of the task when enqueued. --- draft/0000-background-workers.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 85f8195b..25e03a21 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -108,7 +108,7 @@ A ``Task`` is used as a handle to the running task, and contains useful informat .. code:: python from datetime import datetime - from typing import Any + from typing import Any, Callable from django.tasks import BaseTask, TaskStatus @@ -131,6 +131,18 @@ A ``Task`` is used as a handle to the running task, and contains useful informat raw: Any | None """Return the underlying runner's task handle""" + priority: int | None + """The priority of the task""" + + func: Callable + """The task function""" + + args: list + """The arguments to pass to the task function""" + + kwargs: dict + """The keyword arguments to pass to the task function""" + def __init__(self, **kwargs): """ Unpacking the raw response from the backend and storing it here for future use From a21d00fa8cced8d373e58a5ef3b76022e8720e57 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 13 Feb 2024 12:55:12 +0000 Subject: [PATCH 11/54] Add async stub for `get_task` --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 25e03a21..402d5c5e 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -90,7 +90,7 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... -If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide stubs which expose ``enqueue`` and ``defer`` wrapped with ``asgiref.sync_to_async``. +If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue``, ``defer`` and ``get_task`` wrapped with ``asgiref.sync_to_async``. Django will ship with 2 implementations: From c24c2a335565d9e53475e9d4a72fea97f2771c0c Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 13 Feb 2024 12:56:08 +0000 Subject: [PATCH 12/54] Use a generic exception for tasks not existing --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 402d5c5e..fb565093 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -161,7 +161,7 @@ A ``Task`` is used as a handle to the running task, and contains useful informat """ ... -A ``Task`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. +A ``Task`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. If called with a ``task_id`` which doesn't exist, a ``TaskDoesNotExist`` exception is raised. A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``arefresh`` to reload the values from the task store. From e6ebec7e79ffffa05cecee54d98601ab598c7297 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 13 Feb 2024 12:57:19 +0000 Subject: [PATCH 13/54] Note that task functions must be globally importable --- draft/0000-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index fb565093..b2a99ddb 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -172,7 +172,7 @@ To enable a ``Task`` to be backend-agnostic, statuses must include a set of know :Failed: The task failed :Complete: The task is complete, and the result is accessible -Running tasks +Queueing tasks ------------- Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on the default task backend: @@ -207,7 +207,7 @@ Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. W When enqueueing tasks, ``args`` and ``kwargs`` are intentionally their own dedicated arguments to make the API simpler and backwards-compatible should other attributes be added in future. -Here, ``do_a_task`` can either be a regular function or coroutine. It will be up to the backend implementor to determine whether coroutines are supported. +Here, ``do_a_task`` can either be a regular function or coroutine. It will be up to the backend implementor to determine whether coroutines are supported. In either case, the function must be globally importable. Deferring tasks --------------- From 7b89e7ba98d3c54a23555030259d3e8d1b80ad7d Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 13 Feb 2024 12:58:27 +0000 Subject: [PATCH 14/54] Queueing an async task will not execute it immediately --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index b2a99ddb..0522a0aa 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -234,7 +234,7 @@ Django will ship with an additional task-based SMTP email backend, configured id Async tasks ----------- -Where the underlying task runner supports it, backends may also provide an ``async``-compatible interface for task running, using ``a``-prefixed methods: +Where the underlying task runner supports it, backends may also provide an ``async``-compatible interface for task queueing, using ``a``-prefixed methods: .. code:: python From fd679ed37c3ab8f2ccf84acfb1bdc67aac811f9b Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 13 Feb 2024 12:59:40 +0000 Subject: [PATCH 15/54] Add a `DummyBackend` --- draft/0000-background-workers.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 0522a0aa..1db4c3d5 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -100,6 +100,9 @@ ImmediateBackend DatabaseBackend This backend uses the Django ORM as a task store. This backend will support all features, and should be considered production-grade. +DummyBackend + This backend doesn't execute tasks at all, and instead stores the ``Task`` objects in memory. This backend is mostly useful in tests. + Tasks ----- From 4bb8f2371a5f2b22459bf7070edf6a7cf773a676 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 16:58:19 +0000 Subject: [PATCH 16/54] Correct number of built-in implementations --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 1db4c3d5..69bb148f 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -92,7 +92,7 @@ A backend will be a class which extends a Django-defined base class, and provide If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue``, ``defer`` and ``get_task`` wrapped with ``asgiref.sync_to_async``. -Django will ship with 2 implementations: +Django will ship with 3 implementations: ImmediateBackend This backend runs the tasks immediately, rather than offloading to a background process. This is useful both for a graceful transition towards background workers, but without impacting existing functionality. From a58f0c2edc3bd5576c532b5eeae810c6c2334710 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 16:59:20 +0000 Subject: [PATCH 17/54] Explicitly note that an exception is raised if a task doesn't exist --- draft/0000-background-workers.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 69bb148f..2a7f7872 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -74,13 +74,15 @@ A backend will be a class which extends a Django-defined base class, and provide def get_task(self, task_id: str) -> BaseTask: """ - Retrieve a task by its id (if one exists) + Retrieve a task by its id (if one exists). + If one doesn't, raises self.TaskDoesNotExist. """ ... async def aget_task(self, task_id: str) -> BaseTask: """ - Retrieve a task by its id (if one exists) + Retrieve a task by its id (if one exists). + If one doesn't, raises self.TaskDoesNotExist. """ ... From 9fb9e344dd664e2ad5e279124004bd9fe7621b8e Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:07:04 +0000 Subject: [PATCH 18/54] Upper-case task statuses so it's clearer they're enum values --- draft/0000-background-workers.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 2a7f7872..702e1b25 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -172,10 +172,10 @@ A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``ar To enable a ``Task`` to be backend-agnostic, statuses must include a set of known values. Additional fields may be added if the backend supports them, but these attributes must be supported: -:New: The task has been created, but hasn't started running yet -:Running: The task is currently running -:Failed: The task failed -:Complete: The task is complete, and the result is accessible +:NEW: The task has been created, but hasn't started running yet +:RUNNING: The task is currently running +:FAILED: The task failed +:COMPLETE: The task is complete, and the result is accessible Queueing tasks ------------- From 16632772f234f073bf045a31b7e68aa16e5b07c3 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:07:27 +0000 Subject: [PATCH 19/54] Fix typo --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 702e1b25..afeaba5f 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -320,7 +320,7 @@ The field of background tasks is vast, and attempting to implement everything su - Completion hooks, to run subsequent tasks automatically - Bulk queueing - Automated task retrying -- A generic way of executing task runners. This will remain the repsonsibility of the underlying implementation, and the user to execute correctly. +- A generic way of executing task runners. This will remain the responsibility of the underlying implementation, and the user to execute correctly. - Observability into task queues, including monitoring and reporting - Cron-based scheduling From df9cf1f3d69cb38f9c4d672f9edac457e08376ee Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:08:06 +0000 Subject: [PATCH 20/54] Note that `result` may be None if the task hasn't completed --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index afeaba5f..fb556b5f 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -124,7 +124,7 @@ A ``Task`` is used as a handle to the running task, and contains useful informat status: TaskStatus """The status of the task""" - result: Any + result: Any | None """The return value from the task function""" queued_at: datetime From ffefb6ca0007e6e7cd3eaa103f59d3dd554253bb Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:17:46 +0000 Subject: [PATCH 21/54] Allow sync APIs to execute coroutines --- draft/0000-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index fb556b5f..ce1e5795 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -50,13 +50,13 @@ A backend will be a class which extends a Django-defined base class, and provide def enqueue(self, func: Callable, priority: int | None, args: List, kwargs: Dict) -> BaseTask: """ - Queue up a task function to be executed + Queue up a task function (or coroutine) to be executed """ ... def defer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: """ - Add a task to be completed at a specific time + Add a task function (or coroutine) to be completed at a specific time """ ... From 961d7c7e276e267d3fc76f305cdce6be795ad6b4 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:21:05 +0000 Subject: [PATCH 22/54] Add a method for validating a task function --- draft/0000-background-workers.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index ce1e5795..12a7e23f 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -48,6 +48,12 @@ A backend will be a class which extends a Django-defined base class, and provide """ super().__init__(options) + def is_valid_task_function(self, func: Callable) -> bool: + """ + Determine whether the provided callable is valid as a task function. + """ + ... + def enqueue(self, func: Callable, priority: int | None, args: List, kwargs: Dict) -> BaseTask: """ Queue up a task function (or coroutine) to be executed @@ -94,6 +100,8 @@ A backend will be a class which extends a Django-defined base class, and provide If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue``, ``defer`` and ``get_task`` wrapped with ``asgiref.sync_to_async``. +``is_valid_task_function`` determines whether the provided function (or possibly coroutine) is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. The default implementation will ensure the callable is globally importable. + Django will ship with 3 implementations: ImmediateBackend From c289041ad3ad5456ad3cfe935e17923237ede9d6 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:30:06 +0000 Subject: [PATCH 23/54] Don't allow different task statuses This makes casting back to the defined enum safer. Raw statuses can be obtained from the `.raw` property --- draft/0000-background-workers.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 12a7e23f..e0f9ca8d 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -178,13 +178,15 @@ A ``Task`` is obtained either when scheduling a task function, or by calling ``g A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``arefresh`` to reload the values from the task store. -To enable a ``Task`` to be backend-agnostic, statuses must include a set of known values. Additional fields may be added if the backend supports them, but these attributes must be supported: +A ``Task``'s ``status`` must be one of the follwing values (as defined by an ``enum``): :NEW: The task has been created, but hasn't started running yet :RUNNING: The task is currently running :FAILED: The task failed :COMPLETE: The task is complete, and the result is accessible +If a backend supports more than these statuses, it should compress them into one of these. + Queueing tasks ------------- From 70893bb0a30af750eedc167e0575e06337055663 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:40:56 +0000 Subject: [PATCH 24/54] Explicitly allow async task functions These can be queued either in a sync or async context --- draft/0000-background-workers.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index e0f9ca8d..5abb00de 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -257,6 +257,21 @@ Where the underlying task runner supports it, backends may also provide an ``asy await aenqueue(do_a_task) +Similarly, a backend may support queueing an async task function: + +.. code:: python + + from django.tasks import aenqueue, enqueue, task + + @task + async def do_an_async_task(): + pass + + await aenqueue(do_an_async_task) + + # Also works + enqueue(do_an_async_task) + Settings --------- From 39bd0c3633c69792ab2d26b6edcb41af6db2411b Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 26 Feb 2024 17:41:13 +0000 Subject: [PATCH 25/54] Require task functions be marked --- draft/0000-background-workers.rst | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 5abb00de..141ae15d 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -100,7 +100,7 @@ A backend will be a class which extends a Django-defined base class, and provide If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue``, ``defer`` and ``get_task`` wrapped with ``asgiref.sync_to_async``. -``is_valid_task_function`` determines whether the provided function (or possibly coroutine) is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. The default implementation will ensure the callable is globally importable. +``is_valid_task_function`` determines whether the provided function (or possibly coroutine) is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. Django will ship with 3 implementations: @@ -187,6 +187,28 @@ A ``Task``'s ``status`` must be one of the follwing values (as defined by an ``e If a backend supports more than these statuses, it should compress them into one of these. +Task functions +-------------- + +A task function is any globally-importable callable which can be used as the function for a task (ie passed into ``enqueue``). + +Before a task can be run, it must be marked: + +.. code:: python + + from django.tasks import task + + @task + def do_a_task(*args, **kwargs): + pass + +The decorator "marks" the task as being a valid function to be executed. This prevent arbitrary methods from being queued, potentially resulting in a security vulnerability (eg ``subprocess.run``). + +Tasks will be validated against the backend's ``is_valid_task_function`` before queueing. The default implementation will validate all generic assumptions: + +- Is the task function globally importable +- Has the task function been marked + Queueing tasks ------------- @@ -194,8 +216,9 @@ Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on .. code:: python - from django.tasks import enqueue + from django.tasks import enqueue, task + @task def do_a_task(*args, **kwargs): pass @@ -209,8 +232,9 @@ Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. W .. code:: python - from django.tasks import tasks + from django.tasks import tasks, task + @task def do_a_task(*args, **kwargs): pass From 0ea25025fb54d550b3eae535884d32ce5ce0f1cf Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 8 Mar 2024 09:54:58 +0000 Subject: [PATCH 26/54] Be explicit that `when` should be timezone-aware --- draft/0000-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 141ae15d..bbb859b0 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -62,7 +62,7 @@ A backend will be a class which extends a Django-defined base class, and provide def defer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: """ - Add a task function (or coroutine) to be completed at a specific time + Add a task function (or coroutine) to be completed at a specific (timezone-aware) time """ ... @@ -74,7 +74,7 @@ A backend will be a class which extends a Django-defined base class, and provide async def adefer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: """ - Add a task function (or coroutine) to be completed at a specific time + Add a task function (or coroutine) to be completed at a specific (timezone-aware) time """ ... From e9ffd7d8e5a4f4a94663551ae849b0eeed2be56b Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 8 Mar 2024 09:57:23 +0000 Subject: [PATCH 27/54] Note that the result will raise a value error if the task hasn't completed. --- draft/0000-background-workers.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index bbb859b0..8f927a66 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -132,9 +132,6 @@ A ``Task`` is used as a handle to the running task, and contains useful informat status: TaskStatus """The status of the task""" - result: Any | None - """The return value from the task function""" - queued_at: datetime """When the task was added to the queue""" @@ -174,6 +171,15 @@ A ``Task`` is used as a handle to the running task, and contains useful informat """ ... + @property + def result(self) -> Any: + """ + The return value from the task function. + If the task raised an exception, the result will contain that exception. + If the task has not completed, a `ValueError` is raised when accessing. + """ + ... + A ``Task`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. If called with a ``task_id`` which doesn't exist, a ``TaskDoesNotExist`` exception is raised. A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``arefresh`` to reload the values from the task store. From 7efb22cf6ab9d7de2d110ec170244c72acf10d54 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 8 Mar 2024 09:59:14 +0000 Subject: [PATCH 28/54] Remove the `raw` field Nothing is added by documenting this explicitly. People can rely on introspecting the `Task` if they need internal values --- draft/0000-background-workers.rst | 3 --- 1 file changed, 3 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 8f927a66..19d3a5b4 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -138,9 +138,6 @@ A ``Task`` is used as a handle to the running task, and contains useful informat completed_at: datetime | None """When the task was completed""" - raw: Any | None - """Return the underlying runner's task handle""" - priority: int | None """The priority of the task""" From 3da595fc67ae646634d4b72da3aae269d0a9638e Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 28 Mar 2024 16:07:36 +0000 Subject: [PATCH 29/54] Simplify exposed API and allow for more customisation A "Task" becomes a defined item, with metadata hanging off it for more customisation. Also unify the `defer` and `enqueue` APIs so only a single method needs to be used. This also makes naming simpler. --- draft/0000-background-workers.rst | 198 ++++++++++++++---------------- 1 file changed, 95 insertions(+), 103 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 19d3a5b4..a1446d17 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -37,58 +37,36 @@ A backend will be a class which extends a Django-defined base class, and provide from datetime import datetime from typing import Callable, Dict, List - from django.tasks import BaseTask + from django.tasks import Task, TaskResult from django.tasks.backends.base import BaseTaskBackend class MyBackend(BaseTaskbackend): + task_class = Task + def __init__(self, options: Dict): """ Any connections which need to be setup can be done here """ super().__init__(options) - def is_valid_task_function(self, func: Callable) -> bool: - """ - Determine whether the provided callable is valid as a task function. - """ - ... - - def enqueue(self, func: Callable, priority: int | None, args: List, kwargs: Dict) -> BaseTask: - """ - Queue up a task function (or coroutine) to be executed - """ - ... - - def defer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: - """ - Add a task function (or coroutine) to be completed at a specific (timezone-aware) time - """ - ... - - async def aenqueue(self, func: Callable, priority: int | None, args: List, kwargs: Dict) -> BaseTask: - """ - Queue up a task function (or coroutine) to be executed - """ - ... - - async def adefer(self, func: Callable, priority: int | None, when: datetime, args: List, kwargs: Dict) -> BaseTask: + @classmethod + def is_valid_task(cls, task: Task) -> bool: """ - Add a task function (or coroutine) to be completed at a specific (timezone-aware) time + Determine whether the provided task is one which can be executed by the backend. """ ... - def get_task(self, task_id: str) -> BaseTask: + def enqueue(self, task: Task, *, args: List, kwargs: Dict, priority: int | None = None, run_after: datetime | None = None) -> TaskResult: """ - Retrieve a task by its id (if one exists). - If one doesn't, raises self.TaskDoesNotExist. + Queue up a task to be executed """ ... - async def aget_task(self, task_id: str) -> BaseTask: + def get_task(self, task_id: str) -> TaskResult: """ Retrieve a task by its id (if one exists). - If one doesn't, raises self.TaskDoesNotExist. + If one doesn't, raises TaskDoesNotExist. """ ... @@ -98,7 +76,12 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... -If a backend doesn't support a particular scheduling mode, it simply does not define the method. Convenience methods ``supports_enqueue`` and ``supports_defer`` will be implemented by ``BaseTaskBackend``. Similarly, ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue``, ``defer`` and ``get_task`` wrapped with ``asgiref.sync_to_async``. + +``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue`` and ``get_task`` using ``asgiref.sync_to_async``. + +If a backend receives a task which is not valid (ie ``is_valid_task`` returns ``False``), it should raise ``InvalidTaskError``. + +If a backend cannot support deferred tasks (ie passing the ``run_after`` argument), it should raise ``InvalidTaskError``. The ``supports_defer`` method can be used to determine whether the backend supports deferring tasks. ``is_valid_task_function`` determines whether the provided function (or possibly coroutine) is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. @@ -116,28 +99,18 @@ DummyBackend Tasks ----- -A ``Task`` is used as a handle to the running task, and contains useful information the application may need when referencing the task. +A ``Task`` is the action which the task runner will execute. It is a class which holds a callable and some defaults for ``enqueue``. + +Backend implementors aren't required to implement their own ``Task``, but may for additional functionality. .. code:: python from datetime import datetime - from typing import Any, Callable - - from django.tasks import BaseTask, TaskStatus - - class MyBackendTask(BaseTask): - id: str - """A unique identifier for the task""" - - status: TaskStatus - """The status of the task""" - - queued_at: datetime - """When the task was added to the queue""" + from typing import Callable - completed_at: datetime | None - """When the task was completed""" + from django.tasks import Task + class MyBackendTask(Task): priority: int | None """The priority of the task""" @@ -150,38 +123,63 @@ A ``Task`` is used as a handle to the running task, and contains useful informat kwargs: dict """The keyword arguments to pass to the task function""" - def __init__(self, **kwargs): - """ - Unpacking the raw response from the backend and storing it here for future use - """ - super().__init__(**kwargs) - def refresh(self) -> None: - """ - Reload the cached task data from the task store - """ - ... +A ``Task`` is created by decorating a function with ``@task``: - async def arefresh(self) -> None: - """ - Reload the cached task data from the task store - """ - ... +.. code:: python - @property - def result(self) -> Any: - """ - The return value from the task function. - If the task raised an exception, the result will contain that exception. - If the task has not completed, a `ValueError` is raised when accessing. - """ - ... + from django.tasks import task + + @task() + def do_a_task(*args, **kwargs): + pass + + +A ``Task`` can only be created for globally-importable callables. The task will be validated against the backend's ``is_valid_task`` callable during construction. + +``@task`` may be used on functions or coroutines. It will be up to the backend implementor to determine whether coroutines are supported. In either case, the function must be globally importable. + +Task Results +------------ + +A ``TaskResult`` is used as a handle to the running task, and contains useful information the application may need when referencing the execution of a ``Task``. -A ``Task`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. If called with a ``task_id`` which doesn't exist, a ``TaskDoesNotExist`` exception is raised. +A ``TaskResult`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. If called with a ``task_id`` which doesn't exist, a ``TaskDoesNotExist`` exception is raised. -A ``Task`` will cache its values, relying on the user calling ``refresh`` / ``arefresh`` to reload the values from the task store. +Backend implementors aren't required to implement their own ``TaskResult``, but may for additional functionality. -A ``Task``'s ``status`` must be one of the follwing values (as defined by an ``enum``): +.. code:: python + + from datetime import datetime + from typing import Callable + + from django.tasks import TaskResult, TaskStatus, Task + + class MyBackendTaskResult(TaskResult): + task: TaskResult + """The task of which this is a result""" + + priority: int | None + """The priority of the task""" + + run_after: datetime | None + """The earliest time the task will be executed""" + + status: TaskStatus + """The status of the running task""" + + def refresh(self) -> None: + """ + Reload the cached task data from the task store + """ + ... + + +Attributes such as ``priority`` will reflect the values used to enqueue the task, as opposed to the defaults from the ``Task``. If no overridden values are provided, the value will mirror the ``Task``. + +A ``TaskResult`` will cache its values, relying on the user calling ``refresh`` to reload the values from the task store. An ``async`` version of ``refresh`` is automatically provided by ``TaskResult`` using ``asgiref.sync_to_async``. + +A ``TaskResult``'s ``status`` must be one of the following values (as defined by an ``enum``): :NEW: The task has been created, but hasn't started running yet :RUNNING: The task is currently running @@ -190,27 +188,18 @@ A ``Task``'s ``status`` must be one of the follwing values (as defined by an ``e If a backend supports more than these statuses, it should compress them into one of these. -Task functions --------------- - -A task function is any globally-importable callable which can be used as the function for a task (ie passed into ``enqueue``). - -Before a task can be run, it must be marked: +For convenience, calling a ``TaskResult`` will execute the task's function directly, which allows for graceful transitioning towards background tasks: .. code:: python from django.tasks import task - @task + @task() def do_a_task(*args, **kwargs): pass -The decorator "marks" the task as being a valid function to be executed. This prevent arbitrary methods from being queued, potentially resulting in a security vulnerability (eg ``subprocess.run``). - -Tasks will be validated against the backend's ``is_valid_task_function`` before queueing. The default implementation will validate all generic assumptions: - -- Is the task function globally importable -- Has the task function been marked + # Calls `do_a_task` as if it weren't a task + do_a_task() Queueing tasks ------------- @@ -221,7 +210,7 @@ Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on from django.tasks import enqueue, task - @task + @task(priority=1) def do_a_task(*args, **kwargs): pass @@ -231,7 +220,10 @@ Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on # Optionally, provide arguments task = enqueue(do_a_task, args=[], kwargs={}) -Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. When multiple task backends are configured, each can be obtained from a global ``tasks`` connection handler: + # Override the priority defined by the `Task` + task = enqueue(do_a_task, priority=10) + +When multiple task backends are configured, each can be obtained from a global ``tasks`` connection handler. Whilst it's unlikely multiple backends will be configured for a single project, support is available. .. code:: python @@ -249,22 +241,22 @@ Similar methods are also available for ``defer``, ``aenqueue`` and ``adefer``. W When enqueueing tasks, ``args`` and ``kwargs`` are intentionally their own dedicated arguments to make the API simpler and backwards-compatible should other attributes be added in future. -Here, ``do_a_task`` can either be a regular function or coroutine. It will be up to the backend implementor to determine whether coroutines are supported. In either case, the function must be globally importable. - Deferring tasks --------------- -Tasks may also be "deferred" to run at a specific time in the future: +Tasks may also be "deferred" to run at a specific time in the future, by passing the ``run_after`` argument: .. code:: python from django.utils import timezone from datetime import timedelta - from django.tasks import defer + from django.tasks import enqueue + + task = enqueue(do_a_task, run_after=timezone.now() + timedelta(minutes=5)) - task = defer(do_a_task, when=timezone.now() + timedelta(minutes=5)) +``run_after`` must be a timezone-aware ``datetime``. -When scheduling a task, it may not be **exactly** that time a task is executed, however it should be accurate to within a few seconds. This will depend on the current state of the queue and task runners, and is out of the control of Django. +When deferring a task, it may not be **exactly** that time a task is executed, however it should be accurate to within a few seconds. This will depend on the current state of the queue and task runners, and is out of the control of Django. Sending emails -------------- @@ -290,7 +282,7 @@ Similarly, a backend may support queueing an async task function: from django.tasks import aenqueue, enqueue, task - @task + @task() async def do_an_async_task(): pass @@ -331,26 +323,26 @@ But what about *X*? The most obvious alternative to this DEP would be to standardise on a task implementation and vendor it in to Django. The Django ecosystem is already full of background worker libraries, eg Celery and RQ. Writing a production-ready task runner is a complex and nuanced undertaking, and discarding the work already done is a waste. -This proposal doesn't seek to replace existing tools, nor add yet another option for developers to consider. The primary motivation is creating a shared API contract between worker libaries and developers. It does however provide a simple way to get started, with a solution suitable for most sizes of projects (``DatabaseBackend``). Slowly increasing features, adding more built-in storage backends and a first-party task runner aren't out of the question for the future, but must be done with careful planning and consideration. +This proposal doesn't seek to replace existing tools, nor add yet another option for developers to consider. The primary motivation is creating a shared API contract between worker libraries and developers. It does however provide a simple way to get started, with a solution suitable for most sizes of projects (``DatabaseBackend``). Slowly increasing features, adding more built-in storage backends and a first-party task runner aren't out of the question for the future, but must be done with careful planning and consideration. Rationale ========= This proposed implementation specifically doesn't assume anything about the user's setup. This not only reduces the chances of Django conflicting with existing task systems a user may be using (eg Celery, RQ), but also allows it to work with almost any hosting environment a user might be using. -This proposal started out as `Wagtail RFC 72 `_, as it was becoming clear a unified interface for background tasks was required, without imposing on a developer's decisions for how the tasks are executed. Wagtail is run in many different forms at many differnt scales, so it needed to be possible to allow developers to choose the backend they're comfortable with, in a way which Wagtail and its associated packages can execute tasks without assuming anything of the environment it's running in. +This proposal started out as `Wagtail RFC 72 `_, as it was becoming clear a unified interface for background tasks was required, without imposing on a developer's decisions for how the tasks are executed. Wagtail is run in many different forms at many different scales, so it needed to be possible to allow developers to choose the backend they're comfortable with, in a way which Wagtail and its associated packages can execute tasks without assuming anything of the environment it's running in. -The global task connection ``tasks`` is used to access the configured backends, with global versions of those methods available for the default backend. This contradicts the pattern already used for storage and caches. A "task" is already used in a number of places to refer to an executed task, so using it to refer to the default backend is confusing and may lead to it being overridden in the current scope: +The global task connection ``tasks`` is used to access the configured backends, with global versions of those methods available for the default backend. This contradicts the pattern already used for storage and caches. A "task" is already used in a number of places, so using it to refer to the default backend is confusing and may lead to it being overridden in the current scope: .. code:: python - from django.tasks import task + from django.tasks import tasks # Later... - task = task.enqueue(do_a_thing) + result = tasks.enqueue(do_a_thing) # Clearer - thing_task = task.enqueue(do_a_thing) + thing_result = tasks.enqueue(do_a_thing) Backwards Compatibility ======================= From 3bf55ab4d0df518f3cf85664d535cc221701483f Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 28 Mar 2024 16:32:44 +0000 Subject: [PATCH 30/54] Support multiple queues This is needed for dedicated capacity --- draft/0000-background-workers.rst | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index a1446d17..4231eb35 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -57,7 +57,7 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... - def enqueue(self, task: Task, *, args: List, kwargs: Dict, priority: int | None = None, run_after: datetime | None = None) -> TaskResult: + def enqueue(self, task: Task, *, args: List, kwargs: Dict, priority: int | None = None, run_after: datetime | None = None, queue_name: str | None = None) -> TaskResult: """ Queue up a task to be executed """ @@ -123,6 +123,9 @@ Backend implementors aren't required to implement their own ``Task``, but may fo kwargs: dict """The keyword arguments to pass to the task function""" + queue_name: str | None + """The name of the queue the task will run on """ + A ``Task`` is created by decorating a function with ``@task``: @@ -168,6 +171,9 @@ Backend implementors aren't required to implement their own ``TaskResult``, but status: TaskStatus """The status of the running task""" + queue_name: str | None + """The name of the queue the task will run on """ + def refresh(self) -> None: """ Reload the cached task data from the task store @@ -175,7 +181,7 @@ Backend implementors aren't required to implement their own ``TaskResult``, but ... -Attributes such as ``priority`` will reflect the values used to enqueue the task, as opposed to the defaults from the ``Task``. If no overridden values are provided, the value will mirror the ``Task``. +Attributes such as ``priority`` and ``queue_name`` will reflect the values used to enqueue the task, as opposed to the defaults from the ``Task``. If no overridden values are provided, the value will mirror the ``Task``. A ``TaskResult`` will cache its values, relying on the user calling ``refresh`` to reload the values from the task store. An ``async`` version of ``refresh`` is automatically provided by ``TaskResult`` using ``asgiref.sync_to_async``. @@ -299,11 +305,15 @@ Settings TASKS = { "default": { "BACKEND": "django.tasks.backends.ImmediateBackend", + "QUEUES": [] "OPTIONS": {} } } -``OPTIONS`` is passed as-is to the backend's constructor. + +``QUEUES`` contains a list of valid queue names for the backend. If a task is queued to a queue which doesn't exist, an exception is raised. If omitted or empty, any name is valid. + +``OPTIONS`` is passed as-is to the backend's constructor. ``QUEUES`` is additionally passed to the constructor as the ``queues`` keyword argument. Motivation ========== From dc53f2f04dfbde5cb54609b5bb133bc263e79bbf Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 28 Mar 2024 16:33:10 +0000 Subject: [PATCH 31/54] Timeouts and failed hooks are out of scope --- draft/0000-background-workers.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 4231eb35..7e4508ec 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -371,12 +371,13 @@ Future iterations The field of background tasks is vast, and attempting to implement everything supported by existing tools in the first iteration is futile. The following functionality has been considered, and deemed explicitly out of scope of the first pass, but still worthy of future development: -- Completion hooks, to run subsequent tasks automatically +- Completion / failed hooks, to run subsequent tasks automatically - Bulk queueing - Automated task retrying - A generic way of executing task runners. This will remain the responsibility of the underlying implementation, and the user to execute correctly. - Observability into task queues, including monitoring and reporting - Cron-based scheduling +- Task timeouts Copyright ========= From 3d6b866f6ee0b09939ef0e0df69694e93ce7ee2f Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 28 Mar 2024 16:35:24 +0000 Subject: [PATCH 32/54] The reference implementation will eventually be a backport --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 7e4508ec..c70b8b6a 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -357,7 +357,7 @@ The global task connection ``tasks`` is used to access the configured backends, Backwards Compatibility ======================= -So that library maintainers can use this integration without concern as to whether a Django project has configured background workers, the default configuration will use the ``ImmediateBackend``. Developers on older versions of Django but who need libraries which assume tasks are available can use the reference implementation. +So that library maintainers can use this integration without concern as to whether a Django project has configured background workers, the default configuration will use the ``ImmediateBackend``. Developers on older versions of Django but who need libraries which assume tasks are available can use the reference implementation, which will serve as a backport and be API-compatible with Django. Reference Implementation ======================== From b973804f6e9489f1de05e90157ba3cadde29e020 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 28 Mar 2024 16:38:40 +0000 Subject: [PATCH 33/54] Arguments must be JSON serializable. --- draft/0000-background-workers.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index c70b8b6a..5efc0330 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -142,6 +142,8 @@ A ``Task`` can only be created for globally-importable callables. The task will ``@task`` may be used on functions or coroutines. It will be up to the backend implementor to determine whether coroutines are supported. In either case, the function must be globally importable. +Task arguments must be JSON serializable, to avoid compatibility and versioning issues. Complex arguments should be converted to a format which is JSON-serializable. + Task Results ------------ From d33aa11f133065788d20df251080d3088b3659e7 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 28 Mar 2024 16:54:07 +0000 Subject: [PATCH 34/54] Tie a task to a backend --- draft/0000-background-workers.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 5efc0330..6646b54e 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -126,6 +126,9 @@ Backend implementors aren't required to implement their own ``Task``, but may fo queue_name: str | None """The name of the queue the task will run on """ + backend: str | None + """The name of the backend the task will run on" + A ``Task`` is created by decorating a function with ``@task``: @@ -249,6 +252,8 @@ When multiple task backends are configured, each can be obtained from a global ` When enqueueing tasks, ``args`` and ``kwargs`` are intentionally their own dedicated arguments to make the API simpler and backwards-compatible should other attributes be added in future. +If a ``Task`` is defined to run on a different backend, ``InvalidTaskError`` is raised. + Deferring tasks --------------- From 32611905845b630bb199bf8c9827b6c8c49ca4a9 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 2 Apr 2024 12:06:42 +0100 Subject: [PATCH 35/54] Better document task validation --- draft/0000-background-workers.rst | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 6646b54e..dca0d9fe 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -79,11 +79,12 @@ A backend will be a class which extends a Django-defined base class, and provide ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue`` and ``get_task`` using ``asgiref.sync_to_async``. -If a backend receives a task which is not valid (ie ``is_valid_task`` returns ``False``), it should raise ``InvalidTaskError``. +``is_valid_task`` determines whether the provided ``Task`` is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. If a backend receives a task which is not valid (ie ``is_valid_task`` returns ``False``), it should raise ``InvalidTaskError``. The base implementation of ``is_valid_task`` will validate: -If a backend cannot support deferred tasks (ie passing the ``run_after`` argument), it should raise ``InvalidTaskError``. The ``supports_defer`` method can be used to determine whether the backend supports deferring tasks. +- Is the task's function a valid, globally-importable callable? +- Is the task allowed to be run on the current backend? -``is_valid_task_function`` determines whether the provided function (or possibly coroutine) is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. +If a backend cannot support deferred tasks (ie passing the ``run_after`` argument), it should raise ``InvalidTaskError``. The ``supports_defer`` method can be used to determine whether the backend supports deferring tasks. Django will ship with 3 implementations: @@ -117,12 +118,6 @@ Backend implementors aren't required to implement their own ``Task``, but may fo func: Callable """The task function""" - args: list - """The arguments to pass to the task function""" - - kwargs: dict - """The keyword arguments to pass to the task function""" - queue_name: str | None """The name of the queue the task will run on """ @@ -143,7 +138,7 @@ A ``Task`` is created by decorating a function with ``@task``: A ``Task`` can only be created for globally-importable callables. The task will be validated against the backend's ``is_valid_task`` callable during construction. -``@task`` may be used on functions or coroutines. It will be up to the backend implementor to determine whether coroutines are supported. In either case, the function must be globally importable. +``@task`` may be used on functions or coroutines. It will be up to the backend implementor to determine whether coroutines are supported. Support for coroutine tasks can be determined with the ``supports_coroutine_tasks`` method on the backend. In either case, the function must be globally importable. Task arguments must be JSON serializable, to avoid compatibility and versioning issues. Complex arguments should be converted to a format which is JSON-serializable. @@ -179,11 +174,17 @@ Backend implementors aren't required to implement their own ``TaskResult``, but queue_name: str | None """The name of the queue the task will run on """ + args: list + """The arguments to pass to the task function""" + + kwargs: dict + """The keyword arguments to pass to the task function""" + def refresh(self) -> None: - """ - Reload the cached task data from the task store - """ - ... + """ + Reload the cached task data from the task store + """ + ... Attributes such as ``priority`` and ``queue_name`` will reflect the values used to enqueue the task, as opposed to the defaults from the ``Task``. If no overridden values are provided, the value will mirror the ``Task``. From 1c3daef92c1efd9c885f284bedcfafd43a38253c Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 2 Apr 2024 12:08:25 +0100 Subject: [PATCH 36/54] Match `DATABASES` convention for settings --- draft/0000-background-workers.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index dca0d9fe..04f6cca7 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -44,11 +44,11 @@ A backend will be a class which extends a Django-defined base class, and provide class MyBackend(BaseTaskbackend): task_class = Task - def __init__(self, options: Dict): + def __init__(self, settings_dict: Dict): """ Any connections which need to be setup can be done here """ - super().__init__(options) + super().__init__(settings_dict) @classmethod def is_valid_task(cls, task: Task) -> bool: @@ -321,8 +321,6 @@ Settings ``QUEUES`` contains a list of valid queue names for the backend. If a task is queued to a queue which doesn't exist, an exception is raised. If omitted or empty, any name is valid. -``OPTIONS`` is passed as-is to the backend's constructor. ``QUEUES`` is additionally passed to the constructor as the ``queues`` keyword argument. - Motivation ========== From 193a5c9f64ea340b75185ddd6446c7cac122ef97 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 2 Apr 2024 12:18:42 +0100 Subject: [PATCH 37/54] Assume the default backend when not defined --- draft/0000-background-workers.rst | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 04f6cca7..5939ca70 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -121,8 +121,8 @@ Backend implementors aren't required to implement their own ``Task``, but may fo queue_name: str | None """The name of the queue the task will run on """ - backend: str | None - """The name of the backend the task will run on" + backend: str + """The name of the backend the task will run on""" A ``Task`` is created by decorating a function with ``@task``: @@ -138,6 +138,8 @@ A ``Task`` is created by decorating a function with ``@task``: A ``Task`` can only be created for globally-importable callables. The task will be validated against the backend's ``is_valid_task`` callable during construction. +If a task doesn't define a backend, it is assumed it will only use the default backend. + ``@task`` may be used on functions or coroutines. It will be up to the backend implementor to determine whether coroutines are supported. Support for coroutine tasks can be determined with the ``supports_coroutine_tasks`` method on the backend. In either case, the function must be globally importable. Task arguments must be JSON serializable, to avoid compatibility and versioning issues. Complex arguments should be converted to a format which is JSON-serializable. @@ -180,6 +182,9 @@ Backend implementors aren't required to implement their own ``TaskResult``, but kwargs: dict """The keyword arguments to pass to the task function""" + backend: str + """The name of the backend the task will run on""" + def refresh(self) -> None: """ Reload the cached task data from the task store From d07a9ab446b2578a40a69a3aa164ddf9d95b029a Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Tue, 2 Apr 2024 12:20:04 +0100 Subject: [PATCH 38/54] Bump last modified date Lots has changed since Feb! --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 5939ca70..00406269 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -9,7 +9,7 @@ DEP XXXX: Background workers :Status: Draft :Type: Feature :Created: 2024-02-07 -:Last-Modified: 2024-02-09 +:Last-Modified: 2024-04-02 .. contents:: Table of Contents :depth: 3 From 310460e060366b4c90365adf839a94ba407c7aae Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 4 Apr 2024 17:16:48 +0100 Subject: [PATCH 39/54] Explicitly note swappable argument serialization is out of scope --- draft/0000-background-workers.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 00406269..64bf2e9d 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -219,7 +219,7 @@ For convenience, calling a ``TaskResult`` will execute the task's function direc do_a_task() Queueing tasks -------------- +-------------- Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on the default task backend: @@ -389,6 +389,7 @@ The field of background tasks is vast, and attempting to implement everything su - Observability into task queues, including monitoring and reporting - Cron-based scheduling - Task timeouts +- Swappable argument serialization (eg `pickle`) Copyright ========= From be6fa49d371d6bc9553bf537d26d92404621138c Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 5 Apr 2024 11:15:22 +0100 Subject: [PATCH 40/54] Move primary enqueue APIs onto the task This allows them to be statically typed, and avoids the need to import the Django methods all over the place. --- draft/0000-background-workers.rst | 99 +++++++++++++++++-------------- 1 file changed, 54 insertions(+), 45 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 64bf2e9d..73dc1b64 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -9,7 +9,7 @@ DEP XXXX: Background workers :Status: Draft :Type: Feature :Created: 2024-02-07 -:Last-Modified: 2024-04-02 +:Last-Modified: 2024-04-05 .. contents:: Table of Contents :depth: 3 @@ -57,7 +57,7 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... - def enqueue(self, task: Task, *, args: List, kwargs: Dict, priority: int | None = None, run_after: datetime | None = None, queue_name: str | None = None) -> TaskResult: + def enqueue(self, task: Task, *args, **kwargs) -> TaskResult: """ Queue up a task to be executed """ @@ -107,9 +107,9 @@ Backend implementors aren't required to implement their own ``Task``, but may fo .. code:: python from datetime import datetime - from typing import Callable + from typing import Callable, Self - from django.tasks import Task + from django.tasks import Task, TaskResult class MyBackendTask(Task): priority: int | None @@ -124,6 +124,27 @@ Backend implementors aren't required to implement their own ``Task``, but may fo backend: str """The name of the backend the task will run on""" + run_after: datetime | None + """The earliest this task will run""" + + def using(self, priority: int | None = None, queue_name: str | None = None, run_after: datetime | None = None) -> Self: + """ + Create a new task with modified defaults + """ + ... + + def enqueue(self, *args, **kwargs) -> TaskResult: + """ + Queue up the task to be executed + """ + ... + + def get(self, task_id: str) -> Self: + """ + Retrieve a task of this type by its id (if one exists). + If one doesn't, or is the wrong type, raises TaskDoesNotExist. + """ + ... A ``Task`` is created by decorating a function with ``@task``: @@ -144,6 +165,8 @@ If a task doesn't define a backend, it is assumed it will only use the default b Task arguments must be JSON serializable, to avoid compatibility and versioning issues. Complex arguments should be converted to a format which is JSON-serializable. +The ``using`` method returns a clone of the task with the given attributes modified. This allows modification of the task before calling ``enqueue``. ``run_after`` cannot be passed to ``@task``, and can only be configued with ``using``. + Task Results ------------ @@ -161,21 +184,12 @@ Backend implementors aren't required to implement their own ``TaskResult``, but from django.tasks import TaskResult, TaskStatus, Task class MyBackendTaskResult(TaskResult): - task: TaskResult - """The task of which this is a result""" - - priority: int | None - """The priority of the task""" - - run_after: datetime | None - """The earliest time the task will be executed""" + task: Task + """The task for which this is a result""" status: TaskStatus """The status of the running task""" - queue_name: str | None - """The name of the queue the task will run on """ - args: list """The arguments to pass to the task function""" @@ -192,8 +206,6 @@ Backend implementors aren't required to implement their own ``TaskResult``, but ... -Attributes such as ``priority`` and ``queue_name`` will reflect the values used to enqueue the task, as opposed to the defaults from the ``Task``. If no overridden values are provided, the value will mirror the ``Task``. - A ``TaskResult`` will cache its values, relying on the user calling ``refresh`` to reload the values from the task store. An ``async`` version of ``refresh`` is automatically provided by ``TaskResult`` using ``asgiref.sync_to_async``. A ``TaskResult``'s ``status`` must be one of the following values (as defined by an ``enum``): @@ -221,24 +233,30 @@ For convenience, calling a ``TaskResult`` will execute the task's function direc Queueing tasks -------------- -Tasks can be queued using ``enqueue``, a proxy method which calls ``enqueue`` on the default task backend: +Tasks can be queued using the ``enqueue`` method, which in turn calls ``enqueue`` on the task backend: .. code:: python - from django.tasks import enqueue, task + from django.tasks import task @task(priority=1) def do_a_task(*args, **kwargs): pass # Submit the task function to be run - task = enqueue(do_a_task) + result = do_a_task.enqueue() # Optionally, provide arguments - task = enqueue(do_a_task, args=[], kwargs={}) + result = do_a_task.enqueue(1, two="three") # Override the priority defined by the `Task` - task = enqueue(do_a_task, priority=10) + result = do_a_task.using(priority=10).enqueue() + + # The modified task can be saved and reused + do_a_high_priority_task = do_a_task.using(priority=20) + for i in range(5): + do_a_high_priority_task.enqueue(i) + When multiple task backends are configured, each can be obtained from a global ``tasks`` connection handler. Whilst it's unlikely multiple backends will be configured for a single project, support is available. @@ -246,17 +264,20 @@ When multiple task backends are configured, each can be obtained from a global ` from django.tasks import tasks, task - @task + @task() def do_a_task(*args, **kwargs): pass # Submit the task function to be run - task = tasks["special"].enqueue(do_a_task) + result = tasks["special"].enqueue(do_a_task) # Optionally, provide arguments - task = tasks["special"].enqueue(do_a_task, args=[], kwargs={}) + result = tasks["special"].enqueue(do_a_task, 1, two="three") + + # Alternatively + result = do_a_task.using(backend="special").enqueue(1, two="three") -When enqueueing tasks, ``args`` and ``kwargs`` are intentionally their own dedicated arguments to make the API simpler and backwards-compatible should other attributes be added in future. +Whilst this API is available, it's best to call ``enqueue`` on the ``Task`` directly instead and configure the backend using the ``backend`` argument. If a ``Task`` is defined to run on a different backend, ``InvalidTaskError`` is raised. @@ -269,9 +290,8 @@ Tasks may also be "deferred" to run at a specific time in the future, by passing from django.utils import timezone from datetime import timedelta - from django.tasks import enqueue - task = enqueue(do_a_task, run_after=timezone.now() + timedelta(minutes=5)) + result = do_a_task.using(run_after=timezone.now() + timedelta(minutes=5)).enqueue() ``run_after`` must be a timezone-aware ``datetime``. @@ -291,24 +311,23 @@ Where the underlying task runner supports it, backends may also provide an ``asy .. code:: python - from django.tasks import aenqueue - - await aenqueue(do_a_task) + await do_a_task.aenqueue() + await do_a_task.using(priority=10).aenqueue() Similarly, a backend may support queueing an async task function: .. code:: python - from django.tasks import aenqueue, enqueue, task + from django.tasks import task @task() async def do_an_async_task(): pass - await aenqueue(do_an_async_task) + await do_an_async_task.aenqueue() # Also works - enqueue(do_an_async_task) + do_an_async_task.enqueue() Settings --------- @@ -353,17 +372,7 @@ This proposed implementation specifically doesn't assume anything about the user This proposal started out as `Wagtail RFC 72 `_, as it was becoming clear a unified interface for background tasks was required, without imposing on a developer's decisions for how the tasks are executed. Wagtail is run in many different forms at many different scales, so it needed to be possible to allow developers to choose the backend they're comfortable with, in a way which Wagtail and its associated packages can execute tasks without assuming anything of the environment it's running in. -The global task connection ``tasks`` is used to access the configured backends, with global versions of those methods available for the default backend. This contradicts the pattern already used for storage and caches. A "task" is already used in a number of places, so using it to refer to the default backend is confusing and may lead to it being overridden in the current scope: - -.. code:: python - - from django.tasks import tasks - - # Later... - result = tasks.enqueue(do_a_thing) - - # Clearer - thing_result = tasks.enqueue(do_a_thing) +The API design has been intentionally designed with type-safety in mind, including support for statically validating task arguments. Using ``Task.enqueue`` allows its arguments to be statically typed, and ``using`` allows the ``Task`` to be immutable (much like ``QuerySet``). Types should be able to flow from the task function, through the ``Task`` and eventually to the ``TaskResult``. Backwards Compatibility ======================= From cbe24fe84b13ba04f95a264d9d6372402198ce28 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 19 Apr 2024 11:05:01 +0100 Subject: [PATCH 41/54] Allow passing a `timedelta` directly to `run_after` --- draft/0000-background-workers.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 73dc1b64..138feea6 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -9,7 +9,7 @@ DEP XXXX: Background workers :Status: Draft :Type: Feature :Created: 2024-02-07 -:Last-Modified: 2024-04-05 +:Last-Modified: 2024-04-19 .. contents:: Table of Contents :depth: 3 @@ -127,7 +127,7 @@ Backend implementors aren't required to implement their own ``Task``, but may fo run_after: datetime | None """The earliest this task will run""" - def using(self, priority: int | None = None, queue_name: str | None = None, run_after: datetime | None = None) -> Self: + def using(self, priority: int | None = None, queue_name: str | None = None, run_after: datetime | timedelta | None = None) -> Self: """ Create a new task with modified defaults """ @@ -291,9 +291,13 @@ Tasks may also be "deferred" to run at a specific time in the future, by passing from django.utils import timezone from datetime import timedelta + # Run the task at a specific time. result = do_a_task.using(run_after=timezone.now() + timedelta(minutes=5)).enqueue() -``run_after`` must be a timezone-aware ``datetime``. + # Or, pass the `timedelta` directly. + result = do_a_task.using(run_after=timedelta(minutes=5)).enqueue() + +``run_after`` must be a ``timedelta`` or timezone-aware ``datetime``. When deferring a task, it may not be **exactly** that time a task is executed, however it should be accurate to within a few seconds. This will depend on the current state of the queue and task runners, and is out of the control of Django. From 95b3205c3ee26a9fb9b2a3ca7817d077f4d0b112 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 19 Apr 2024 11:05:18 +0100 Subject: [PATCH 42/54] Avoid explicit count of included backends --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 138feea6..0ca3fb7e 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -86,7 +86,7 @@ A backend will be a class which extends a Django-defined base class, and provide If a backend cannot support deferred tasks (ie passing the ``run_after`` argument), it should raise ``InvalidTaskError``. The ``supports_defer`` method can be used to determine whether the backend supports deferring tasks. -Django will ship with 3 implementations: +Django will ship with the following implementations: ImmediateBackend This backend runs the tasks immediately, rather than offloading to a background process. This is useful both for a graceful transition towards background workers, but without impacting existing functionality. From 7f314e81ede57297856b148145aaea9da5668d3b Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 19 Apr 2024 11:09:06 +0100 Subject: [PATCH 43/54] Fix typo when defining example task backend --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 0ca3fb7e..1c633aee 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -41,7 +41,7 @@ A backend will be a class which extends a Django-defined base class, and provide from django.tasks.backends.base import BaseTaskBackend - class MyBackend(BaseTaskbackend): + class MyBackend(BaseTaskBackend): task_class = Task def __init__(self, settings_dict: Dict): From f05a5f828c69d58a03f2c8824fdd11e2bf59e832 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 19 Apr 2024 16:33:26 +0100 Subject: [PATCH 44/54] Flesh out abstract --- draft/0000-background-workers.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 1c633aee..2bc14d9c 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -18,14 +18,20 @@ DEP XXXX: Background workers Abstract ======== +Whilst Django is a web framework, there's more to web applications than just the request-response lifecycle. Sending emails, communicating with external services or running complex actions should all be done outside the request-response cycle. + Django doesn't have a first-party solution for long-running tasks, however the ecosystem is filled with incredibly popular frameworks, all of which interact with Django in slightly different ways. Other frameworks such as Laravel have background workers built-in, allowing them to push tasks into the background to be processed at a later date, without requiring the end user to wait for them to occur. Library maintainers must implement support for any possible task backend separately, should they wish to offload functionality to the background. This includes smaller libraries, but also larger meta-frameworks with their own package ecosystem such as `Wagtail `_. +This proposal sets out to provide an interface and base implementation for long-running background tasks in Django. + Specification ============= -The proposed implementation will be in the form of an application wide "task backend" interface. This backend will be what connects Django to the task runners with a single pattern. The task backend will provide an interface for either third-party libraries, or application developers to specify how tasks should be created and pushed into the background. +The proposed implementation will be in the form of an application wide "task backend" interface(s). This backend will be what connects Django to the task runners with a single pattern. The task backend will provide an interface for either third-party libraries, or application developers to specify how tasks should be created and pushed into the background. + +Alongside this interface, Django will provide a few built-in backends, useful for testing, local development and production use cases. Backends -------- From 54b45158561a4418b509599f20ccdca299a64dcc Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 19 Apr 2024 17:35:04 +0100 Subject: [PATCH 45/54] Restore id to task result --- draft/0000-background-workers.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 2bc14d9c..11d88f6c 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -193,6 +193,9 @@ Backend implementors aren't required to implement their own ``TaskResult``, but task: Task """The task for which this is a result""" + id: str + """A unique identifier for the task result""" + status: TaskStatus """The status of the running task""" From 74d2d7b67449ad1af86a1dd4f8cc4d6459d7c237 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 19 Apr 2024 17:38:53 +0100 Subject: [PATCH 46/54] It's the result which is "got" --- draft/0000-background-workers.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 11d88f6c..772ebc1f 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -69,10 +69,10 @@ A backend will be a class which extends a Django-defined base class, and provide """ ... - def get_task(self, task_id: str) -> TaskResult: + def get_result(self, result_id: str) -> TaskResult: """ - Retrieve a task by its id (if one exists). - If one doesn't, raises TaskDoesNotExist. + Retrieve a result by its id (if one exists). + If one doesn't, raises ResultDoesNotExist. """ ... @@ -83,7 +83,7 @@ A backend will be a class which extends a Django-defined base class, and provide ... -``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue`` and ``get_task`` using ``asgiref.sync_to_async``. +``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue`` and ``get_result`` using ``asgiref.sync_to_async``. ``is_valid_task`` determines whether the provided ``Task`` is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. If a backend receives a task which is not valid (ie ``is_valid_task`` returns ``False``), it should raise ``InvalidTaskError``. The base implementation of ``is_valid_task`` will validate: @@ -145,10 +145,10 @@ Backend implementors aren't required to implement their own ``Task``, but may fo """ ... - def get(self, task_id: str) -> Self: + def get(self, result_id: str) -> Self: """ - Retrieve a task of this type by its id (if one exists). - If one doesn't, or is the wrong type, raises TaskDoesNotExist. + Retrieve a result for a task of this type by its id (if one exists). + If one doesn't, or is the wrong type, raises ResultDoesNotExist. """ ... @@ -178,7 +178,7 @@ Task Results A ``TaskResult`` is used as a handle to the running task, and contains useful information the application may need when referencing the execution of a ``Task``. -A ``TaskResult`` is obtained either when scheduling a task function, or by calling ``get_task`` on the backend. If called with a ``task_id`` which doesn't exist, a ``TaskDoesNotExist`` exception is raised. +A ``TaskResult`` is obtained either when scheduling a task function, or by calling ``get_result`` on the backend. If called with a ``task_id`` which doesn't exist, a ``TaskDoesNotExist`` exception is raised. Backend implementors aren't required to implement their own ``TaskResult``, but may for additional functionality. From 1c8d2f02fdc1b1bc5d83e2cb117c936e0ad577a2 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 22 Apr 2024 15:36:41 +0100 Subject: [PATCH 47/54] Rename task validation function This allows it to handle raising the exception itself, which makes errors more descriptive --- draft/0000-background-workers.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 772ebc1f..2882d7c3 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -57,7 +57,7 @@ A backend will be a class which extends a Django-defined base class, and provide super().__init__(settings_dict) @classmethod - def is_valid_task(cls, task: Task) -> bool: + def validate_task(cls, task: Task) -> None: """ Determine whether the provided task is one which can be executed by the backend. """ @@ -85,10 +85,7 @@ A backend will be a class which extends a Django-defined base class, and provide ``BaseTaskBackend`` will provide ``a``-prefixed stubs for ``enqueue`` and ``get_result`` using ``asgiref.sync_to_async``. -``is_valid_task`` determines whether the provided ``Task`` is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. If a backend receives a task which is not valid (ie ``is_valid_task`` returns ``False``), it should raise ``InvalidTaskError``. The base implementation of ``is_valid_task`` will validate: - -- Is the task's function a valid, globally-importable callable? -- Is the task allowed to be run on the current backend? +``validate_task`` determines whether the provided ``Task`` is valid for the backend. This can be used to prevent coroutines from being executed, or otherwise validate the callable. If the provided task is invalid, it will raise ``InvalidTaskError``. If a backend cannot support deferred tasks (ie passing the ``run_after`` argument), it should raise ``InvalidTaskError``. The ``supports_defer`` method can be used to determine whether the backend supports deferring tasks. @@ -163,7 +160,7 @@ A ``Task`` is created by decorating a function with ``@task``: pass -A ``Task`` can only be created for globally-importable callables. The task will be validated against the backend's ``is_valid_task`` callable during construction. +A ``Task`` can only be created for globally-importable callables. The task will be validated against the backend's ``validate_task`` during construction. If a task doesn't define a backend, it is assumed it will only use the default backend. From c165ffd0281edaedbf2520b1dfc287c7c2a73895 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 22 Apr 2024 16:51:18 +0100 Subject: [PATCH 48/54] Give task result getter a more descriptive name --- draft/0000-background-workers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 2882d7c3..12067ec9 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -142,7 +142,7 @@ Backend implementors aren't required to implement their own ``Task``, but may fo """ ... - def get(self, result_id: str) -> Self: + def get_result(self, result_id: str) -> Self: """ Retrieve a result for a task of this type by its id (if one exists). If one doesn't, or is the wrong type, raises ResultDoesNotExist. From a141ba12f25ecd2ac770957f1261333abfa62d0b Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 22 Apr 2024 17:15:35 +0100 Subject: [PATCH 49/54] Expose the task's return value --- draft/0000-background-workers.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index 12067ec9..d5f821cd 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -182,7 +182,7 @@ Backend implementors aren't required to implement their own ``TaskResult``, but .. code:: python from datetime import datetime - from typing import Callable + from typing import Any, Callable from django.tasks import TaskResult, TaskStatus, Task @@ -205,6 +205,9 @@ Backend implementors aren't required to implement their own ``TaskResult``, but backend: str """The name of the backend the task will run on""" + result: Any + """The return value from the task""" + def refresh(self) -> None: """ Reload the cached task data from the task store @@ -219,7 +222,7 @@ A ``TaskResult``'s ``status`` must be one of the following values (as defined by :NEW: The task has been created, but hasn't started running yet :RUNNING: The task is currently running :FAILED: The task failed -:COMPLETE: The task is complete, and the result is accessible +:COMPLETE: The task is complete, and the ``result`` is accessible If a backend supports more than these statuses, it should compress them into one of these. From 23791232c7e951a3b2edb1b1d7d82cb63ab74552 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 26 Apr 2024 15:30:42 +0100 Subject: [PATCH 50/54] Rename `TaskStatus` enum It's the `TaskResult` which has the status --- draft/0000-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0000-background-workers.rst b/draft/0000-background-workers.rst index d5f821cd..8cc9bcba 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0000-background-workers.rst @@ -184,7 +184,7 @@ Backend implementors aren't required to implement their own ``TaskResult``, but from datetime import datetime from typing import Any, Callable - from django.tasks import TaskResult, TaskStatus, Task + from django.tasks import TaskResult, ResultStatus, Task class MyBackendTaskResult(TaskResult): task: Task @@ -193,7 +193,7 @@ Backend implementors aren't required to implement their own ``TaskResult``, but id: str """A unique identifier for the task result""" - status: TaskStatus + status: ResultStatus """The status of the running task""" args: list From 46be8f3bbc8a4303003929d44ae3d8296e5b9a88 Mon Sep 17 00:00:00 2001 From: Carlton Gibson Date: Mon, 13 May 2024 16:31:31 +0200 Subject: [PATCH 51/54] Number DEP 14 Background Workers. --- ...000-background-workers.rst => 0014-background-workers.rst} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename draft/{0000-background-workers.rst => 0014-background-workers.rst} (99%) diff --git a/draft/0000-background-workers.rst b/draft/0014-background-workers.rst similarity index 99% rename from draft/0000-background-workers.rst rename to draft/0014-background-workers.rst index 8cc9bcba..e0f0b601 100644 --- a/draft/0000-background-workers.rst +++ b/draft/0014-background-workers.rst @@ -1,8 +1,8 @@ ============================= -DEP XXXX: Background workers +DEP 0014: Background workers ============================= -:DEP: XXXX +:DEP: 0014 :Author: Jake Howard :Implementation Team: Jake Howard :Shepherd: Carlton Gibson From cecaf44cffda6e25a2874aeb1178113d530c9f80 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 13 May 2024 16:15:08 +0100 Subject: [PATCH 52/54] Improve wording for the reference implementation --- draft/0014-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/draft/0014-background-workers.rst b/draft/0014-background-workers.rst index e0f0b601..81ea8b8e 100644 --- a/draft/0014-background-workers.rst +++ b/draft/0014-background-workers.rst @@ -395,9 +395,9 @@ So that library maintainers can use this integration without concern as to wheth Reference Implementation ======================== -The reference implementation will be developed alongside this DEP process. This implementation will serve both as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django core, but also as a backport for users of supported Django versions prior to this work being released. +A reference implementation is being developed alongside this DEP process. This implementation will serve as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django itself, but also as a backport for users of supported Django versions prior to this work being released. -A more complete implementation picture can be found at https://github.com/RealOrangeOne/django-core-tasks, however it should not be considered final. +The reference implementation can be found at https://github.com/RealOrangeOne/django-core-tasks, along with its progression. Future iterations ================= From a8e08a955b8a96634b033236081f3071e265f44c Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Mon, 13 May 2024 16:20:23 +0100 Subject: [PATCH 53/54] Clarify backwards compatibility with phased support --- draft/0014-background-workers.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/draft/0014-background-workers.rst b/draft/0014-background-workers.rst index 81ea8b8e..28a8ad4d 100644 --- a/draft/0014-background-workers.rst +++ b/draft/0014-background-workers.rst @@ -392,10 +392,12 @@ Backwards Compatibility So that library maintainers can use this integration without concern as to whether a Django project has configured background workers, the default configuration will use the ``ImmediateBackend``. Developers on older versions of Django but who need libraries which assume tasks are available can use the reference implementation, which will serve as a backport and be API-compatible with Django. +For users who need newer libraries which require this interface, but can't update Django itself, the reference implementation can be used. Users can use either ``django_tasks.task`` or ``django.tasks.task`` to register a task, which is usable with any configured backend, regardless of its source. + Reference Implementation ======================== -A reference implementation is being developed alongside this DEP process. This implementation will serve as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django itself, but also as a backport for users of supported Django versions prior to this work being released. +A reference implementation (``django_tasks``) is being developed alongside this DEP process. This implementation will serve as an "early-access" demo to get initial feedback and start using the interface, as the basis for the integration with Django itself, but also as a backport for users of supported Django versions prior to this work being released. The reference implementation can be found at https://github.com/RealOrangeOne/django-core-tasks, along with its progression. From d277b9df132778674cf778c764a7413b2b06d0e4 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Wed, 29 May 2024 09:12:19 +0100 Subject: [PATCH 54/54] Mark DEP 0014 as "Accepted" --- {draft => accepted}/0014-background-workers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename {draft => accepted}/0014-background-workers.rst (99%) diff --git a/draft/0014-background-workers.rst b/accepted/0014-background-workers.rst similarity index 99% rename from draft/0014-background-workers.rst rename to accepted/0014-background-workers.rst index 28a8ad4d..b619555e 100644 --- a/draft/0014-background-workers.rst +++ b/accepted/0014-background-workers.rst @@ -6,10 +6,10 @@ DEP 0014: Background workers :Author: Jake Howard :Implementation Team: Jake Howard :Shepherd: Carlton Gibson -:Status: Draft +:Status: Accepted :Type: Feature :Created: 2024-02-07 -:Last-Modified: 2024-04-19 +:Last-Modified: 2024-05-13 .. contents:: Table of Contents :depth: 3