Improve the django experience

[ewjoachim]

@ignaciocabeza shared that the experience of integrating procrastinate in a Django project could be improved.

I think we would be wise listening to this feedback, reproduce and solve the problems, and making sure that developing with procrastinate & Django works well !

(Agate, I think you might be interested as well :) ping @EliotBerriot )

[turicas]

#745 enhances the docs about Django integration.

Also, I was thinking in creating a procrastinate management command (+ a settings var PROCRASTINATE_APP with the dotted app path) that deals with everything related to running the worker, so the user just run python manage.py procrastinate. What do you think?

[ewjoachim]

Yep, I think you're on the right track

[turicas]

I have some thoughts to share regarding enhancing procrastinate's Django experience. In summary, I think the best Django experience would be something like:

1. pip install procrastinate[django]
2. Add "procrastinate.contrib.django" to INSTALLED_APPS
3. Tweak one or two PROCRASTINATE_* variables inside Django's settings.py file
4. Create tasks inside myapp/tasks.py, decorating them with procrastinate.task
5. Run the worker with python manage.py procrastinate-worker (set DJANGO_SETTINGS_MODULE env var when running the worker)
6. Monitor tasks inside Django Admin (running, finished, failed, waiting etc.)

To implement this experience I think procrastinate needs to:

• Must have a simple and easy way to define tasks, integrated with Django ( tasks/__init__.py as cited
  in the docs is very complex and error-prone)
• We should not need to expose tasks after declaring them (like myapp/tasks.py in the docs), it's redundant
• Must have an integrated way to run the worker (using python manage.py procrastinate-worker or procrastinate django-worker)

For this to work, I think we need to be able to define a task without instantiating the procrastinate App or we could implement a helper function to instantiate a procrastinate App based on Django settings. I prefer the second method, but I'm going with the first here to simplify the code examples. Also, I think procrastinate should auto discover tasks declared in all Django apps (like Celery does), so it can attach them to the App instance and run the worker automatically.

Let's say we define these two functions and expose them as procrastinate.task and procrastinate.is_task:

def task(func):  # Needs lots of enhancements to be more like `App.task`, of course
    func._procrastinate_task = True
    return func


def is_task(func):
    return getattr(func, "_procrastinate_task", None) == True

And inside myapp/tasks.py:

from procrastinate import task


@task
def double(number: int):
    return number ** 2


@task
def hello(name: str):
    return f"Hello, {name}"

Note that I'm not attaching a task to an App instance here - it's just a way to flag the function as something that should be exposed (when an App is instantiated). So now we have the concept of "unbound" and "bound" tasks.

We can then implement a function to auto discover tasks by inspecting all Django apps:

# Some module inside procrastinate.contrib.django
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
# TODO: what if my project settings file is not inside `project/`?
# XXX: we may need to force the user to set `DJANGO_SETTINGS_MODULE` env var to use procrastinate with Django, so we
# don't need the `os.environ.setdefault` above, since it will not work for all the projects.

import django
django.setup()

import importlib
import inspect
from dataclasses import dataclass
from typing import Callable

from django.apps import apps
from procrastinate import is_task  # as defined in the code section above


@dataclass
class AppTask:
    name: str
    func: Callable
    app_name: str


def autodiscover_tasks():
    for app_config in apps.get_app_configs():
        app_name = app_config.name
        try:
            tasks_module = importlib.import_module(f"{app_name}.tasks")
        except ModuleNotFoundError:
            continue
        for name, func in inspect.getmembers(tasks_module):
            if is_task(func):
                yield AppTask(name=name, func=func, app_name=app_name)

Finally, try it:

from procrastinate.contrib.django import autodiscover_tasks


for app_task in autodiscover_tasks():
    print(app_task)

With the autodiscover_tasks and a helper function to instantiate procrastinate App based on Django settings we can easily (maybe) implement a procrastinate-django worker, so the user won't need to think about it.

[ewjoachim]

I believe the lib is now in a state where it's reasonable to start tackling this.

@turicas is this something you'd like to contribute ? Or would you rather I do it and you'd give some feedback ? Last option, we could try and work together on that if it's something that works for you :)

[ewjoachim]

Ok, so I'm all for making it easy to get the app defined for you if you want, but I think once the app is easy to get, we should use @app.task(). We may need to do some magic to avoid issues where the app hasn't been defined yet (as it will be defined in AppConfig.ready() and @app.task might appear as early as import time) but still.

I'm implementing the rest.

I'm wondering what would be the best place to put the app so it's easily importable from everywhere.