Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert orphan files deletion task to async using celery task. Update celery config to make it possible #1609

Merged
merged 7 commits into from
Oct 14, 2024

Conversation

Didayolo
Copy link
Member

@Didayolo Didayolo commented Oct 1, 2024

Original PR

Description

Make the orphan files deletion task asynchronous.
This comes with few changes to make it possible:

  • Enable the backend results of the celery tasks. This allows to store the states of any celery tasks. We use this to check is the task is running or not
  • upgrade the celery package to the closest version that fixes a known bug on incompatible version with python 3.9 that manifests only when working with celery task results
  • Fixes the patch number of the python version in the Dockerfile to match the one in the pyproject.toml file (else poetry won't be happy)
  • Add an API route to check the status of the orphan files deletion task
  • Update the frontend to reflect the status of the deletion task

For testing

Do not forget to build the docker images !

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

OhMaley and others added 2 commits September 26, 2024 17:07
…es-deletion-async

convert orphan files deletion task to async using celery task. Update celery config to make it possible
@Didayolo Didayolo self-assigned this Oct 1, 2024
@Didayolo
Copy link
Member Author

Didayolo commented Oct 1, 2024

@OhMaley Error during the building:

=> ERROR [site_worker 7/9] RUN poetry lock                                1.7s
 => CANCELED [flower 7/9] RUN poetry add redis=3.0.1                       0.7s
------
 > [site_worker 7/9] RUN poetry lock:
1.366 The currently activated Python version 3.9.19 is not supported by the project (3.9.20).
1.366 Trying to find and use a compatible version. 
1.534 
1.535 Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.
------
failed to solve: process "/bin/sh -c poetry lock" did not complete successfully: exit code: 1

Exited with code exit status 17

Any thoughts?

@OhMaley
Copy link
Collaborator

OhMaley commented Oct 1, 2024

Oui il faut que la version de python noté dans le Dockerfile et dans pyproject.toml soit identique.
Je l'ai update dans le dockerfile pour être la même version que celle du pyproject.toml, mais ce dernier fichier a été mis à jour il y a 4h on dirai. Il faut donc mettre à jour le dockerfile

@Didayolo
Copy link
Member Author

Didayolo commented Oct 1, 2024

Ce qui m'étonne c'est que je suis reparti de develop à l'instant. Mais ça devrait être bon maintenant j'imagine.

@Didayolo
Copy link
Member Author

@bbearce How to solve the poetry lock? I not able to run the commands locally

@bbearce
Copy link
Collaborator

bbearce commented Oct 14, 2024

pyenv install 3.9.20
pyenv virtualenv 3.9.20 codabench_django
pyenv activate codabench_django
# pyenv uninstall codabench_django
poetry config virtualenvs.create false
poetry config virtualenvs.in-project false
poetry install # installs from lock file

@Didayolo Didayolo merged commit 231fe22 into develop Oct 14, 2024
1 check passed
@Didayolo Didayolo deleted the feature/storage/make-orphan-files-deletion-async branch October 14, 2024 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants