After celery lib upgrade to 5.3.3, get sql exception. #8470

Pokerface9711 · 2023-09-01T02:14:57Z

Checklist

I have verified that the issue exists against the main branch of Celery.
This has already been asked to the discussions forum first.
I have read the relevant section in the
contribution guide
on reporting bugs.
I have checked the issues list
for similar or identical bug reports.
I have checked the pull requests list
for existing proposed fixes.
I have checked the commit log
to find out if the bug was already fixed in the main branch.
I have included all related issues and possible duplicate issues
in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.
(if you are not able to do this, then at least specify the Celery
version affected).
I have verified that the issue exists against the main branch of Celery.
I have included the contents of pip freeze in the issue.
I have included all the versions of all the external dependencies required
to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version
and/or implementation.
I have tried reproducing the issue on more than one message broker and/or
result backend.
I have tried reproducing the issue on more than one version of the message
broker and/or result backend.
I have tried reproducing the issue on more than one operating system.
I have tried reproducing the issue on more than one workers pool.
I have tried reproducing the issue with autoscaling, retries,
ETA/Countdown & rate limits disabled.
I have tried reproducing the issue after downgrading
and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown
Minimal Celery Version: N/A or Unknown
Minimal Kombu Version: N/A or Unknown
Minimal Broker Version: N/A or Unknown
Minimal Result Backend Version: N/A or Unknown
Minimal OS and/or Kernel Version: N/A or Unknown
Minimal Broker Client Version: N/A or Unknown
Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/billiard/pool.py", line 1793, in safe_apply_callback
fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/worker/request.py", line 624, in on_failure
self.task.backend.mark_as_failure(
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 167, in mark_as_failure
self.store_result(task_id, exc, state,
File "/usr/local/lib/python3.8/site-packages/celery/backends/base.py", line 526, in store_result
self._store_result(task_id, result, state, traceback,
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/init.py", line 47, in _inner
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/backends/database/init.py", line 117, in _store_result
task = list(session.query(self.task_cls).filter(self.task_cls.task_id == task_id))
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2861, in iter
result = self._iter()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2875, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
return self._execute_internal(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2124, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/context.py", line 252, in orm_execute_statement
result = conn.execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1414, in execute
return meth(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 489, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1638, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
return self._exec_single_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1983, in _exec_single_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2325, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1964, in _exec_single_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 747, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ..._taskmeta.traceback AS celery_taskmeta_traceback, celery_tas...
^

[SQL: SELECT celery_taskmeta.id AS celery_taskmeta_id, celery_taskmeta.task_id AS celery_taskmeta_task_id, celery_taskmeta.status AS celery_taskmeta_status, celery_taskmeta.result AS celery_taskmeta_result, celery_taskmeta.date_done AS celery_taskmeta_date_done, celery_taskmeta.traceback AS celery_taskmeta_traceback, celery_taskmeta.children AS celery_taskmeta_children
FROM celery_taskmeta
WHERE celery_taskmeta.task_id = %(task_id_1)s]
[parameters: {'task_id_1': 'b9e469f3-e53c-4243-92aa-adeaaf5a9757'}]
(Background on this error at: https://sqlalche.me/e/20/f405)

The text was updated successfully, but these errors were encountered:

Pokerface9711 · 2023-09-01T05:36:35Z

Rollback to 5.3.1 version resolved the exception.

auvipy · 2023-09-01T08:24:39Z

UndefinedColumn can be seen, can you check the logs to find what commit caused this?

Nusnus · 2023-09-01T08:28:58Z

Maybe this? #8338
Due to error:

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ..._taskmeta.traceback AS celery_taskmeta_traceback, celery_tas...

auvipy · 2023-09-01T08:31:45Z

since that introduced a new model field, that field need to be applied on DB as well! missing migrations strikes here most probably!

Nusnus · 2023-09-01T08:36:16Z

since that introduced a new model field, that field need to be applied on DB as well! missing migrations strikes here most probably!

This is a bit of a big deal, I agree.
If we can't resolve this very soon I think we'll need to revert the other PR and release another version as this may cause a lot of issues! @auvipy

Nusnus · 2023-09-01T08:37:09Z

@Pokerface9711 Thank you for the report! ❤️

auvipy · 2023-09-01T08:41:31Z

I will handle that. lets wait

ldacey · 2023-09-02T00:34:00Z

I ran into this with Airflow 2.7.0 when I updated without using constraints:

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column celery_taskmeta.children does not exist
LINE 1: ...ry_taskmeta.date_done, celery_taskmeta.traceback, celery_tas...
                                                             ^
[SQL: SELECT celery_taskmeta.id, celery_taskmeta.task_id, celery_taskmeta.status, celery_taskmeta.result, celery_taskmeta.date_done, celery_taskmeta.traceback, celery_taskmeta.children 
FROM celery_taskmeta

potiuk · 2023-09-02T08:14:13Z

Question - we are about to release Airflow 2.7.1 with limitted constraints and we will exclude both 5.3.2, 5.3.3 from our celery provider requirements - assuming that it will be fixed by added migrations to celery in 5.3.4.

Is it a sound assumption :)?

auvipy · 2023-09-02T10:03:30Z

Question - we are about to release Airflow 2.7.1 with limitted constraints and we will exclude both 5.3.2, 5.3.3 from our celery provider requirements - assuming that it will be fixed by added migrations to celery in 5.3.4.

Is it a sound assumption :)?

I would suggest sticking with celery 5.3.1 for that and wait for a new celery point release with proper schema migrations support!

potiuk · 2023-09-03T06:45:24Z

I would suggest sticking with celery 5.3.1 for that and wait for a new celery point release with proper schema migrations support!

Slightly off-topic but FYI - Just in case you might be interested how it works for us @auvipy: this is what effectively we did by setting our !=5.3.2, !=5.3.3 limits.

The way how airflow dependencies work - we do not "pin" dependencies (because airflow is both library and applicaiton) - we produce constraint files that can be use to make sure the installation is reproducible, but we do not limit the dependencies to be upgraded separately (for example to make it possible to update to newer versions with bugfixes and new features and security fixes). We have a complete mechanism to automatically build the set of constraints for airflow when released. Any by excluding 5.3.2 and 5.3.3 our constraints automatically landed at 5.3.1. See main and 2.7 branch constraints automatically pushed by our CI. Also we realised that the sqlalchemy configuration of celery is not (yet) "integration tested" by Airflow and we added a task for that apache/airflow#34032 otherwise we would have detected this isssue a bit earlier (basically right after release).

but whenever you release 5.3.4 we will run our tests and when they pass we will upgrade our constraints to 5.3.4 (all fully automated). We have 650 transitive dependencies in total,so we had to automate it :) .

We very rarely upper-bind the releases, only when we know that they can cause big issues (for example we now have celery <6 because we know celery generally only introduce breaking changes in major version releases and those are really breaking - we had to deliberately made efforts to make Airlfow Celery 5-compatible and that's cool).

Also as discussed in apache/airflow#34031 (comment) PyPI has protections against re-releasing already released versions (so celery will have to go with 5.3.4 effectively) so - all is cool I think for us.

So effectively: we are sticking with 5.3.1 as you suggested, while we are stil open to upgrade to new versions of celery automaticaly when it happens.

auvipy · 2023-09-03T07:06:12Z

yes I learnt from PyPI today that we can't release a same version again! we yanked them and will release 5.3.4 very soon.

miketheman · 2023-09-03T22:21:57Z

we yanked them ...

To clarify - yanking makes the files remain available on PyPI for anyone explicitly wanting them, and installaser are meant to ignore yanked versions unless made explicit via == or a lockfile. See https://pypi.org/help/#yanked

5.3.2 and 5.3.3 were deleted from PyPI - so anyone who opted in to using these would now have a broken dependency chain.

Thanks for performing new releases with fixes in 5.3.4, but something to consider in the future.

auvipy · 2023-09-04T06:00:29Z

we yanked them ...

To clarify - yanking makes the files remain available on PyPI for anyone explicitly wanting them, and installaser are meant to ignore yanked versions unless made explicit via == or a lockfile. See https://pypi.org/help/#yanked

5.3.2 and 5.3.3 were deleted from PyPI - so anyone who opted in to using these would now have a broken dependency chain.

Thanks for performing new releases with fixes in 5.3.4, but something to consider in the future.

yes that is what we learned yesterday and will follow along.

auvipy · 2023-11-13T13:17:56Z

this should be fixed in 5.3.4/5.3.5 release already, right? @Nusnus we did some revert

quack39 · 2024-11-11T14:52:30Z

could this problem be related to this one?
42136

Pokerface9711 added the Issue Type: Bug Report label Sep 1, 2023

auvipy added the Issue Type: Regression label Sep 1, 2023

auvipy added this to the 5.3.x milestone Sep 1, 2023

auvipy mentioned this issue Sep 1, 2023

store children with database backend #8338

Merged

potiuk mentioned this issue Sep 2, 2023

Limit celery by excluding 5.3.2 and 5.3.3 apache/airflow#34031

Merged

potiuk mentioned this issue Sep 2, 2023

Split Celery integration tests in to "redis", "rabbitmq", "sqlalchemy" apache/airflow#34032

Closed

1 task

potiuk mentioned this issue Sep 3, 2023

Unable to read or write logs on Azure Blob after 2.7.0 update apache/airflow#34022

Closed

2 tasks

auvipy added this to v5.3 Nov 13, 2023

auvipy closed this as completed Nov 16, 2023

github-project-automation bot moved this to Done in v5.3 Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After celery lib upgrade to 5.3.3, get sql exception. #8470

After celery lib upgrade to 5.3.3, get sql exception. #8470

Pokerface9711 commented Sep 1, 2023 •

edited

Loading

Pokerface9711 commented Sep 1, 2023

auvipy commented Sep 1, 2023

Nusnus commented Sep 1, 2023 •

edited

Loading

auvipy commented Sep 1, 2023

Nusnus commented Sep 1, 2023

Nusnus commented Sep 1, 2023

auvipy commented Sep 1, 2023

ldacey commented Sep 2, 2023

potiuk commented Sep 2, 2023

auvipy commented Sep 2, 2023

potiuk commented Sep 3, 2023

auvipy commented Sep 3, 2023

miketheman commented Sep 3, 2023

auvipy commented Sep 4, 2023

auvipy commented Nov 13, 2023

quack39 commented Nov 11, 2024

After celery lib upgrade to 5.3.3, get sql exception. #8470

After celery lib upgrade to 5.3.3, get sql exception. #8470

Comments

Pokerface9711 commented Sep 1, 2023 • edited Loading

Checklist

Mandatory Debugging Information

Optional Debugging Information

Related Issues and Possible Duplicates

Related Issues

Possible Duplicates

Environment & Settings

Steps to Reproduce

Required Dependencies

Python Packages

Other Dependencies

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior

Pokerface9711 commented Sep 1, 2023

auvipy commented Sep 1, 2023

Nusnus commented Sep 1, 2023 • edited Loading

auvipy commented Sep 1, 2023

Nusnus commented Sep 1, 2023

Nusnus commented Sep 1, 2023

auvipy commented Sep 1, 2023

ldacey commented Sep 2, 2023

potiuk commented Sep 2, 2023

auvipy commented Sep 2, 2023

potiuk commented Sep 3, 2023

auvipy commented Sep 3, 2023

miketheman commented Sep 3, 2023

auvipy commented Sep 4, 2023

auvipy commented Nov 13, 2023

quack39 commented Nov 11, 2024

Pokerface9711 commented Sep 1, 2023 •

edited

Loading

Nusnus commented Sep 1, 2023 •

edited

Loading