Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database queue worker doesn't quit on internal SQL error #48486

Closed
mantas-done opened this issue Sep 21, 2023 · 1 comment · Fixed by #48937
Closed

Database queue worker doesn't quit on internal SQL error #48486

mantas-done opened this issue Sep 21, 2023 · 1 comment · Fixed by #48937

Comments

@mantas-done
Copy link

mantas-done commented Sep 21, 2023

Laravel Version

10.24.0

PHP Version

8.2.8

Database Driver & Version

8.0.31-0ubuntu0.20.04.2 (Homestead)

Description

I am using the Amazon RDS database. When the master database fails over to slave, the master database is set by Amazon to read-only mode.
I am running queue workers with a database driver. Queue worker gets the exception that it can't write to the database anymore. The queue worker doesn't quit, it just retries the same query.
If the queue worker would quit, the supervisor would start a new one and it would get the new MySQL connection that is writable.

Steps To Reproduce

  1. write at least one queue job to the jobs table
  2. set MySQL into read-only mode
    DB::statement("ALTER DATABASE database_name READ ONLY = 1;");
  3. run
    artisan queue:work

What is the problem?
In the Laravel logs, I get this error:
[2023-09-21 14:04:22] local.ERROR: SQLSTATE[HY000]: General error: 3989 Schema 'go' is in read-only mode. (Connection: MySQL, SQL: select * from jobs where queue = default and ((reserved_at is null and available_at <= 1695305061) or (reserved_at <= 1695294261)) order by id asc limit 1 FOR UPDATE SKIP LOCKED)...
And the worker doesn't quit after the error. And keeps retrying the same query indefinitely not processing any jobs.

What do I expect?
After such an error I expect a worker to quit.

It is probably not a very big problem, because queue jobs can be run not in "daemon" mode. Which avoids this problem.

@mantas-done mantas-done changed the title Database queue worker doesn't quit on SQL error Database queue worker doesn't quit on internal SQL error Sep 21, 2023
@github-actions
Copy link

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants