[10.x] Add new error messages for detecting lost connections #47398
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ever since I moved from AWS Aurora to using AWS RDS Proxy, which (similar to pgbouncer) is essential to not overload the primary database with connection, I sporadically see these messages. This affects scheduled tasks, worker, HTTP requests; it's across the whole board. This happened across different PHP versions (7.4, 8.2) and different docker base OS images, as well as different Laravel version (8.x, 10.x).
GrahamCampbell
requested changes
Jun 10, 2023
| @@ -63,6 +63,8 @@ protected function causedByLostConnection(Throwable $e) | |||
| 'Reason: Server is in script upgrade mode. Only administrator can connect at this time.', | |||
| 'Unknown $curl_error_code: 77', | |||
| 'SSL: Handshake timed out', | |||
| 'SQLSTATE[08006] [7] SSL error: sslv3 alert unexpected message', | |||
There was a problem hiding this comment.
sslv3 😭 do we really want to support use of SSL v3? Do any modern databases actually still support this?
There was a problem hiding this comment.
AWS Aurora to using AWS RDS Proxy
Seems like a configuration error perhaps? Can you try setting the minimum TLS version to 1.2 in the parameter group?
There was a problem hiding this comment.
This is just the error message being received here; all connections are at minimum TLS 1.2 I can proof this with a query like this:
SELECT * FROM pg_stat_activity JOIN pg_stat_ssl USING(pid);
and the output of any Laravel initiated connection looks like this:
…
ssl | t
version | TLSv1.2
cipher | AES128-SHA256
bits | 128
…
We can change it to SQLSTATE[08006] [7] SSL error: if that makes more sense; I just never had any other message then the two I added in this PR.
taylorotwell
pushed a commit
that referenced
this pull request
Dec 8, 2024
Last year I added those two via #47398 Please see details and reasoning there, it still applies; TL;DR: > Ever since I moved from AWS Aurora to using AWS RDS Proxy, which > (similar to pgbouncer) is essential to not overload the primary database > with connection, I sporadically see these messages. This affects > scheduled tasks, worker, HTTP requests; it's across the whole board. What I didn't knew back then was that the order/format of those messages very much dependent on the underlying libraries (libpq and or libssl presumably) and once we started to upgrade from Debian bullseye to bookworm, they changed and now are randomly interrupting our service again. The new (complete) messages look like this: > `SQLSTATE[08006] [7] connection to server at "…" (…), port 5… failed: SSL error: sslv3 alert unexpected message (Connection: main, SQL: …` That is, it still contains "SSL error: sslv3 alert unexpected message" but there's now connection specific information before the SQLSTATE prefix. Since lost connection detection is based on exact string matching and not regex, we're just removing the magic numbers in front and still keep backwards compatibility. I opened this PR against Laravel 10.x because we're still using and it requires the fix there (we're in the progress moving to L11, but it still takes time). Thanks
taylorotwell
pushed a commit
to illuminate/database
that referenced
this pull request
Dec 8, 2024
Last year I added those two via laravel/framework#47398 Please see details and reasoning there, it still applies; TL;DR: > Ever since I moved from AWS Aurora to using AWS RDS Proxy, which > (similar to pgbouncer) is essential to not overload the primary database > with connection, I sporadically see these messages. This affects > scheduled tasks, worker, HTTP requests; it's across the whole board. What I didn't knew back then was that the order/format of those messages very much dependent on the underlying libraries (libpq and or libssl presumably) and once we started to upgrade from Debian bullseye to bookworm, they changed and now are randomly interrupting our service again. The new (complete) messages look like this: > `SQLSTATE[08006] [7] connection to server at "…" (…), port 5… failed: SSL error: sslv3 alert unexpected message (Connection: main, SQL: …` That is, it still contains "SSL error: sslv3 alert unexpected message" but there's now connection specific information before the SQLSTATE prefix. Since lost connection detection is based on exact string matching and not regex, we're just removing the magic numbers in front and still keep backwards compatibility. I opened this PR against Laravel 10.x because we're still using and it requires the fix there (we're in the progress moving to L11, but it still takes time). Thanks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Ever since I moved from AWS Aurora to using AWS RDS Proxy, which (similar to pgbouncer) is essential to not overload the primary database with connection, I sporadically see these messages. This affects scheduled tasks, worker, HTTP requests; it's across the whole board.
This happened across different PHP versions (7.4, 8.2) and different
docker base OS images, as well as different Laravel version (8.x, 10.x).
The message for
SQLSTATE[08006] [7] unrecognized SSL error code:is actually
SQLSTATE[08006] [7] unrecognized SSL error code: 6but I figured this minor error code shouldn't be relevant.
I also saw other messages starting with
SQLSTATE[08006] [7], so maybe there's a case for just adding this prefix, but I guess no one knows if this would really apply to all of such messages.