-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubernetes CrashLoopBackOff where attempting to upgrade from mysql 5.7.x to 8.0.34 #1041
Comments
Adding more details in case it helps, Dockerfile below:
Where build_type and build_ver are arguments passed where building with Docker.. and my.cnf:
|
Verbose output below, hopefully
After the last line is when the image starts to show in kubernetes as |
Unfortunately, I don't see anything particularly interesting in those logs. Would it be possible for you to try again, but replacing |
Thanks @tianon Seems to have helped find an issue, at least!
I'm going to make sure that all those dirs are created in the Dockerfile, and will try again. |
Ok, after creating the dirs and making the prior mentioned change.. still sticking, and log below:
|
If I'm reading the entrypoint.sh right, the above means we should be good.. as we're at the end of the _main() function: https://github.com/docker-library/mysql/blob/master/docker-entrypoint.sh#L407-L411 |
I don't think it is actually stuck; it is just no longer logging to stdout becuase it was told to log to a file: $ # /var/lib/mysql/ so I didn't have non-existent directory issues
$ docker run -it --rm --name sql -e MYSQL_ROOT_PASSWORD=12345 mysql --log-error=/var/lib/mysql/mysql_error.log
2024-04-08 22:59:10+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
2024-04-08 22:59:10+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-04-08 22:59:10+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.3.0-1.el8 started.
2024-04-08 22:59:10+00:00 [Note] [Entrypoint]: Initializing database files
2024-04-08 22:59:14+00:00 [Note] [Entrypoint]: Database files initialized
2024-04-08 22:59:14+00:00 [Note] [Entrypoint]: Starting temporary server
mysqld will log errors to /var/lib/mysql/mysql_error.log
mysqld is running as pid 126
2024-04-08 22:59:14+00:00 [Note] [Entrypoint]: Temporary server started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
2024-04-08 22:59:15+00:00 [Note] [Entrypoint]: Stopping temporary server
2024-04-08 22:59:17+00:00 [Note] [Entrypoint]: Temporary server stopped
2024-04-08 22:59:17+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up. And in another terminal: $ docker exec -it sql bash
bash-4.4# cat /var/lib/mysql/mysql_error.log
2024-04-08T22:59:10.736937Z 0 [System] [MY-015017] [Server] MySQL Server Initialization - start.
2024-04-08T22:59:10.737825Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.3.0) initializing of server in progress as process 80
2024-04-08T22:59:10.742891Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-04-08T22:59:10.969314Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-04-08T22:59:11.733066Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2024-04-08T22:59:14.068064Z 0 [System] [MY-015018] [Server] MySQL Server Initialization - end.
2024-04-08T22:59:14.153071Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-04-08T22:59:14.325924Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 124
2024-04-08T22:59:14.349655Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-04-08T22:59:14.422623Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-04-08T22:59:14.600078Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-04-08T22:59:14.600094Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-04-08T22:59:14.601968Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-04-08T22:59:14.610625Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2024-04-08T22:59:14.610835Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.3.0' socket: '/var/run/mysqld/mysqld.sock' port: 0 MySQL Community Server - GPL.
2024-04-08T22:59:15.906063Z 10 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.3.0).
2024-04-08T22:59:17.192951Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.3.0) MySQL Community Server - GPL.
2024-04-08T22:59:17.192961Z 0 [System] [MY-015016] [Server] MySQL Server - end.
2024-04-08T22:59:17.913668Z 0 [System] [MY-015015] [Server] MySQL Server - start.
2024-04-08T22:59:18.091380Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.3.0) starting as process 1
2024-04-08T22:59:18.106253Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-04-08T22:59:18.177430Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-04-08T22:59:18.330705Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-04-08T22:59:18.330721Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-04-08T22:59:18.332498Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-04-08T22:59:18.342427Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2024-04-08T22:59:18.342484Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.3.0' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL. |
Thanks @yosifkit, it seems that the pod might not be running long enough for the logs to be sent to our SIEM.. I have made a change and can see the below in
I have been issuing a shutdown via mysqladmin prior, guessing this doesn't suffice. https://serverfault.com/a/1110090 advises to delete the After following the above, and restarting mysql.. the database was upgraded and all looks good! Is there a "cleaner" shutdown process? The steps at: https://dev.mysql.com/doc/refman/8.3/en/docker-mysql-getting-started.html#docker-upgrading seem to suggest that stopping the pod and starting with the new version will suffice.. but that's also 8.0 > 8.3, and not 5.7.x to 8.0.34. |
Hi folks,
We've been running various flavours of mysql 5.7.x for a few years, and now one of the items we're hosting requires 8.0.34 (prior release was 5.7.39).
In attempting to upgrade the deployment, it seems that the
/usr/local/bin/docker-entrypoint.sh
runs once as root and then attempts to run as the usermysql
.. this latter step fails.. and then we have no logging to advise why (despite passing--verbose
to/usr/local/bin/docker-entrypoint.sh
)..The
/usr/local/bin/docker-entrypoint.sh
, is loaded within our images own entrypoint and per the below:Prior to upgrading, we're gracefully shutting down mysql 5.7 via mysqladmin.
I'm just a little stumped as no real helpful logs to advise what the issue is, so help appreciated!
The text was updated successfully, but these errors were encountered: