`CREATE USER` fails in subsequent hooks if bootstrap fails after creating user #157

carlcsaposs-canonical · 2023-10-31T09:51:03Z

If CREATE USER succeeds but the bootstrap fails (in the same hook), the user credentials won't be saved to the databag.

Router currently relies on the databag to check which users it's already created. In subsequent hooks, router fails to create the user because it already exists

Steps to reproduce

Steps 3-6 https://microstack.run/#get-started
Unable to reproduce consistently (server connection must succeed for create user and then immediately fail for bootstrap)

Expected behavior

Router charm raises uncaught exception but then recovers when server is reachable

Actual behavior

Router charm gets stuck in exception after server becomes reachable

Log output

Juju debug log:
logs.txt
logs-unit.txt

unit-cinder-mysql-router-0: 09:24:06 ERROR unit.cinder-mysql-router/0.juju-log backend-database:71: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 272, in connect
    self.sock.connect(self.socket_path)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1599, in _request_raw
    response = self.opener.open(request, timeout=self.timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 286, in http_open
    return self.do_open(_UnixSocketConnection, req,  # type:ignore
  File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/./src/kubernetes_charm.py", line 177, in <module>
    ops.main.main(KubernetesRouterCharm)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 342, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 928, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1059, in _on_relation_changed_event
    getattr(self.on, "database_created").emit(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 342, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 928, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/abstract_charm.py", line 216, in reconcile
    workload_.enable(tls=self._tls_certificate_saved, unit_name=self.unit.name)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/workload.py", line 216, in enable
    self._bootstrap_router(tls=tls)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/workload.py", line 179, in _bootstrap_router
    self._container.run_mysql_router(command, timeout=30)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/container.py", line 120, in run_mysql_router
    return self._run_command(args, timeout=timeout)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/rock.py", line 119, in _run_command
    output, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1354, in wait_output
    exit_code: int = self._wait()
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1294, in _wait
    change = self._client.wait_change(self._change_id, timeout=timeout)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1794, in wait_change
    return self._wait_change_using_wait(change_id, timeout)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1815, in _wait_change_using_wait
    return self._wait_change(change_id, this_timeout)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1829, in _wait_change
    resp = self._request('GET', f'/v1/changes/{change_id}/wait', query)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1564, in _request
    response = self._request_raw(method, path, query, headers, data)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1612, in _request_raw
    raise ConnectionError(e.reason)
ops.pebble.ConnectionError: [Errno 2] No such file or directory
unit-cinder-mysql-router-0: 09:24:07 ERROR juju.worker.uniter.operation hook "backend-database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
unit-cinder-mysql-router-0: 09:24:07 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-cinder-mysql-router-0: 09:24:08 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-cinder-mysql-router-0: 09:24:12 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook
unit-cinder-mysql-router-0: 09:24:12 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: database created at 2023-10-31 08:24:12.863756
unit-cinder-mysql-router-0: 09:24:13 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/openstack/statefulsets/cinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:13 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/openstack/statefulsets/cinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:13 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/openstack/statefulsets/cinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:13 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/openstack/statefulsets/cinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:13 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/openstack/pods?labelSelector=app.kubernetes.io/name%3Dcinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:13 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/openstack/statefulsets/cinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:14 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/openstack/statefulsets/cinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:14 INFO unit.cinder-mysql-router/0.juju-log backend-database:71: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/openstack/pods?labelSelector=app.kubernetes.io/name%3Dcinder-mysql-router "HTTP/1.1 200 OK"
unit-cinder-mysql-router-0: 09:24:15 ERROR unit.cinder-mysql-router/0.juju-log backend-database:71: Failed to run logged_commands=["shell.connect('relation-71:***@mysql-primary.openstack.svc.cluster.local:3306')", 'session.run_sql("CREATE DATABASE IF NOT EXISTS `cinder`")', 'session.run_sql("CREATE USER `relation-71-35` IDENTIFIED BY \'TY9OGMnx8hDNYEaQQieGGZaz\' ATTRIBUTE \'{\\"created_by_user\\": \\"relation-71\\"}\'")', 'session.run_sql("GRANT ALL PRIVILEGES ON `cinder`.* TO `relation-71-35`")']
stderr:
Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory
Traceback (most recent call last):
  File "<string>", line 3, in <module>
mysqlsh.DBError: MySQL Error (1396): ClassicSession.run_sql: Operation CREATE USER failed for 'relation-71-35'@'%'

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/rock.py", line 119, in _run_command
    output, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1359, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlsh', '--no-wizard', '--python', '--file', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n  File "<string>", line 3, in <module>\nmysqlsh.DBError: MySQL Error (1396): ClassicSession.run_sql: Operation CREATE USER failed for \'relation-71-35\'@\'%\'\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/mysql_shell.py", line 58, in _run_commands
    output = self._container.run_mysql_shell(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/container.py", line 130, in run_mysql_shell
    return self._run_command(args, timeout=timeout)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/rock.py", line 121, in _run_command
    raise container.CalledProcessError(
container.CalledProcessError: Command '['mysqlsh', '--no-wizard', '--python', '--file', '/tmp/script.py']' returned non-zero exit status 1.
unit-cinder-mysql-router-0: 09:24:15 ERROR unit.cinder-mysql-router/0.juju-log backend-database:71: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/rock.py", line 119, in _run_command
    output, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/pebble.py", line 1359, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlsh', '--no-wizard', '--python', '--file', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n  File "<string>", line 3, in <module>\nmysqlsh.DBError: MySQL Error (1396): ClassicSession.run_sql: Operation CREATE USER failed for \'relation-71-35\'@\'%\'\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/./src/kubernetes_charm.py", line 177, in <module>
    ops.main.main(KubernetesRouterCharm)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 342, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 928, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1059, in _on_relation_changed_event
    getattr(self.on, "database_created").emit(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 342, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 839, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/venv/ops/framework.py", line 928, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/abstract_charm.py", line 209, in reconcile
    self._database_provides.reconcile_users(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/relations/database_provides.py", line 204, in reconcile_users
    relation.create_database_and_user(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/relations/database_provides.py", line 107, in create_database_and_user
    password = shell.create_application_database_and_user(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/mysql_shell.py", line 104, in create_application_database_and_user
    self._run_sql(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/mysql_shell.py", line 81, in _run_sql
    self._run_commands(commands)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/mysql_shell.py", line 58, in _run_commands
    output = self._container.run_mysql_shell(
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/container.py", line 130, in run_mysql_shell
    return self._run_command(args, timeout=timeout)
  File "/var/lib/juju/agents/unit-cinder-mysql-router-0/charm/src/rock.py", line 121, in _run_command
    raise container.CalledProcessError(
container.CalledProcessError: Command '['mysqlsh', '--no-wizard', '--python', '--file', '/tmp/script.py']' returned non-zero exit status 1.
unit-cinder-mysql-router-0: 09:24:15 ERROR juju.worker.uniter.operation hook "backend-database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

Reported by @gboutry—thank you!

The text was updated successfully, but these errors were encountered:

github-actions · 2023-10-31T09:51:20Z

https://warthogs.atlassian.net/browse/DPE-2875

Fixes #157 Previously, the charm assumed that if a user was created or deleted that the databag would be updated accordingly. However, if a user is created or deleted and the event/hook fails, the user creation/deletion will go through for MySQL but the databag will not be updated (since the event failed). When the event is retried, the charm tried to create a user that already exists or delete a user that doesn't exist

carlcsaposs-canonical added the bug Something isn't working label Oct 31, 2023

carlcsaposs-canonical mentioned this issue Nov 27, 2023

Router fails to start after machine reboot #146

Closed

carlcsaposs-canonical mentioned this issue Nov 27, 2023

Recover from hook errors when creating/deleting MySQL users #165

Merged

carlcsaposs-canonical closed this as completed in #165 Nov 28, 2023

github-actions bot added a commit to carlcsaposs-canonical/mysql-router-k8s-operator that referenced this issue Jul 16, 2024

Allure report canonical#157

e22bb08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`CREATE USER` fails in subsequent hooks if bootstrap fails after creating user #157

`CREATE USER` fails in subsequent hooks if bootstrap fails after creating user #157

carlcsaposs-canonical commented Oct 31, 2023

github-actions bot commented Oct 31, 2023

CREATE USER fails in subsequent hooks if bootstrap fails after creating user #157

CREATE USER fails in subsequent hooks if bootstrap fails after creating user #157

Comments

carlcsaposs-canonical commented Oct 31, 2023

Steps to reproduce

Expected behavior

Actual behavior

Log output

Additional context

github-actions bot commented Oct 31, 2023

`CREATE USER` fails in subsequent hooks if bootstrap fails after creating user #157

`CREATE USER` fails in subsequent hooks if bootstrap fails after creating user #157