Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds connection error exception to handle redis connection failure #11296

Merged
merged 2 commits into from
Sep 29, 2023

Conversation

akolson
Copy link
Member

@akolson akolson commented Sep 25, 2023

Summary

This pr adds a defensive check to ensure that Kolibri execution is halted in case it fails to connect to Redis.

References

Fixes #10551

Reviewer guidance

  • Ensure that redis is not running on any of the ports specified below
  • Run KOLIBRI_CACHE_BACKEND="redis" KOLIBRI_CACHE_LOCATION="redis:657" kolibri start on a pex file or local environment setup
  • Kolibri should not start. The logs returned should be similar to the below;
INFO     2023-09-26 21:14:03,723 Option CACHE_BACKEND in section [Cache] being overridden by environment variable KOLIBRI_CACHE_BACKEND
INFO     2023-09-26 21:14:03,723 Option CACHE_LOCATION in section [Cache] being overridden by environment variable KOLIBRI_CACHE_LOCATION
INFO     2023-09-26 21:14:03,724 Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     2023-09-26 21:14:03,724 Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
INFO     2023-09-26 21:14:03,943 Running Kolibri with the following settings: kolibri.deployment.default.settings.base
WARNING  2023-09-26 21:14:03,986 Unable to connect to Redis: Error 8 connecting to redis:657. nodename nor servname provided, or not known.
Error: Traceback (most recent call last):
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/redis/connection.py", line 492, in connect
    sock = self._connect()
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/redis/connection.py", line 519, in _connect
    for res in socket.getaddrinfo(self.host, self.port, self.socket_type,
  File "/Users/akol/.pyenv/versions/3.8.13/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/core/apps.py", line 132, in check_redis_settings
    maxmemory_policy = helper.get_maxmemory_policy()
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/core/utils/cache.py", line 51, in get_maxmemory_policy
    return self.get("maxmemory-policy", default_value="noeviction")
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/core/utils/cache.py", line 34, in get
    return self.client.config_get(key).get(key, default_value)
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/redis/client.py", line 902, in config_get
    return self.execute_command('CONFIG GET', pattern)
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/redis/client.py", line 772, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/redis/connection.py", line 1142, in get_connection
    connection.connect()
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/redis/connection.py", line 497, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 8 connecting to redis:657. nodename nor servname provided, or not known.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/utils/cli.py", line 193, in invoke
    initialize(**get_initialize_params())
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/utils/main.py", line 295, in initialize
    _setup_django()
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/utils/main.py", line 153, in _setup_django
    django.setup()
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/akol/.pyenv/versions/3.8.13/envs/kolibri-py3.8.13/lib/python3.8/site-packages/django/apps/registry.py", line 116, in populate
    app_config.ready()
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/core/apps.py", line 45, in ready
    self.check_redis_settings()
  File "/Users/akol/Desktop/leDev/kolibri/kolibri/core/apps.py", line 172, in check_redis_settings
    raise RedisConnectionError("Unable to connect to Redis: {}".format(str(e)))
kolibri.core.errors.RedisConnectionError: Unable to connect to Redis: Error 8 connecting to redis:657. nodename nor servname provided, or not known.
  • Look out for the Unable to connect to Redis... in the list of logs(RedisConnectionError should be raised)
  • Also, running KOLIBRI_CACHE_BACKEND="redis" KOLIBRI_CACHE_LOCATION="redis:657" kolibri services should yield the same results

Testing checklist

  • Contributor has fully tested the PR manually
  • If there are any front-end changes, before/after screenshots are included
  • Critical user journeys are covered by Gherkin stories
  • Critical and brittle code paths are covered by unit tests

PR process

  • PR has the correct target branch and milestone
  • PR has 'needs review' or 'work-in-progress' label
  • If PR is ready for review, a reviewer has been added. (Don't use 'Assignees')
  • If this is an important user-facing change, PR or related issue has a 'changelog' label
  • If this includes an internal dependency change, a link to the diff is provided

Reviewer checklist

  • Automated test coverage is satisfactory
  • PR is fully functional
  • PR has been tested for accessibility regressions
  • External dependency files were updated if necessary (yarn and pip)
  • Documentation is updated
  • Contributor is in AUTHORS.md

@akolson akolson added this to the 0.16 Future Patches triage milestone Sep 25, 2023
@github-actions github-actions bot added DEV: backend Python, databases, networking, filesystem... SIZE: small labels Sep 25, 2023
@akolson akolson force-pushed the halt-kolibri-startup branch 3 times, most recently from 0f61aa7 to 7f31b29 Compare September 25, 2023 13:48
@akolson akolson force-pushed the halt-kolibri-startup branch from 7f31b29 to c9efdf5 Compare September 26, 2023 16:09
@akolson akolson requested review from bjester and rtibbles September 26, 2023 18:26
@akolson akolson marked this pull request as ready for review September 26, 2023 18:30
Copy link
Member

@rtibbles rtibbles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behaviourally checks out - one non-blocking suggestion. Interested in @bjester's review in case he had anything additional in mind when filing the original issue.

kolibri/core/apps.py Outdated Show resolved Hide resolved
Copy link
Member

@rtibbles rtibbles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All comments addressed, and this seems to do what is needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DEV: backend Python, databases, networking, filesystem... SIZE: small
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants