Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Secret key generation fails when deploying via Helm on Kubernetes (AKS) #8002

Open
2 tasks done
mathijscarlu opened this issue Jun 7, 2024 · 0 comments · May be fixed by #8003
Open
2 tasks done

Secret key generation fails when deploying via Helm on Kubernetes (AKS) #8002

mathijscarlu opened this issue Jun 7, 2024 · 0 comments · May be fixed by #8003
Assignees
@azhavoro
Labels
bug Something isn't working

Comments

@mathijscarlu
Copy link

Actions before raising this issue

  • I searched the existing issues and did not find anything similar.
  • I read/searched the docs

Steps to Reproduce

  1. Deploy via Helm on Kubernetes (AKS in my case, can't speak for other cloud providers)
  2. None of the backend Pods will succeed in creating the /home/django/keys/secret_key.py file.

Expected Behavior

The first backend Pod that reaches the generate_secret_key() function should be able to create it.

Possible Solution

Manually creating the /home/django/keys/secret_key.py file solves the issue, but it would be great if there would be a solution contained in the Helm. An initContainer that creates the file, e.g.

Context

Related to #7892.

All of the backend Pods encounter the same issue, with the following stacktrace:

Traceback (most recent call last):
  File "/home/django/cvat/settings/base.py", line 77, in <module>
    from keys.secret_key import SECRET_KEY # pylint: disable=unused-import
ModuleNotFoundError: No module named 'keys.secret_key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/django/manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/opt/venv/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/opt/venv/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/venv/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/django/cvat/settings/production.py", line 5, in <module>
    from .base import *
  File "/home/django/cvat/settings/base.py", line 79, in <module>
    generate_secret_key()
  File "/home/django/cvat/settings/base.py", line 69, in generate_secret_key
    os.link(f.name, os.path.join(keys_dir, secret_key_fname))
OSError: [Errno 95] Operation not supported: '/home/django/keys/secret_key.py.qmky3wru' -> '/home/django/keys/secret_key.py'

Environment

- Image tag: v2.14.1
- Kubernetes version 1.27.9, provided by AKS, with Linux Nodes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@azhavoro azhavoro

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Generate secret key in initContainer instead of python code mathijscarlu/cvat
2 participants
@azhavoro @mathijscarlu