Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] - Internal server error (conda store server) when system metrics are not available #1028

Closed
soapy1 opened this issue Dec 19, 2024 · 3 comments · Fixed by #1029
Closed

[BUG] - Internal server error (conda store server) when system metrics are not available #1028

soapy1 opened this issue Dec 19, 2024 · 3 comments · Fixed by #1029
Labels
good first issue Good for newcomers type: bug 🐛 Something isn't working

Comments

@soapy1
Copy link
Contributor

soapy1 commented Dec 19, 2024 •
edited
Loading

Describe the bug

On a fresh instance of conda-store server (eg. with no worker having been setup), navigating to the admin console at <conda-store url>/admin/user/ results in a 500 Internal server error.

The error log from the conda-store server instance

conda-store-server-1  | ERROR:    Exception in ASGI application
conda-store-server-1  |   + Exception Group Traceback (most recent call last):
conda-store-server-1  |   |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
conda-store-server-1  |   |     yield
conda-store-server-1  |   |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 186, in __call__
conda-store-server-1  |   |     async with anyio.create_task_group() as task_group:
conda-store-server-1  |   |                ^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 815, in __aexit__
conda-store-server-1  |   |     raise BaseExceptionGroup(
conda-store-server-1  |   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
conda-store-server-1  |   +-+---------------- 1 ----------------
conda-store-server-1  |     | Traceback (most recent call last):
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
conda-store-server-1  |     |     await self.app(scope, receive, _send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
conda-store-server-1  |     |     with collapse_excgroups():
conda-store-server-1  |     |          ^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/contextlib.py", line 158, in __exit__
conda-store-server-1  |     |     self.gen.throw(value)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
conda-store-server-1  |     |     raise exc
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
conda-store-server-1  |     |     response = await self.dispatch_func(request, call_next)
conda-store-server-1  |     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda-store-server/conda_store_server/_internal/server/app.py", line 258, in conda_store_middleware
conda-store-server-1  |     |     response = await call_next(request)
conda-store-server-1  |     |                ^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
conda-store-server-1  |     |     raise app_exc
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
conda-store-server-1  |     |     await self.app(scope, receive_or_disconnect, send_no_error)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/sessions.py", line 85, in __call__
conda-store-server-1  |     |     await self.app(scope, receive, send_wrapper)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
conda-store-server-1  |     |     await self.app(scope, receive, send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
conda-store-server-1  |     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
conda-store-server-1  |     |     raise exc
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
conda-store-server-1  |     |     await app(scope, receive, sender)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
conda-store-server-1  |     |     await self.middleware_stack(scope, receive, send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
conda-store-server-1  |     |     await route.handle(scope, receive, send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
conda-store-server-1  |     |     await self.app(scope, receive, send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
conda-store-server-1  |     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
conda-store-server-1  |     |     raise exc
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
conda-store-server-1  |     |     await app(scope, receive, sender)
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
conda-store-server-1  |     |     response = await f(request)
conda-store-server-1  |     |                ^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
conda-store-server-1  |     |     raw_response = await run_endpoint_function(
conda-store-server-1  |     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
conda-store-server-1  |     |     return await dependant.call(**values)
conda-store-server-1  |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda-store-server/conda_store_server/_internal/server/views/ui.py", line 282, in ui_get_user
conda-store-server-1  |     |     return templates.TemplateResponse(request, "user.html", context)
conda-store-server-1  |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/templating.py", line 209, in TemplateResponse
conda-store-server-1  |     |     return _TemplateResponse(
conda-store-server-1  |     |            ^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/templating.py", line 40, in __init__
conda-store-server-1  |     |     content = template.render(context)
conda-store-server-1  |     |               ^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/jinja2/environment.py", line 1304, in render
conda-store-server-1  |     |     self.environment.handle_exception()
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception
conda-store-server-1  |     |     raise rewrite_traceback_stack(source=source)
conda-store-server-1  |     |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/user.html", line 7, in top-level template code
conda-store-server-1  |     |     {% extends 'base.html' %}
conda-store-server-1  |     |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/base.html", line 38, in top-level template code
conda-store-server-1  |     |     {% block content %}{% endblock %}
conda-store-server-1  |     |     ^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/user.html", line 13, in block 'content'
conda-store-server-1  |     |     {% block user %}
conda-store-server-1  |     |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/user.html", line 116, in block 'user'
conda-store-server-1  |     |     <li class="list-group-item">Storage Total<span class="badge badge-light">{{ system_metrics.disk_total | filesizeformat(true) }}</span></li>
conda-store-server-1  |     |     ^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |     |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/jinja2/filters.py", line 697, in do_filesizeformat
conda-store-server-1  |     |     bytes = float(value)
conda-store-server-1  |     |             ^^^^^^^^^^^^
conda-store-server-1  |     | jinja2.exceptions.UndefinedError: 'None' has no attribute 'disk_total'
conda-store-server-1  |     +------------------------------------
conda-store-server-1  | 
conda-store-server-1  | During handling of the above exception, another exception occurred:
conda-store-server-1  | 
conda-store-server-1  | Traceback (most recent call last):
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
conda-store-server-1  |     await self.app(scope, receive, _send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
conda-store-server-1  |     with collapse_excgroups():
conda-store-server-1  |          ^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/contextlib.py", line 158, in __exit__
conda-store-server-1  |     self.gen.throw(value)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
conda-store-server-1  |     raise exc
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
conda-store-server-1  |     response = await self.dispatch_func(request, call_next)
conda-store-server-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/app.py", line 258, in conda_store_middleware
conda-store-server-1  |     response = await call_next(request)
conda-store-server-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
conda-store-server-1  |     raise app_exc
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
conda-store-server-1  |     await self.app(scope, receive_or_disconnect, send_no_error)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/sessions.py", line 85, in __call__
conda-store-server-1  |     await self.app(scope, receive, send_wrapper)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
conda-store-server-1  |     await self.app(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
conda-store-server-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
conda-store-server-1  |     raise exc
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
conda-store-server-1  |     await app(scope, receive, sender)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
conda-store-server-1  |     await self.middleware_stack(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
conda-store-server-1  |     await route.handle(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
conda-store-server-1  |     await self.app(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
conda-store-server-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
conda-store-server-1  |     raise exc
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
conda-store-server-1  |     await app(scope, receive, sender)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
conda-store-server-1  |     response = await f(request)
conda-store-server-1  |                ^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
conda-store-server-1  |     raw_response = await run_endpoint_function(
conda-store-server-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
conda-store-server-1  |     return await dependant.call(**values)
conda-store-server-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/views/ui.py", line 282, in ui_get_user
conda-store-server-1  |     return templates.TemplateResponse(request, "user.html", context)
conda-store-server-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/templating.py", line 209, in TemplateResponse
conda-store-server-1  |     return _TemplateResponse(
conda-store-server-1  |            ^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/templating.py", line 40, in __init__
conda-store-server-1  |     content = template.render(context)
conda-store-server-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/jinja2/environment.py", line 1304, in render
conda-store-server-1  |     self.environment.handle_exception()
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception
conda-store-server-1  |     raise rewrite_traceback_stack(source=source)
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/user.html", line 7, in top-level template code
conda-store-server-1  |     {% extends 'base.html' %}
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/base.html", line 38, in top-level template code
conda-store-server-1  |     {% block content %}{% endblock %}
conda-store-server-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/user.html", line 13, in block 'content'
conda-store-server-1  |     {% block user %}
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/templates/user.html", line 116, in block 'user'
conda-store-server-1  |     <li class="list-group-item">Storage Total<span class="badge badge-light">{{ system_metrics.disk_total | filesizeformat(true) }}</span></li>
conda-store-server-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/jinja2/filters.py", line 697, in do_filesizeformat
conda-store-server-1  |     bytes = float(value)
conda-store-server-1  |             ^^^^^^^^^^^^
conda-store-server-1  | jinja2.exceptions.UndefinedError: 'None' has no attribute 'disk_total'
conda-store-server-1  | 
conda-store-server-1  | During handling of the above exception, another exception occurred:
conda-store-server-1  | 
conda-store-server-1  | Traceback (most recent call last):
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
conda-store-server-1  |     result = await app(  # type: ignore[func-returns-value]
conda-store-server-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
conda-store-server-1  |     await super().__call__(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
conda-store-server-1  |     await self.middleware_stack(scope, receive, send)
conda-store-server-1  |   File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/errors.py", line 177, in __call__
conda-store-server-1  |     response = await self.handler(request, exc)
conda-store-server-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/app.py", line 277, in exception_handler
conda-store-server-1  |     return await http_exception_handler(request, exc)
conda-store-server-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
conda-store-server-1  |   File "/opt/conda-store-server/conda_store_server/_internal/server/app.py", line 266, in http_exception_handler
conda-store-server-1  |     "message": exc.detail,
conda-store-server-1  |                ^^^^^^^^^^
conda-store-server-1  | AttributeError: 'UndefinedError' object has no attribute 'detail'

This error shows that the error comes from getting the system metrics. When conda-store does not find what it expects to exist in the database it errors out.

This is because the conda_store_configuration table gets populated initially by the conda-store worker. So, on a fresh install, if no worker has been started, this table will not be populated.

Expected behavior

Conda store server should render the page <conda-store url>/admin/user/ without the missing system metrics.

How to Reproduce the problem?

To reproduce the problem using docker-compose you need a fresh install of conda-store.

0. reset the db if you have an existing docker compose setup

If you previously were using docker-compose to work on conda-store be sure to clear the db

$ docker exec -it <id of the postgres container> psql
# \c conda-store
# drop database "conda-store"
# create database "conda-store"

1. bring up the conda-store services

now start the conda-store services, except for the conda-store-worker

$ docker compose up -d postgres redis minio conda-store-server

2. log in as the admin user

navigate to http://localhost:8080/conda-store/login/?next=http://localhost:8080/conda-store/ui/
log in as the admin user with the credentials admin/password

3. navigate to the admin panel

go to the url http://localhost:8080/conda-store/admin/user/ and notice the internal server error.
@soapy1 soapy1 added good first issue Good for newcomers type: bug 🐛 Something isn't working labels Dec 19, 2024
@imprvhub
Copy link
Contributor

Hey @soapy1 can i be assigned to this? I would like to contribute.

@soapy1
Copy link
Contributor Author

soapy1 commented Dec 19, 2024

Sounds great, it's all yours! Feel free to update the assignee section of the issue.

Let me know if you need any clarifications about the bug/how to reproduce it.

imprvhub added a commit to imprvhub/conda-store that referenced this issue Dec 19, 2024
@imprvhub
fix: handle None system metrics in user admin panel
2f825e2 
- Add default values for system metrics when None
- Add conditional system metrics display in template
- Show informative message when metrics unavailable

Closes conda-incubator#1028
@imprvhub imprvhub mentioned this issue Dec 19, 2024
Merged
@imprvhub
Copy link
Contributor

@soapy1 this fix worked for me.

Image

If there's anything else you'd like me to review or adjust, please let me know. 👍

imprvhub added a commit to imprvhub/conda-store that referenced this issue Dec 19, 2024
@imprvhub
fix: handle None system metrics in admin panel
571b436 
- Return raw system_metrics from API without default values
- Update user.html template to check for None metrics
- Display info message when metrics not available

Closes conda-incubator#1028
@github-project-automation github-project-automation bot moved this from New 🚦 to Done 💪🏾 in conda-store 🐍 Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers type: bug 🐛 Something isn't working
Projects
conda-store 🐍
Status: Done 💪🏾
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: handle None system metrics in user admin panel imprvhub/conda-store
2 participants
@soapy1 @imprvhub