[AAP-27692] Add middleware that apps can use to handle graceful timeout signal #581

kdelee · 2024-08-21T17:39:26Z

uwsgi 2.0.22+ offers option to send a user-defined signal at for a "graceful" timeout before the real harakiri occurs (we can configure any signal we want -- so we can go with SIGABRT)

gunicorn sends SIGABRT (signal 6) as graceful timeout signal before sending SIGKILL.

Apps can use this middleware to log a traceback indicating what code was executing when the graceful timeout signal was received.

Will use this instead of middleware defined in awx for ansible/awx#15447

kdelee · 2024-08-21T17:41:01Z

dropped introspecting into the X-REQUEST-ID header because I think if apps are following

django-ansible-base/docs/logging.md

Lines 12 to 29 in cdbea30

    
           ## Logging request IDs 
        
           django-ansible-base provides machinery (middleware and logging filters) to inject the request id into the logging format string. 
        
           The request id should come through as a header in the request, `X-Request-Id` (case-insensitive). 
        
           To enable logging of request ids, make the following changes to your application's `settings.py`: 
        
           1. Add `'ansible_base.lib.middleware.logging.LogRequestMiddleware'` to `MIDDLEWARE`. This is a middleware that simply 
        
              adds the current request object to the thread-local state, so that the logging filter (in the following steps) can 
        
              make use of it. 
        
           2. Configure `LOGGING` 
        
              a. Add the logging filter to `LOGGING['filters']` 
        
              b. Enable the filter in `LOGGING['handlers']` 
        
              c. Make use of the `request_id` parameter in `LOGGING['formatters']`

will "just work"

test_app/apps.py

AlanCoding · 2024-08-21T19:23:30Z

kdelee#1

ansible_base/lib/middleware/logging/log_request.py

AlanCoding

Looks great. Quality utility.

kdelee · 2024-08-27T14:11:52Z

re-confirmed with awx PR ansible/awx#15447

tools_awx_1       | Tue Aug 27 14:09:09 2024 - *** HARAKIRI ON WORKER 1 (pid: 2908, try: 1, graceful: yes) ***
tools_awx_1       | Tue Aug 27 14:09:09 2024 - HARAKIRI !!! worker 1 status !!!
tools_awx_1       | Tue Aug 27 14:09:09 2024 - HARAKIRI [core 0] 172.23.0.1 - GET /api/ since 1724767747
tools_awx_1       | Tue Aug 27 14:09:09 2024 - HARAKIRI !!! end of worker 1 status !!!
tools_awx_1       | Tue Aug 27 14:09:09 2024 - HARAKIRI: graceful termination attempt on worker 1 with signal 6. Next harakiri: 1724767750
tools_awx_1       | Tue Aug 27 14:09:09 2024 - HARAKIRI triggered by worker 1 core 0 !!!
tools_awx_1       | 2024-08-27 14:09:09,673 ERROR    [d459e624] ansible_base.lib.middleware.logging.log_request Received graceful timeout signal for GET
tools_awx_1       |                 path: /api/ while in stack:   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/wsgi.py", line 124, in __call__
tools_awx_1       |     response = self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/base.py", line 140, in get_response
tools_awx_1       |     response = self._middleware_chain(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/debug_toolbar/middleware.py", line 48, in __call__
tools_awx_1       |     return self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django_guid/middleware.py", line 79, in middleware
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/lib/middleware/logging/log_request.py", line 40, in __call__
tools_awx_1       |     return self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/corsheaders/middleware.py", line 56, in __call__
tools_awx_1       |     result = self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/social_django/middleware.py", line 28, in __call__
tools_awx_1       |     return self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/crum/__init__.py", line 97, in __call__
tools_awx_1       |     response = self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/deprecation.py", line 134, in __call__
tools_awx_1       |     response = response or self.get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
tools_awx_1       |     response = get_response(request)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
tools_awx_1       |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
tools_awx_1       |   File "/usr/lib64/python3.11/contextlib.py", line 81, in inner
tools_awx_1       |     return func(*args, **kwds)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
tools_awx_1       |     return view_func(*args, **kwargs)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/views/generic/base.py", line 104, in view
tools_awx_1       |     return self.dispatch(request, *args, **kwargs)
tools_awx_1       |   File "/awx_devel/awx/api/generics.py", line 351, in dispatch
tools_awx_1       |     return super(APIView, self).dispatch(request, *args, **kwargs)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
tools_awx_1       |     response = handler(request, *args, **kwargs)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/decorators.py", line 46, in _wrapper
tools_awx_1       |     return bound_method(*args, **kwargs)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/utils/decorators.py", line 134, in _wrapper_view
tools_awx_1       |     response = view_func(request, *args, **kwargs)
tools_awx_1       |   File "/awx_devel/awx/api/views/root.py", line 49, in get
tools_awx_1       |     import time; time.sleep(100)
tools_awx_1       |   File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/ansible_base/lib/middleware/logging/log_request.py", line 19, in handle_signal
tools_awx_1       |     path: {request.path} while in stack: {''.join(traceback.format_stack())}
tools_awx_1       |

uwsgi 2.0.22+ offers option to send a user-defined signal at for a "graceful" timeout before the real harakiri occurs. gunicorn sends SIGABRT (signal 6) as graceful timeout signal before sending SIGKILL Apps can use this middleware to log a traceback indicating what code was executing when the graceful timeout signal was received. log error message if trying to use middleware in incompatible environment

sonarqubecloud · 2024-08-27T16:36:38Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
87.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

kdelee force-pushed the timeout_traceback_middleware branch 3 times, most recently from 56a9e2d to fec8a71 Compare August 21, 2024 18:54

AlanCoding reviewed Aug 21, 2024

View reviewed changes

test_app/apps.py Outdated Show resolved Hide resolved

kdelee commented Aug 21, 2024

View reviewed changes

ansible_base/lib/middleware/logging/log_request.py Show resolved Hide resolved

kdelee force-pushed the timeout_traceback_middleware branch 3 times, most recently from 7f43463 to 6023e24 Compare August 21, 2024 20:21

kdelee requested a review from AlanCoding August 21, 2024 21:06

kdelee force-pushed the timeout_traceback_middleware branch 7 times, most recently from 731809d to 6fb318c Compare August 22, 2024 17:59

AlanCoding approved these changes Aug 23, 2024

View reviewed changes

AlanCoding added the app:lib label Aug 23, 2024

kdelee force-pushed the timeout_traceback_middleware branch from ae1c43e to 85d4464 Compare August 26, 2024 18:53

kdelee changed the title ~~Add middleware that apps can use to handle graceful timeout signal~~ [AAP-27692] Add middleware that apps can use to handle graceful timeout signal Aug 27, 2024

kdelee force-pushed the timeout_traceback_middleware branch 3 times, most recently from d653471 to 9ceba08 Compare August 27, 2024 14:11

kdelee force-pushed the timeout_traceback_middleware branch from 9ceba08 to cd45969 Compare August 27, 2024 14:13

kdelee force-pushed the timeout_traceback_middleware branch from cd45969 to a56bdee Compare August 27, 2024 16:29

mabashian merged commit 21c6526 into ansible:devel Aug 27, 2024
11 checks passed

kdelee deleted the timeout_traceback_middleware branch August 27, 2024 17:04

kdelee mentioned this pull request Aug 27, 2024

Catch harakiri graceful signal in middlware and log debug info ansible/awx#15447

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AAP-27692] Add middleware that apps can use to handle graceful timeout signal #581

[AAP-27692] Add middleware that apps can use to handle graceful timeout signal #581

kdelee commented Aug 21, 2024 •

edited

Loading

kdelee commented Aug 21, 2024

AlanCoding commented Aug 21, 2024

AlanCoding left a comment

kdelee commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

[AAP-27692] Add middleware that apps can use to handle graceful timeout signal #581

[AAP-27692] Add middleware that apps can use to handle graceful timeout signal #581

Conversation

kdelee commented Aug 21, 2024 • edited Loading

kdelee commented Aug 21, 2024

AlanCoding commented Aug 21, 2024

AlanCoding left a comment

Choose a reason for hiding this comment

kdelee commented Aug 27, 2024

sonarqubecloud bot commented Aug 27, 2024

Quality Gate passed

kdelee commented Aug 21, 2024 •

edited

Loading