add uwsgi config as configmap with vars

[kdelee]

Before this, uwsgi config was written into the container at build time and there was no way to customize config at deployment time. (see https://github.com/ansible/awx/blob/8cfb704f86af3f95fcd3926ffe1729cf51aef4b8/tools/ansible/roles/dockerfile/templates/Dockerfile.j2#L207 )

We have been experimenting with good results customizations to the uwsgi config in other deployments, particularly around the listen queue which allows uwsgi to build up a backlog of unhandled requests. This allows us to deal with bursts of requests better without dropping everything that exceeds the uwsgi listen queue length.

SUMMARY

Make uwsgi config mounted at deploy time so we can edit some values w/o building new image

ISSUE TYPE

• Bug, Docs Fix or other nominal change

ADDITIONAL INFORMATION

Doesn't break anything or really introduce anything "new", just surfaces config in a way that it is now editable at deploytime -- whereas previously you would need to build a new AWX container image to get uwsgi config updates

[kdelee]

Ok, I've tested this now, both with default values and with custom values.
One thing to keep in mind, is that

1. uwsgi listen parameter has to be <= net.core.somaxconn
2. net.core.somaxconn can be set on a pod-by-pod basis -- but must be allowed in the kubelet config
3. If it is not allowed, and you set the variable here uwsgi_listen_queue_size to a value greater than the default of most kernels (128), you will get an error at pod creation time that says setting the net.core.somaxconn is not allowd by the kubelet and must be allowed to allowlist.

See https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod
Also see https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#enabling-unsafe-sysctls for how

How I tested:

1. Create minikube cluster w/ somaxconn allowed sysctl

minikube start     --driver=docker     --cpus=4     --memory=8g     --addons=ingress --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn"

2. Build and deploy operator from my branch

IMG=quay.io/kdelee/awx-operator:uwsgi-config2 make docker-build                                                                                
docker push quay.io/kdelee/awx-operator:uwsgi-config2
IMG=quay.io/kdelee/awx-operator:uwsgi-config2 make deploy

3. Create awx with following CR:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx 
  namespace: awx
spec:
  service_type: nodeport
  uwsgi_listen_queue_size: 1024
  uwsgi_processes: 10

4. Confirm configmap has templated values:
   

5. Confirm securityContext has somaxconn setting on awx-web container:
   

6. Confirm awx web running

minikube service awx-service --url -n awx

Visited in web browser and it was live

7. Confirm uwsgi config correctly mounted in container:

$ kubectl exec -it awx-web-6fdd875668-84wvk -n awx bash -c awx-web
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-5.1$ cat /etc/tower/uwsgi.ini 
[uwsgi]
socket = 127.0.0.1:8050
processes = 10
listen = 1024
master = true
vacuum = true
no-orphans = true
lazy-apps = true
manage-script-name = true
master-fifo = /var/lib/awx/awxfifo
max-requests = 1000
buffer-size = 32768

if-env = UWSGI_MOUNT_PATH
mount = %(_)=awx.wsgi:application
endif =

if-not-env = UWSGI_MOUNT_PATH
mount = /=awx.wsgi:application
endif =

Final comments

So the way we have it, either:

• do not set/all behavior is equivalent to what we have today
• if you DO set values and set listen queue to > 128, you need to have set your kubelet to allow the sysctls or you will get an error.

There is a 3rd scenario which is, you are running a kernel with a higher default or have manually changed somaxconn on node level to higher value and DON'T want to allow the sysctls setting on pod level. In this case, it would be conceivable that you don't need to set the somaxconn on the pod level. If we want to cater to this scenario we maybe need another variable "set_pod_somaxconn" with default to true and allow people to disable. But this is getting really nitpicky and I think its better we wait until someone actually asks.

[kdelee]

pushed another commit to also up the values in the nginx config if the uwsgi settings are set to higher values than the ngninx defaults

Confirmed this rendered correctly:

worker_processes  1;
pid        /tmp/nginx.pid;

events {
    worker_connections 4096;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;
    client_max_body_size 5M;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
    }

    upstream daphne {
        server 127.0.0.1:8051;
    }


    
    server {
                listen 8052 default_server backlog=4096;
                    listen [::]:8052 default_server backlog=4096;
                
        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";
        # Protect against MIME content sniffing https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options
        add_header X-Content-Type-Options nosniff;

        location /nginx_status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            deny all;
        }

        location /static {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico {
            alias /var/lib/awx/public/static/media/favicon.ico;
        }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;
            include /etc/nginx/conf.d/*.conf;
            proxy_set_header X-Forwarded-Port 443;
            uwsgi_param HTTP_X_FORWARDED_PORT 443;

            add_header Strict-Transport-Security max-age=15768000;
            # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
            add_header X-Frame-Options "DENY";
            add_header X-Content-Type-Options nosniff;
            add_header Cache-Control "no-cache, no-store, must-revalidate";
            add_header Expires "0";
            add_header Pragma "no-cache";
        }
    }
}

Then I ran some tests using this web load frame work k6 that essentially loops hitting api/v2/me with the admin users with a ramping # of concurrent clients.

With default config, see variety of errors such as:
"1024 worker_connections are not enough while connecting to upstream" which is when nginx doesnt have enough workers and " uWSGI listen queue of socket ....(omitted).... full !!!" around 400 to 500 concurrent users (client receives variety of 502, 504 and broken connection errors)

With this config and CR such as:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx 
  namespace: awx
spec:
  service_type: nodeport
  uwsgi_listen_queue_size: 4096
  uwsgi_processes: 10

able to handle constant load of 1000 users hammering /api/v2/me without errors (single web pod)

[rebeccahhh]

looks good

[rooftopcellist]

@kdelee Do you mind adding an entry in the README.md for this with an example?

[kdelee]

@rooftopcellist added some information in the README

@Zokormazo I've tested the following three configurations:

1. Set all the vars (confirm correctly added to CRD etc)

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  uwsgi_processes: 10
  uwsgi_listen_queue_size: 512
  nginx_worker_processes: 5
  nginx_worker_connections: 512
  nginx_worker_cpu_affinity: 'auto'
  nginx_listen_queue_size: 512

2. Set only uwsgi vars

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  uwsgi_processes: 10
  uwsgi_listen_queue_size: 4096

3. Don't set anything, all defaults

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport

[rooftopcellist]

Wonderful, thank you @kdelee for all of the hard work here! And thank you for all of the details you have included on this PR, that helps greatly, not only for review, but also for users and future contributors.

[rooftopcellist]

😍