Add priority class options to high priority pods

[fosterseth]

Adds options for setting the priority class to the control plane and postgres pods
the priorityclasses are presumed to already exist

e.g.

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport
  control_plane_priority_class: awx-demo-high-priority
  postgres_priority_class: awx-demo-medium-priority

also sets default requests for the postgres pod to ensure a Burstable QoS

[rooftopcellist]

@fosterseth I noticed this PR after putting up this other PR that makes a similar change: #876

I think your default request values may be a little too high (requests are guaranteed resources, it doesn't prevent the container from using more when needed).

[rooftopcellist]

Let's sync up next week on how to merge these. Proposal: if you fold in my changes in #876 and fix the conflicts, we can just merge this and close the other.

[fosterseth]

I pulled your commit into this PR

[rooftopcellist]

Thank you @fosterseth

I will pull this down and test it out again.

[ikhan2010]

@shanemcd do you mind giving this a review when you get chance, please?

[rooftopcellist]

Here are a couple relevant snippets from my deployment:

AWX Deployment

        name: awx-ee
        resources:
          requests:
            cpu: 100m
            memory: 64Mi

        name: awx-task
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

        name: awx-web
        resources:
          requests:
            cpu: 100m
            memory: 128Mi

Postgres StatefulSet

        name: database-check # <------------- init container
        resources:
          requests:
            cpu: 10m
            memory: 64Mi

        name: postgres
        ports:
        - containerPort: 5432
          name: postgres
          protocol: TCP
        resources:
          requests:
            cpu: 10m
            memory: 64Mi
...
      resources:
        requests:
          storage: 8Gi

[rooftopcellist]

There is a typo here: control_plane_priority_class

[rooftopcellist]

It would be nice to have a section about setting these priority classes in the README.md.

[rooftopcellist]

@fosterseth This commit adds some info to the readme and fixes the typo noted above if you want to just cherry-pick it in (dd3aa99, and here is the branch: https://github.com/ansible/awx-operator/compare/devel...rooftopcellist:add-to-priorityclass-option?expand=1).
The priority classes are now properly added to the deployment and statefulset:

$ oc get deployment awx-demo -o yaml | grep priorityClassName:
            f:priorityClassName: {}
      priorityClassName: high-priority

$ oc get sts awx-demo-postgres -o yaml | grep priorityClassName:
            f:priorityClassName: {}
      priorityClassName: high-priority

[rooftopcellist]

Hmm... looks like CI is failing because the pod to run the job template is timing out:

Error creating pod: timed out waiting for the condition

This may be because of that typo. fwiw, with the commit I linked earlier, I am able to launch jobs on my deployment of awx without issue.

[rooftopcellist]

Looks like CI still failed, it is because the environment that GitHub workflows runs ion does not have a lot of resources. The new defaults the pg init container that were added were enough to make it so that the pods never got scheduled. I also reduced the requests for the other containers to avoid running into something like this in the future. Changes are in this commit - 4b6f052

[rooftopcellist]

Thank you for this contribution!