CASMPET-6915: use newer plugin image and change it to daemonset

[bo-quan]

Summary and Scope

Due to limitations of server-side load balancing in kubernetes, especially with OPA as it uses GRPC protocol leveraging persistent connections, we often run into situations where only 1 or 2 OPA ingressgateway pods are used. This has exposed OPA memory leakage bug found in older OPA envoy plugin versions. This PR attempts to address the issue by changing OPA deployment to daemonset and uses a kubernetes beta feature that improves load balancing, in addition to using a newer OPA envoy plugin version v0.62.0 that has fixes for a memory leakage issue (open-policy-agent/opa#5320).

Issues and Related PRs

List and characterize relationship to Jira/Github issues and other pull requests. Be sure to list dependencies.

• Resolves CASMPET-6915
• Change will also be needed in CSM v1.5.1
• Future work required by [issue id](issue link)
• Documentation changes required in [issue id](issue link)
• Merge with/before/after <insert PR URL here>

Testing

List the environments in which these changes were tested.

Tested on:

• beau
• Local development environment
• Virtual Shasta

Test description:

How were the changes tested and success verified? If schema changes were part of this change, how were those handled in your upgrade/downgrade testing?

• Were the install/upgrade-based validation checks/tests run (goss tests/install-validation doc)?
• Were continuous integration tests run? If not, why?
• Was upgrade tested? If not, why?
• Was downgrade tested? If not, why?
• Were new tests (or test issues/Jiras) created for this change?

Risks and Mitigations

Low.

Pull Request Checklist

• Version number(s) incremented, if applicable
• Copyrights updated
• License file intact
• Target branch correct
• CHANGELOG.md updated
• Testing is appropriate and complete, if applicable
• HPC Product Announcement prepared, if applicable