feat(insights): optionally deploy a proxy for Insights

[ebaron]

Welcome to Cryostat! 👋

Before contributing, make sure you have:

• Read the contributing guidelines
• Linked a relevant issue which this PR resolves
• Linked any other relevant issues, PR's, or documentation, if any
• Resolved all conflicts, if any
• Rebased your branch PR on top of the latest upstream main branch
• Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
• Signed all commits: git commit -S -m "YOUR_COMMIT_MESSAGE"

---

Related to: https://github.com/cryostatio/cryostat/issues/1763

Description of the change:

This change is expected to do nothing unless the environment variable INSIGHTS_ENABLED is set to true. If the variable is set to true, then the operator takes several actions:

1. Creates a secret containing an APICast configuration containing the cloud.openshift.com authentication token from the global pull secret.
2. Creates a corresponding APICast deployment and service that appends an authentication header using the credentials from the previous step, before forwarding requests to the Insights server.
3. Configures all Cryostat deployments to find the APICast proxy at the URL within the INSIGHTS_PROXY environment variable.

Motivation for the change:

Red Hat customers will automatically be able to integrate Red Hat Insights with their workloads by using the Cryostat Operator, Cryostat, and Cryostat Agent, from the downstream Red Hat build of Cryostat.

How to manually test:

1. make oci-build SKIP_TESTS=true ENABLE_INSIGHTS=true OPERATOR_IMG=quay.io/ebaron/cryostat-operator:insights-test-11
2. make deploy ENABLE_INSIGHTS=true OPERATOR_IMG=quay.io/ebaron/cryostat-operator:insights-test-11
3. Observe proxy deployment, service, and APICast config secret in cryostat-operator-system namespace
4. make cryostat_cr
5. Cryostat deployment should have INSIGHTS_PROXY set to http://insights-proxy.cryostat-operator-system.svc.cluster.local.
6. When deployed without ENABLE_INSIGHTS=true, none of the objects from step 3 should be present, and the INSIGHTS_PROXY environment variable should not be set.

[andrewazores]

Looks like it makes sense from a cursory reading - just checking that it lines up with my rough prototyping work.

[andrewazores]

/build_test

[andrewazores]

https://github.com/cryostatio/cryostat-operator/actions/runs/6776793816

[andrewazores]

I see errors like this in the log:

2023-11-07T18:52:36Z INFO controllers.Insights Reconciling Insights Proxy {"Request.Namespace": "insights-proxy", "Request.Name": "insights-proxy"}
2023-11-07T18:52:36Z INFO Starting workers {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "worker count": 1}
2023-11-07T18:52:36Z INFO Starting workers {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "worker count": 1}
2023-11-07T18:52:36Z INFO controllers.Insights Secret created {"name": "apicastconf", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Deployment created {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Service created {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Reconciling Insights Proxy {"Request.Namespace": "insights-proxy", "Request.Name": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Secret updated {"name": "apicastconf", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Deployment updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Service updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Reconciling Insights Proxy {"Request.Namespace": "insights-proxy", "Request.Name": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Secret updated {"name": "apicastconf", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z ERROR Reconciler error {"controller": "insights", "namespace": "insights-proxy", "name": "insights-proxy", "reconcileID": "9e7f8f65-c596-4cd7-8248-8ebac6b1e724", "error": "Operation cannot be fulfilled on deployments.apps \"insights-proxy\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-11-07T18:52:36Z INFO controllers.Insights Reconciling Insights Proxy {"Request.Namespace": "insights-proxy", "Request.Name": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Secret updated {"name": "apicastconf", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Deployment updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Service updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Reconciling Insights Proxy {"Request.Namespace": "insights-proxy", "Request.Name": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Secret updated {"name": "apicastconf", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Deployment updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:52:36Z INFO controllers.Insights Service updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:53:06Z INFO controllers.Insights Reconciling Insights Proxy {"Request.Namespace": "insights-proxy", "Request.Name": "insights-proxy"}
2023-11-07T18:53:06Z INFO controllers.Insights Secret updated {"name": "apicastconf", "namespace": "insights-proxy"}
2023-11-07T18:53:06Z INFO controllers.Insights Deployment updated {"name": "insights-proxy", "namespace": "insights-proxy"}
2023-11-07T18:53:06Z INFO controllers.Insights Service updated {"name": "insights-proxy", "namespace": "insights-proxy"}

but the Insights Proxy Deployment and Service still get created, so maybe they're harmless. I tried this both with an insights-proxy namespace as well as cryostat-operator-system. The following is the complete log output from the operator controller in cryostat-operator-system after deploying it, waiting for the Insights Proxy stuff to come up, and then make create_cryostat_cr:

I1107 18:55:25.542231 1 request.go:690] Waited for 1.044843786s due to client-side throttling, not priority and fairness, request: GET:https://10.217.4.1:443/apis/machine.openshift.io/v1beta1?timeout=32s
2023-11-07T18:55:26Z INFO setup detected OpenShift environment
2023-11-07T18:55:26Z INFO setup found cert-manager installation
2023-11-07T18:55:26Z INFO setup Config Map for Insights created {"name": "insights-proxy", "namespace": "cryostat-operator-system"}
2023-11-07T18:55:26Z INFO setup starting manager
2023-11-07T18:55:26Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
I1107 18:55:26.659742 1 leaderelection.go:248] attempting to acquire leader lease cryostat-operator-system/d696d7ab.redhat.com...
I1107 18:55:26.668700 1 leaderelection.go:258] successfully acquired lease cryostat-operator-system/d696d7ab.redhat.com
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1beta1.Cryostat"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Deployment"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Service"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Secret"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "insights", "source": "kind source: *v1.Secret"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "insights", "source": "kind source: *v1.Deployment"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "insights", "source": "kind source: *v1.Service"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.ServiceAccount"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Role"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.RoleBinding"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Ingress"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Route"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Issuer"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat", "source": "kind source: *v1.Certificate"}
2023-11-07T18:55:26Z INFO Starting Controller {"controller": "cryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "Cryostat"}
2023-11-07T18:55:26Z INFO Starting Controller {"controller": "insights"}
2023-11-07T18:55:26Z DEBUG events cryostat-operator-controller-manager-868b7d79cd-ssbvj_cbe9ff59-923c-4435-aa31-ec9a1a1c6bda became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"cryostat-operator-system","name":"d696d7ab.redhat.com","uid":"6504839e-ffee-46c7-b2fb-32deedd27806","apiVersion":"coordination.k8s.io/v1","resourceVersion":"70133"}, "reason": "LeaderElection"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1beta1.ClusterCryostat"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1.Deployment"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1.Service"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1.Secret"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1.PersistentVolumeClaim"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1.ServiceAccount"}
2023-11-07T18:55:26Z INFO Starting EventSource {"controller": "clustercryostat", "controllerGroup": "operator.cryostat.io", "controllerKind": "ClusterCryostat", "source": "kind source: *v1.Role"}

When I do make create_cryostat_cr the Operator controller doesn't seem to respond to it. I don't see any new log output indicating that it has even seen the Cryostat CR appear, and there are no Deployment/Service/etc. corresponding to the CR.

[ebaron]

The "object has been modified" errors are indeed harmless. This can happen from things like Kubernetes modifying a field in the deployment in between us doing a get and update. The changes are retried on the next iteration of the reconcile loop.

For the second problem, I may have broken the controller watches. I'll take a look.

[ebaron]

For the second problem, I may have broken the controller watches. I'll take a look.

Fixed now. The controller watches weren't handling the empty WATCH_NAMESPACE properly for AllNamespaces install mode.