Unable to add webhooks

[aantn]

Hi, I'm using the configure-alertmanager-operator with OpenShift 4.14 and I'm unable to add a new webhook by editing the alertmanager-main secret as described here.

Whenever I make changes to the secret they are reverted immediately.

Here are logs from the configure-alertmanager-operator pod:

1.7266830314908295e+09	INFO	secret_controller	Reconciling Object	{"Request.Name": "alertmanager-main"}
1.7266830314908586e+09	INFO	secret_controller	DEBUG: Started reconcile loop	{"Request.Name": "alertmanager-main"}
1.726683031490865e+09	INFO	readiness	DEBUG: Using cached positive cluster readiness.
1.7266830314916618e+09	INFO	secret_controller	DEBUG: Secret named 'goalert-secret' not found	{"Request.Name": "alertmanager-main"}
1.726683031492164e+09	INFO	secret_controller	DEBUG: Secret named 'pd-secret' found	{"Request.Name": "alertmanager-main"}
1.7266830314922059e+09	INFO	secret_controller	DEBUG: Secret named 'dms-secret' found	{"Request.Name": "alertmanager-main"}
1.726683031492231e+09	INFO	secret_controller	INFO: Pager Duty secret exists	{"Request.Name": "alertmanager-main"}
1.726683031492254e+09	INFO	secret_controller	INFO: Cluster is ready; configuring Pager Duty	{"Request.Name": "alertmanager-main"}
1.7266830314922936e+09	INFO	secret_controller	INFO: Dead Man's Snitch secret exists	{"Request.Name": "alertmanager-main"}
1.7266830314923384e+09	INFO	secret_controller	INFO: Goalert secret does not exist	{"Request.Name": "alertmanager-main"}
1.726683031492397e+09	INFO	secret_controller	DEBUG: ConfigMap named 'managed-namespaces' found	{"Request.Name": "alertmanager-main"}
1.726683031492749e+09	INFO	secret_controller	DEBUG: ConfigMap named 'ocp-namespaces' found	{"Request.Name": "alertmanager-main"}
1.7266830314931371e+09	INFO	secret_controller	DEBUG: Adding PagerDuty routes for the following namespaces	{"Request.Name": "alertmanager-main", "Namespaces": ["^dedicated-admin$", "^openshift-addon-operator$", "^openshift-aqua$", "^openshift-aws-vpce-operator$", "^openshift-backplane$", "^openshift-backplane-cee$", "^openshift-backplane-csa$", "^openshift-backplane-cse$", "^openshift-backplane-csm$", "^openshift-backplane-managed-scripts$", "^openshift-backplane-mobb$", "^openshift-backplane-srep$", "^openshift-backplane-tam$", "^openshift-cloud-ingress-operator$", "^openshift-codeready-workspaces$", "^openshift-compliance$", "^openshift-compliance-monkey$", "^openshift-container-security$", "^openshift-custom-domains-operator$", "^openshift-customer-monitoring$", "^openshift-deployment-validation-operator$", "^openshift-managed-node-metadata-operator$", "^openshift-file-integrity$", "^openshift-logging$", "^openshift-managed-upgrade-operator$", "^openshift-must-gather-operator$", "^openshift-observability-operator$", "^openshift-ocm-agent-operator$", "^openshift-operators-redhat$", "^openshift-osd-metrics$", "^openshift-rbac-permissions$", "^openshift-route-monitor-operator$", "^openshift-scanning$", "^openshift-security$", "^openshift-splunk-forwarder-operator$", "^openshift-sre-pruning$", "^openshift-suricata$", "^openshift-validation-webhook$", "^openshift-velero$", "^openshift-monitoring$", "^openshift$", "^openshift-cluster-version$", "^goalert$", "^keycloak$", "^configure-goalert-operator$", "^kube-system$", "^openshift-apiserver$", "^openshift-apiserver-operator$", "^openshift-authentication$", "^openshift-authentication-operator$", "^openshift-cloud-controller-manager$", "^openshift-cloud-controller-manager-operator$", "^openshift-cloud-credential-operator$", "^openshift-cloud-network-config-controller$", "^openshift-cluster-api$", "^openshift-cluster-csi-drivers$", "^openshift-cluster-machine-approver$", "^openshift-cluster-node-tuning-operator$", "^openshift-cluster-samples-operator$", "^openshift-cluster-storage-operator$", "^openshift-config$", "^openshift-config-managed$", "^openshift-config-operator$", "^openshift-console$", "^openshift-console-operator$", "^openshift-console-user-settings$", "^openshift-controller-manager$", "^openshift-controller-manager-operator$", "^openshift-dns$", "^openshift-dns-operator$", "^openshift-etcd$", "^openshift-etcd-operator$", "^openshift-host-network$", "^openshift-image-registry$", "^openshift-ingress$", "^openshift-ingress-canary$", "^openshift-ingress-operator$", "^openshift-insights$", "^openshift-kni-infra$", "^openshift-kube-apiserver$", "^openshift-kube-apiserver-operator$", "^openshift-kube-controller-manager$", "^openshift-kube-controller-manager-operator$", "^openshift-kube-scheduler$", "^openshift-kube-scheduler-operator$", "^openshift-kube-storage-version-migrator$", "^openshift-kube-storage-version-migrator-operator$", "^openshift-machine-api$", "^openshift-machine-config-operator$", "^openshift-marketplace$", "^openshift-monitoring$", "^openshift-multus$", "^openshift-network-diagnostics$", "^openshift-network-operator$", "^openshift-nutanix-infra$", "^openshift-oauth-apiserver$", "^openshift-openstack-infra$", "^openshift-operator-lifecycle-manager$", "^openshift-operators$", "^openshift-ovirt-infra$", "^openshift-sdn$", "^openshift-ovn-kubernetes$", "^openshift-platform-operators$", "^openshift-route-controller-manager$", "^openshift-service-ca$", "^openshift-service-ca-operator$", "^openshift-user-workload-monitoring$", "^openshift-vsphere-infra$"]}
1.7266830314932032e+09	INFO	secret_controller	DEBUG: ConfigMap named 'ocm-agent' found	{"Request.Name": "alertmanager-main"}
1.7266830314932537e+09	INFO	secret_controller	INFO: Configuring a watchdog route and receiver	{"Request.Name": "alertmanager-main"}
1.7266830314932847e+09	INFO	secret_controller	INFO: Configuring a PagerDuty route and receiver	{"Request.Name": "alertmanager-main"}
1.726683031493589e+09	INFO	secret_controller	INFO: Not configuring GoAlert receivers	{"Request.Name": "alertmanager-main"}
1.72668303149364e+09	INFO	secret_controller	INFO: Not configuring GoAlert Heartbeat receivers	{"Request.Name": "alertmanager-main"}
1.726683031509792e+09	INFO	secret_controller	INFO: Secret alertmanager-main successfully updated	{"Request.Name": "alertmanager-main"}
1.7266830315098178e+09	INFO	secret_controller	Finished reconcile for secret.	{"Request.Name": "alertmanager-main"}
1.726683031509864e+09	INFO	secret_controller	Reconciling Object	{"Request.Name": "alertmanager-main-generated"}
1.7266830315098739e+09	INFO	secret_controller	Skip reconcile: No changes detected to alertmanager secrets.	{"Request.Name": "alertmanager-main-generated"}
1.7266830315105565e+09	INFO	secret_controller	Reconciling Object	{"Request.Name": "alertmanager-main"}
1.7266830315106125e+09	INFO	secret_controller	DEBUG: Started reconcile loop	{"Request.Name": "alertmanager-main"}
1.726683031510639e+09	INFO	readiness	DEBUG: Using cached positive cluster readiness.
1.7266830315114343e+09	INFO	secret_controller	DEBUG: Secret named 'goalert-secret' not found	{"Request.Name": "alertmanager-main"}
1.726683031511492e+09	INFO	secret_controller	DEBUG: Secret named 'pd-secret' found	{"Request.Name": "alertmanager-main"}
1.7266830315115237e+09	INFO	secret_controller	DEBUG: Secret named 'dms-secret' found	{"Request.Name": "alertmanager-main"}
1.7266830315115461e+09	INFO	secret_controller	INFO: Pager Duty secret exists	{"Request.Name": "alertmanager-main"}
1.7266830315115674e+09	INFO	secret_controller	INFO: Cluster is ready; configuring Pager Duty	{"Request.Name": "alertmanager-main"}
1.7266830315115988e+09	INFO	secret_controller	INFO: Dead Man's Snitch secret exists	{"Request.Name": "alertmanager-main"}
1.7266830315116103e+09	INFO	secret_controller	INFO: Goalert secret does not exist	{"Request.Name": "alertmanager-main"}
1.7266830315116155e+09	INFO	secret_controller	DEBUG: ConfigMap named 'managed-namespaces' found	{"Request.Name": "alertmanager-main"}
1.726683031511903e+09	INFO	secret_controller	DEBUG: ConfigMap named 'ocp-namespaces' found	{"Request.Name": "alertmanager-main"}
1.7266830315123167e+09	INFO	secret_controller	DEBUG: Adding PagerDuty routes for the following namespaces	{"Request.Name": "alertmanager-main", "Namespaces": ["^dedicated-admin$", "^openshift-addon-operator$", "^openshift-aqua$", "^openshift-aws-vpce-operator$", "^openshift-backplane$", "^openshift-backplane-cee$", "^openshift-backplane-csa$", "^openshift-backplane-cse$", "^openshift-backplane-csm$", "^openshift-backplane-managed-scripts$", "^openshift-backplane-mobb$", "^openshift-backplane-srep$", "^openshift-backplane-tam$", "^openshift-cloud-ingress-operator$", "^openshift-codeready-workspaces$", "^openshift-compliance$", "^openshift-compliance-monkey$", "^openshift-container-security$", "^openshift-custom-domains-operator$", "^openshift-customer-monitoring$", "^openshift-deployment-validation-operator$", "^openshift-managed-node-metadata-operator$", "^openshift-file-integrity$", "^openshift-logging$", "^openshift-managed-upgrade-operator$", "^openshift-must-gather-operator$", "^openshift-observability-operator$", "^openshift-ocm-agent-operator$", "^openshift-operators-redhat$", "^openshift-osd-metrics$", "^openshift-rbac-permissions$", "^openshift-route-monitor-operator$", "^openshift-scanning$", "^openshift-security$", "^openshift-splunk-forwarder-operator$", "^openshift-sre-pruning$", "^openshift-suricata$", "^openshift-validation-webhook$", "^openshift-velero$", "^openshift-monitoring$", "^openshift$", "^openshift-cluster-version$", "^goalert$", "^keycloak$", "^configure-goalert-operator$", "^kube-system$", "^openshift-apiserver$", "^openshift-apiserver-operator$", "^openshift-authentication$", "^openshift-authentication-operator$", "^openshift-cloud-controller-manager$", "^openshift-cloud-controller-manager-operator$", "^openshift-cloud-credential-operator$", "^openshift-cloud-network-config-controller$", "^openshift-cluster-api$", "^openshift-cluster-csi-drivers$", "^openshift-cluster-machine-approver$", "^openshift-cluster-node-tuning-operator$", "^openshift-cluster-samples-operator$", "^openshift-cluster-storage-operator$", "^openshift-config$", "^openshift-config-managed$", "^openshift-config-operator$", "^openshift-console$", "^openshift-console-operator$", "^openshift-console-user-settings$", "^openshift-controller-manager$", "^openshift-controller-manager-operator$", "^openshift-dns$", "^openshift-dns-operator$", "^openshift-etcd$", "^openshift-etcd-operator$", "^openshift-host-network$", "^openshift-image-registry$", "^openshift-ingress$", "^openshift-ingress-canary$", "^openshift-ingress-operator$", "^openshift-insights$", "^openshift-kni-infra$", "^openshift-kube-apiserver$", "^openshift-kube-apiserver-operator$", "^openshift-kube-controller-manager$", "^openshift-kube-controller-manager-operator$", "^openshift-kube-scheduler$", "^openshift-kube-scheduler-operator$", "^openshift-kube-storage-version-migrator$", "^openshift-kube-storage-version-migrator-operator$", "^openshift-machine-api$", "^openshift-machine-config-operator$", "^openshift-marketplace$", "^openshift-monitoring$", "^openshift-multus$", "^openshift-network-diagnostics$", "^openshift-network-operator$", "^openshift-nutanix-infra$", "^openshift-oauth-apiserver$", "^openshift-openstack-infra$", "^openshift-operator-lifecycle-manager$", "^openshift-operators$", "^openshift-ovirt-infra$", "^openshift-sdn$", "^openshift-ovn-kubernetes$", "^openshift-platform-operators$", "^openshift-route-controller-manager$", "^openshift-service-ca$", "^openshift-service-ca-operator$", "^openshift-user-workload-monitoring$", "^openshift-vsphere-infra$"]}
1.7266830315124087e+09	INFO	secret_controller	DEBUG: ConfigMap named 'ocm-agent' found	{"Request.Name": "alertmanager-main"}
1.726683031512461e+09	INFO	secret_controller	INFO: Configuring a watchdog route and receiver	{"Request.Name": "alertmanager-main"}
1.7266830315124884e+09	INFO	secret_controller	INFO: Configuring a PagerDuty route and receiver	{"Request.Name": "alertmanager-main"}
1.7266830315127113e+09	INFO	secret_controller	INFO: Not configuring GoAlert receivers	{"Request.Name": "alertmanager-main"}
1.7266830315127356e+09	INFO	secret_controller	INFO: Not configuring GoAlert Heartbeat receivers	{"Request.Name": "alertmanager-main"}
1.726683031522142e+09	INFO	secret_controller	INFO: Secret alertmanager-main successfully updated	{"Request.Name": "alertmanager-main"}
1.7266830315221744e+09	INFO	secret_controller	Finished reconcile for secret.	{"Request.Name": "alertmanager-main"}
1.7266830315374963e+09	INFO	secret_controller	Reconciling Object	{"Request.Name": "alertmanager-main-generated"}
1.7266830315376081e+09	INFO	secret_controller	Skip reconcile: No changes detected to alertmanager secrets.	{"Request.Name": "alertmanager-main-generated"}

Am I making some obvious mistake? I've also tried adding a webhook receiver via the OpenShift console, but those changes are reverted immediately too.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale