Problem setting active targets of kubelet, prometheus server, alertmanager, kube-controller-manager and kube-scheduler

[marcusnh]

What happened?
Installed the kube-prometheus stack on my system, but seemed to have problem accessing the different endpoints in the system. Not quite sure what the reason is. The stack was installed completely similar as the manifest files given in the repositories.

As you can see from the image of the prometheus service discovery it can see the differnt targets, but some are not active.
Below is my services

And here are my pods running in the monitoring namespace:

I seem to be able to access the services from the prometheus pods, but can't scrape metrics. For instance when trying to connect to the prometheus-operated service or the prometheus-k8s service from the promtheus-k8s-0 pod i get a timeout error:

Notice that the prometheus-k8s service is of the type loadbalancer, and i had to add a new policy to make it work:

# Allow for external access to the service
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: prometheus-allow-external
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
  ingress:
  - ports:
    - port: 9090

[marcusnh]

Found one solution using the answer mentioned here, but still do not solve problems for service-monitors not looking at the control-plane.

[marcusnh]

From the prometheus-operator pod i have an error message given by:
level=error ts=2022-10-10T11:33:24.13088649Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="status \"monitoring/k8s\" failed: failed to update status subresource: Operation cannot be fulfilled on prometheuses.monitoring.coreos.com \"k8s\": the object has been modified; please apply your changes to the latest version and try again"