[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633

Lucasgranet · 2024-08-22T13:03:30Z

What this PR does / why we need it:
Adds the ability to add a Prometheus listener on the Octavia LoadBalancer in order to fetch it with any Prometheus scrapper to get metrics from the LoadBalancer.

This PR brings 4 new annotations:

loadbalancer.openstack.org/metrics-enable: "true" # Enable the listener endpoint on the Octavia LoadBalancer (default false)
loadbalancer.openstack.org/metrics-port: "9101" # Listener's port (default 9100)
loadbalancer.openstack.org/metrics-allow-cidrs: "10.0.0.0/8, fe80::/10" # Listener's allowed cidrs (default none) - see below
loadbalancer.openstack.org/load-balancer-vip-address: "10.4.2.3" #  Auto-computed field based on Octavia VIP

Why you should have to use loadbalancer.openstack.org/metrics-allow-cidrs

It's strongly recommended to apply an allowed cidrs on the listener. If a Floating IP is attached to the Octavia, the metric endpoint will be exposed publicly. Apply a restriction to the K8S's subnet is recommended.

More detail on the Prometheus listener:
https://docs.openstack.org/octavia/latest/user/guides/monitoring.html#monitoring-with-prometheus

Which issue this PR fixes(if applicable):
fixes #2465

Special notes for reviewers:
A new E2E test has been added.

Release note:

[openstack-cloud-controller-manager] Support Octavia/Amphora Prometheus endpoint creation using annotations

k8s-ci-robot · 2024-08-22T13:03:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign zetaab for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-08-22T13:03:41Z

Hi @Lucasgranet. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jichenjc · 2024-09-03T01:13:59Z

/ok-to-test

Lucasgranet · 2024-09-10T08:19:16Z

Hello,

Have you any feedback or questions about this PR?

Lucasgranet · 2024-10-15T12:58:46Z

Hello, @jichenjc @dulek,

If you have some time, would you mind reviewing this PR please?

Thank you,

kayrus

Thanks for the PR. See my comments. I think we need to rewrite the ensureOctaviaListener func to avoid excessive code duplication.

kayrus · 2024-10-15T13:53:21Z

pkg/openstack/loadbalancer.go

+	klog.V(4).Infof("getStringArrayFromServiceAnnotationSeparatedByComma(%s/%s, %v, %q)", service.Namespace, service.Name, annotationKey, defaultSetting)
+	if annotationValue, ok := service.Annotations[annotationKey]; ok {
+		returnValue := []string{}
+		splitAnnotation := strings.FieldsFunc( // avoid empty string by using this func instead of strings.split()


let's introduce a new SplitTrim func in pkg/util:

// SplitTrim splits a string of values separated by sep rune into a slice of // strings with trimmed spaces. func SplitTrim(s string, sep rune) []string { f := func(c rune) bool { return unicode.IsSpace(c) || c == sep } return strings.FieldsFunc(s, f) }

kayrus · 2024-10-15T13:55:48Z

pkg/openstack/loadbalancer.go

+			return nil, fmt.Errorf("failed to create metric listener for loadbalancer %s: %v", lbID, err)
+		}
+
+		klog.V(2).Infof("Metric listener %s created for loadbalancer %s", listener.ID, lbID)


let's use early return here and avoid else. instead just add a // update the listener commend right before listenerChanged := false

kayrus · 2024-10-15T13:56:33Z

pkg/openstack/loadbalancer.go

+		updateOpts := listeners.UpdateOpts{}
+
+		if svcConf.supportLBTags {
+			if !cpoutil.Contains(listener.Tags, svcConf.lbName) {


use go built-in https://pkg.go.dev/slices#Contains

kayrus · 2024-10-15T13:58:06Z

pkg/openstack/loadbalancer.go

+			listenerChanged = true
+		}
+
+		if listenerChanged {


Suggested change

if listenerChanged {

if !listenerChanged {

return listener

}

kayrus · 2024-10-15T14:02:20Z

pkg/openstack/loadbalancer.go

+			} else {
+				msg := "Metric Listener cannot be deployed on Service %s, only owner Service can do that"
+				lbaas.eventRecorder.Eventf(service, corev1.EventTypeWarning, eventLBMetricListenerIgnored, msg, serviceName)
+				klog.Infof(msg, serviceName)


Suggested change

klog.Infof(msg, serviceName)

klog.Warningf(msg, serviceName)

kayrus · 2024-10-15T14:15:56Z

pkg/openstack/loadbalancer.go

+		if svcConf.metricEnabled && openstackutil.IsOctaviaFeatureSupported(lbaas.lb, openstackutil.OctaviaFeaturePrometheusListener, lbaas.opts.LBProvider) {
+			// Only a LB owner can add the prometheus listener (to avoid conflict with a shared loadbalancer)
+			if isLBOwner {
+				svcConf.metricPort = getIntFromServiceAnnotation(service, ServiceAnnotationLoadBalancerMetricsPort, 9100)


can we check in advance whether the metricPort intersects with the service.Spec.Ports?

kayrus · 2024-10-15T14:18:08Z

pkg/openstack/loadbalancer.go

+			// Only a LB owner can add the prometheus listener (to avoid conflict with a shared loadbalancer)
+			if isLBOwner {
+				svcConf.metricPort = getIntFromServiceAnnotation(service, ServiceAnnotationLoadBalancerMetricsPort, 9100)
+				lbaas.updateServiceAnnotation(service, ServiceAnnotationLoadBalancerMetricsPort, strconv.Itoa(svcConf.metricPort))


this should be a read-only annotation

kayrus · 2024-10-15T14:26:33Z

pkg/openstack/loadbalancer.go

@@ -1137,6 +1171,58 @@ func (lbaas *LbaasV2) ensureOctaviaListener(lbID string, name string, curListene
 	return listener, nil
 }

+func (lbaas *LbaasV2) ensurePrometheusListener(lbID string, name string, curListenerMapping map[listenerKey]*listeners.Listener, _ corev1.ServicePort, svcConf *serviceConfig, _ *corev1.Service) (*listeners.Listener, error) {


aargh. Now I see that this is a copy paste from the ensureOctaviaListener. Maybe it makes sense to apply findings in this func to the ensureOctaviaListener. Additionally modify the ensureOctaviaListener to support prometheus listener creation. You can replace , _ *corev1.Service in the func arguments to , metrics bool, since it's not used anyway.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Aug 22, 2024

k8s-ci-robot requested review from FengyunPan2 and mdbooth August 22, 2024 13:03

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 22, 2024

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 22, 2024

Lucasgranet force-pushed the feat/occm/metric-annotations branch 6 times, most recently from ea89e68 to 1219946 Compare August 26, 2024 10:05

nicolaslacroux mentioned this pull request Aug 27, 2024

[MKS] Public Cloud Load Balancer to support Prometheus endpoint creation using annotations ovh/public-cloud-roadmap#544

Open

Lucasgranet marked this pull request as ready for review August 27, 2024 14:14

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 27, 2024

k8s-ci-robot requested review from anguslees and dulek August 27, 2024 14:14

Lucasgranet force-pushed the feat/occm/metric-annotations branch 4 times, most recently from fd375fb to 569454a Compare September 2, 2024 12:22

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 2, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 3, 2024

Lucasgranet force-pushed the feat/occm/metric-annotations branch from 569454a to feba13d Compare September 3, 2024 08:09

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 3, 2024

Lucasgranet force-pushed the feat/occm/metric-annotations branch from feba13d to 7fdbbc8 Compare September 3, 2024 10:27

Lucasgranet force-pushed the feat/occm/metric-annotations branch from 7fdbbc8 to 1261d34 Compare September 30, 2024 06:34

feat(occm/lb): octavia prometheus listener's annotation

ff6e28c

Lucasgranet force-pushed the feat/occm/metric-annotations branch from 1261d34 to ff6e28c Compare October 7, 2024 08:30

kayrus reviewed Oct 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633

[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633

Lucasgranet commented Aug 22, 2024 •

edited

Loading

k8s-ci-robot commented Aug 22, 2024

k8s-ci-robot commented Aug 22, 2024

jichenjc commented Sep 3, 2024

Lucasgranet commented Sep 10, 2024

Lucasgranet commented Oct 15, 2024

kayrus left a comment

kayrus Oct 15, 2024

kayrus Oct 15, 2024

kayrus Oct 15, 2024

kayrus Oct 15, 2024

kayrus Oct 15, 2024

kayrus Oct 15, 2024

kayrus Oct 15, 2024

kayrus Oct 15, 2024

[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633

Are you sure you want to change the base?

[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633

Conversation

Lucasgranet commented Aug 22, 2024 • edited Loading

k8s-ci-robot commented Aug 22, 2024

k8s-ci-robot commented Aug 22, 2024

jichenjc commented Sep 3, 2024

Lucasgranet commented Sep 10, 2024

Lucasgranet commented Oct 15, 2024

kayrus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lucasgranet commented Aug 22, 2024 •

edited

Loading