-
Notifications
You must be signed in to change notification settings - Fork 606
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633
base: master
Are you sure you want to change the base?
[occm] Support Octavia/Amphora Prometheus endpoint creation using annotations #2633
Conversation
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Hi @Lucasgranet. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
ea89e68
to
1219946
Compare
fd375fb
to
569454a
Compare
/ok-to-test |
569454a
to
feba13d
Compare
feba13d
to
7fdbbc8
Compare
Hello, Have you any feedback or questions about this PR? |
7fdbbc8
to
1261d34
Compare
1261d34
to
ff6e28c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR. See my comments. I think we need to rewrite the ensureOctaviaListener
func to avoid excessive code duplication.
klog.V(4).Infof("getStringArrayFromServiceAnnotationSeparatedByComma(%s/%s, %v, %q)", service.Namespace, service.Name, annotationKey, defaultSetting) | ||
if annotationValue, ok := service.Annotations[annotationKey]; ok { | ||
returnValue := []string{} | ||
splitAnnotation := strings.FieldsFunc( // avoid empty string by using this func instead of strings.split() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's introduce a new SplitTrim
func in pkg/util
:
// SplitTrim splits a string of values separated by sep rune into a slice of
// strings with trimmed spaces.
func SplitTrim(s string, sep rune) []string {
f := func(c rune) bool {
return unicode.IsSpace(c) || c == sep
}
return strings.FieldsFunc(s, f)
}
return nil, fmt.Errorf("failed to create metric listener for loadbalancer %s: %v", lbID, err) | ||
} | ||
|
||
klog.V(2).Infof("Metric listener %s created for loadbalancer %s", listener.ID, lbID) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's use early return here and avoid else
. instead just add a // update the listener
commend right before listenerChanged := false
updateOpts := listeners.UpdateOpts{} | ||
|
||
if svcConf.supportLBTags { | ||
if !cpoutil.Contains(listener.Tags, svcConf.lbName) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use go built-in https://pkg.go.dev/slices#Contains
listenerChanged = true | ||
} | ||
|
||
if listenerChanged { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if listenerChanged { | |
if !listenerChanged { | |
return listener | |
} |
} else { | ||
msg := "Metric Listener cannot be deployed on Service %s, only owner Service can do that" | ||
lbaas.eventRecorder.Eventf(service, corev1.EventTypeWarning, eventLBMetricListenerIgnored, msg, serviceName) | ||
klog.Infof(msg, serviceName) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
klog.Infof(msg, serviceName) | |
klog.Warningf(msg, serviceName) |
if svcConf.metricEnabled && openstackutil.IsOctaviaFeatureSupported(lbaas.lb, openstackutil.OctaviaFeaturePrometheusListener, lbaas.opts.LBProvider) { | ||
// Only a LB owner can add the prometheus listener (to avoid conflict with a shared loadbalancer) | ||
if isLBOwner { | ||
svcConf.metricPort = getIntFromServiceAnnotation(service, ServiceAnnotationLoadBalancerMetricsPort, 9100) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we check in advance whether the metricPort
intersects with the service.Spec.Ports
?
// Only a LB owner can add the prometheus listener (to avoid conflict with a shared loadbalancer) | ||
if isLBOwner { | ||
svcConf.metricPort = getIntFromServiceAnnotation(service, ServiceAnnotationLoadBalancerMetricsPort, 9100) | ||
lbaas.updateServiceAnnotation(service, ServiceAnnotationLoadBalancerMetricsPort, strconv.Itoa(svcConf.metricPort)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be a read-only annotation
@@ -1137,6 +1171,58 @@ func (lbaas *LbaasV2) ensureOctaviaListener(lbID string, name string, curListene | |||
return listener, nil | |||
} | |||
|
|||
func (lbaas *LbaasV2) ensurePrometheusListener(lbID string, name string, curListenerMapping map[listenerKey]*listeners.Listener, _ corev1.ServicePort, svcConf *serviceConfig, _ *corev1.Service) (*listeners.Listener, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
aargh. Now I see that this is a copy paste from the ensureOctaviaListener
. Maybe it makes sense to apply findings in this func to the ensureOctaviaListener
. Additionally modify the ensureOctaviaListener
to support prometheus listener creation. You can replace , _ *corev1.Service
in the func arguments to , metrics bool
, since it's not used anyway.
What this PR does / why we need it:
Adds the ability to add a Prometheus listener on the Octavia LoadBalancer in order to fetch it with any Prometheus scrapper to get metrics from the LoadBalancer.
This PR brings 4 new annotations:
Why you should have to use
loadbalancer.openstack.org/metrics-allow-cidrs
It's strongly recommended to apply an allowed cidrs on the listener. If a Floating IP is attached to the Octavia, the metric endpoint will be exposed publicly. Apply a restriction to the K8S's subnet is recommended.
More detail on the Prometheus listener:
https://docs.openstack.org/octavia/latest/user/guides/monitoring.html#monitoring-with-prometheus
Which issue this PR fixes(if applicable):
fixes #2465
Special notes for reviewers:
A new E2E test has been added.
Release note: