Add prometheus metrics for antrea controller #325

weiqiangt · 2020-01-17T06:27:30Z

This CL brings metrics of antrea controller events processing, includes: syncing time, processed number, and queue length.

Signed-off-by: Weiqiang TANG [email protected]

antrea-bot · 2020-01-17T06:27:45Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e, conformance and network policy tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests.
/skip-e2e: to skip e2e tests.
/test-conformance: to trigger conformance tests.
/skip-conformance: to skip conformance tests.
/test-networkpolicy: to trigger networkpolicy tests.
/skip-networkpolicy: to skip networkpolicy tests.
/test-all: to trigger all tests.
/skip-all: to skip all tests.

These commands can only be run by members of the vmware-tanzu organization.

weiqiangt · 2020-01-17T06:28:05Z

/test-all

antoninbas · 2020-01-17T21:46:24Z

@weiqiangt what's the relationship to Kobi's PR: #321?

jianjuns · 2020-01-18T00:58:18Z

@salv-orlando @ksamoray @tnqn : do you think we should use prometheus packages and format in Controller/Agent code or we should define stats with our own or some generic format?

weiqiangt · 2020-01-19T05:18:15Z

@weiqiangt what's the relationship to Kobi's PR: #321?

I think this CL is independent with #321, because it only provides some metrics about the controller. BTW, these metrics were used in the scale test.

pkg/controller/metrics/metrics.go

ksamoray · 2020-01-19T12:56:33Z

@weiqiangt what's the relationship to Kobi's PR: #321?

I think this CL is independent with #321, because it only provides some metrics about the controller. BTW, these metrics were used in the scale test.

Were you able to retrieve the metrics without initializing a listener? How do you retrieve these?

ksamoray · 2020-01-19T12:59:34Z

@salv-orlando @ksamoray @tnqn : do you think we should use prometheus packages and format in Controller/Agent code or we should define stats with our own or some generic format?

Prometheus will provide some metrics for process, Prometheus listener, and go related metrics.
Other than this I think that we should provide our own metrics as we do with the monitor CRD.

Unless I'm answering the wrong question :)

jianjuns · 2020-01-20T04:12:04Z

Prometheus will provide some metrics for process, Prometheus listener, and go related metrics.
Other than this I think that we should provide our own metrics as we do with the monitor CRD.

@ksamoray I am asking Prometheus packages, structs, and funcs, like promauto.NewCounter, Summary, etc.
Should we use them or define our own.

ksamoray · 2020-01-20T08:54:49Z

@ksamoray I am asking Prometheus packages, structs, and funcs, like promauto.NewCounter, Summary, etc.
Should we use them or define our own.
@jianjuns
Yeah we should use these unless they do not supply the required functionality for a specific statistic.

jianjuns · 2020-01-21T21:06:16Z

@ksamoray I am asking Prometheus packages, structs, and funcs, like promauto.NewCounter, Summary, etc.
Should we use them or define our own.
@jianjuns
Yeah we should use these unless they do not supply the required functionality for a specific statistic.
So, besides prometheus I believe Antrea will have other consumers (through API and CLI) of the metrics and stats. Do you think the prometheus definition is generic enough for other consumers? @ksamoray

ksamoray · 2020-01-23T14:23:14Z

So, besides prometheus I believe Antrea will have other consumers (through API and CLI) of the metrics and stats. Do you think the prometheus definition is generic enough for other consumers? @ksamoray
Does "generic enough" implies "has support to enough data types"?
Or "has enough means for data retrieval"?
As Prometheus' listener is an API endpoint , we could either leverage it - or do we need another mean maybe?
Maybe it's worthwhile to discuss this during the Antrea weekly. Or I could update that doc again with additional info which I gathered, and use it as a base for discussion.
I'm more concerned with scale issues and performance impact than with genericity - after all stats are about exposing floats, counters in various ways (tagged lists etc) which is what Prometheus does.
@jianjuns

ksamoray · 2020-01-23T14:25:50Z

I still have to gather which metrics should be exposed by the agent - in the meantime I exposed the values which are exposed by the custom crd (# of pods, OVS table stats).
@weiqiangt collected quite a few of these from the controller already.

jianjuns · 2020-01-23T18:56:44Z

@ksamoray: I just wonder if Prometheus structs and funcs are generic enough and widely used for metrics that can be consumed by other consumers through other interfaces, besides Prometheus?
If we want to expose stats/metrics through CLI and API, we need to define Go structs for them (I guess Prometheus definition can not be used directly).

As Prometheus' listener is an API endpoint , we could either leverage it - or do we need another mean maybe?

Sorry, I did not know much about Prometheus. But the listener is in where? Controller/Agent, another process, or Promeheus?

Maybe it's worthwhile to discuss this during the Antrea weekly. Or I could update that doc again with additional info which I gathered, and use it as a base for discussion.

It will be great if you can update the doc with more info.

pkg/controller/networkpolicy/networkpolicy_controller.go

This CL brings metrics of antrea controller events processing, includes: syncing time, processed number, and queue length. Signed-off-by: Weiqiang TANG <[email protected]>

weiqiangt · 2020-04-29T03:29:47Z

/test-all

weiqiangt · 2020-04-29T03:43:53Z

/test-all

tnqn

LGTM, just want to confirm the program won't panic if metrics is disabled

tnqn · 2020-04-29T05:01:04Z

pkg/controller/networkpolicy/networkpolicy_controller.go

@@ -838,6 +839,7 @@ func (n *NetworkPolicyController) deleteNamespace(old interface{}) {
 func (n *NetworkPolicyController) enqueueAppliedToGroup(key string) {
 	klog.V(4).Infof("Adding new key %s to AppliedToGroup queue", key)
 	n.appliedToGroupQueue.Add(key)
+	metrics.LengthAppliedToGroupQueue.Set(float64(n.appliedToGroupQueue.Len()))


Could it panic if metrics is disabled?

No, this PR can pass kind e2e test.

weiqiangt · 2020-04-29T05:10:46Z

/test-e2e

tnqn

LGTM

weiqiangt · 2020-04-29T08:17:29Z

/test-all

Add metrics for antrea controller events processing, includes syncing time, processed number, and queue length. Signed-off-by: Weiqiang TANG <[email protected]>

weiqiangt requested review from salv-orlando, abhiraut, antoninbas, tnqn and ksamoray January 17, 2020 06:27

vmwclabot added the cla-not-required label Jan 17, 2020

ksamoray reviewed Jan 19, 2020

View reviewed changes

pkg/controller/metrics/metrics.go Outdated Show resolved Hide resolved

weiqiangt force-pushed the controller-prometheus branch from a2ca44a to 3a28c00 Compare January 20, 2020 01:42

ksamoray mentioned this pull request Jan 30, 2020

Exporting prometheus metrics for OVS performance and routes #236

Closed

tnqn reviewed Feb 6, 2020

View reviewed changes

pkg/controller/networkpolicy/networkpolicy_controller.go Show resolved Hide resolved

ksamoray mentioned this pull request Apr 1, 2020

Antrea Prometheus integration #321

Merged

weiqiangt force-pushed the controller-prometheus branch from 3a28c00 to 657e394 Compare April 29, 2020 03:19

Add prometheus metrics for antrea controller

4867626

This CL brings metrics of antrea controller events processing, includes: syncing time, processed number, and queue length. Signed-off-by: Weiqiang TANG <[email protected]>

weiqiangt force-pushed the controller-prometheus branch from 657e394 to 4867626 Compare April 29, 2020 03:27

tnqn reviewed Apr 29, 2020

View reviewed changes

tnqn approved these changes Apr 29, 2020

View reviewed changes

weiqiangt merged commit e0d0543 into antrea-io:master Apr 29, 2020

weiqiangt deleted the controller-prometheus branch April 29, 2020 08:56

McCodeman pushed a commit that referenced this pull request Jun 2, 2020

Add prometheus metrics for antrea controller (#325)

119f49d

Add metrics for antrea controller events processing, includes syncing time, processed number, and queue length. Signed-off-by: Weiqiang TANG <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add prometheus metrics for antrea controller #325

Add prometheus metrics for antrea controller #325

weiqiangt commented Jan 17, 2020

antrea-bot commented Jan 17, 2020

weiqiangt commented Jan 17, 2020

antoninbas commented Jan 17, 2020

jianjuns commented Jan 18, 2020

weiqiangt commented Jan 19, 2020

ksamoray commented Jan 19, 2020

ksamoray commented Jan 19, 2020

jianjuns commented Jan 20, 2020

ksamoray commented Jan 20, 2020 •

edited

Loading

jianjuns commented Jan 21, 2020

ksamoray commented Jan 23, 2020

ksamoray commented Jan 23, 2020

jianjuns commented Jan 23, 2020 •

edited

Loading

weiqiangt commented Apr 29, 2020

weiqiangt commented Apr 29, 2020

tnqn left a comment

tnqn Apr 29, 2020

weiqiangt Apr 29, 2020

weiqiangt commented Apr 29, 2020

tnqn left a comment

weiqiangt commented Apr 29, 2020

Add prometheus metrics for antrea controller #325

Add prometheus metrics for antrea controller #325

Conversation

weiqiangt commented Jan 17, 2020

antrea-bot commented Jan 17, 2020

weiqiangt commented Jan 17, 2020

antoninbas commented Jan 17, 2020

jianjuns commented Jan 18, 2020

weiqiangt commented Jan 19, 2020

ksamoray commented Jan 19, 2020

ksamoray commented Jan 19, 2020

jianjuns commented Jan 20, 2020

ksamoray commented Jan 20, 2020 • edited Loading

jianjuns commented Jan 21, 2020

ksamoray commented Jan 23, 2020

ksamoray commented Jan 23, 2020

jianjuns commented Jan 23, 2020 • edited Loading

weiqiangt commented Apr 29, 2020

weiqiangt commented Apr 29, 2020

tnqn left a comment

Choose a reason for hiding this comment

tnqn Apr 29, 2020

Choose a reason for hiding this comment

weiqiangt Apr 29, 2020

Choose a reason for hiding this comment

weiqiangt commented Apr 29, 2020

tnqn left a comment

Choose a reason for hiding this comment

weiqiangt commented Apr 29, 2020

ksamoray commented Jan 20, 2020 •

edited

Loading

jianjuns commented Jan 23, 2020 •

edited

Loading