New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add events metricset for kubernetes metricbeat module #4315

Merged

exekias merged 3 commits into elastic:master from vjsamuel:kubernetes_events

May 18, 2017

Contributor

vjsamuel commented May 15, 2017 •

edited

Loading

This PR adds a kubernetes events metricset to the kubernetes metricbeat module. Events are polled from the kubernetes API server and are written into the outputs interface.

vjsamuel changed the title ~~Adding events metricset for kubernetes metricbeat module~~ WIP: Adding events metricset for kubernetes metricbeat module

Contributor Author

vjsamuel commented May 15, 2017

@ruflin, vendoring needs to be decided as im getting these info messages:

2017/05/15 01:06:58 proto: duplicate proto type registered: github.com/ericchiang.k8s.apis.extensions.v1beta1.DaemonSetList
2017/05/15 01:06:58 proto: duplicate proto type registered: github.com/ericchiang.k8s.apis.extensions.v1beta1.DaemonSetSpec
2017/05/15 01:06:58 proto: duplicate proto type registered: github.com/ericchiang.k8s.apis.extensions.v1beta1.DaemonSetStatus
2017/05/15 01:06:58 proto: duplicate proto type registered: github.com/ericchiang.k8s.apis.extensions.v1beta1.DaemonSetUpdateStrategy
2017/05/15 01:06:58 proto: duplicate proto type registered: github.com/ericchiang.k8s.apis.extensions.v1beta1.Deployment

Collaborator

elasticmachine commented May 15, 2017

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

1 similar comment

Collaborator

elasticmachine commented May 15, 2017

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

monicasarbu added in progress Metricbeat labels

andrewkroh reviewed

View reviewed changes

Member

andrewkroh left a comment

LGTM, did a quick scan of the metricset implementation only.

metricbeat/module/kubernetes/events/events.go Outdated

    
              // Part of new is also setting up the configuration by processing additional

              // configuration entries if needed.

              func New(base mb.BaseMetricSet) (mb.MetricSet, error) {

              	logp.Warn("EXPERIMENTAL: The kubernetes events metricset is experimental")

Member

andrewkroh May 15, 2017

Use logp.Experimental.

Contributor Author

vjsamuel May 15, 2017

Done. We would need to update the metricset generator as this was generated code.

metricbeat/module/kubernetes/events/events.go Outdated

    
              		return nil, fmt.Errorf("fail to unpack the kubernetes events configuration: %s", err)

              	}

              	err = validate(config)

Member

andrewkroh May 15, 2017

go-ucfg allows a config type to implement a Validate() error function that is invoked automatically on when unpacking.

Contributor Author

vjsamuel May 15, 2017

Done.

exekias reviewed

View reviewed changes

metricbeat/module/kubernetes/events/events.go Outdated

    
              			m.watcher.Stop()

              			return

              		case msg := <-m.watcher.eventQueue:

              			// Ignore events that are deleted

Contributor

exekias May 15, 2017

Why are you ignoring deletion events? I guess someone could be interested on them, perhaps we should offer some fine grained filter for desired events?

Contributor Author

vjsamuel May 15, 2017

delete, in case of events is to just delete the API Event object. it carries no significance from monitoring perspective.

vjsamuel changed the title ~~WIP: Adding events metricset for kubernetes metricbeat module~~ Add events metricset for kubernetes metricbeat module

vjsamuel force-pushed the kubernetes_events branch 2 times, most recently from 2e53faf to c87167f Compare

May 15, 2017 19:36

ruflin reviewed

View reviewed changes

Member

ruflin left a comment

Thanks for the PR. I left a few comments mainly related to naming conventions.

One thing that seems to be missing is adding it to the config file.

From which of the two services which we connect to in the kubernetes module at the moment will this info be fetched? What is the installation pattern?

metricbeat/docs/fields.asciidoc Outdated

    
            @@ -6040,6 +6040,194 @@ Used inodes
          
              [float]

              == events Fields

Member

ruflin May 16, 2017

The metricset should probably be called event and not events to be in line with our naming convention.

metricbeat/docs/fields.asciidoc Outdated

    
              [float]

              === kubernetes.events.firstOccurrenceTimestamp

Member

ruflin May 16, 2017

I would suggest to change this to *.event.timestamp.first_occurenceand below to *.event.timestamp.last_occurence.

metricbeat/docs/fields.asciidoc Outdated

    
              [float]

              === kubernetes.events.metadata.creationTimestamp

Member

ruflin May 16, 2017

timestamp.created, timestamp.deleted

metricbeat/docs/fields.asciidoc Outdated

    
              [float]

              === kubernetes.events.metadata.resourceVersion

Member

ruflin May 16, 2017

resource_version or version.resource

metricbeat/docs/fields.asciidoc Outdated

    
              [float]

              === kubernetes.events.metadata.selfLink

Member

ruflin May 16, 2017

self_link

metricbeat/docs/fields.asciidoc Outdated

    
              [float]

              === kubernetes.events.tags.host

Member

ruflin May 16, 2017

Why is this under tags

metricbeat/docs/fields.asciidoc Outdated

    
              [float]

              === kubernetes.events.tags.pod

Member

ruflin May 16, 2017

What is this under tags. Don't we have a common namespace for these?

Contributor Author

vjsamuel May 16, 2017

these are dimensions that can be used to better query information. base implementation off of which I went with is here:
https://github.com/kubernetes/heapster/blob/master/events/sinks/elasticsearch/driver.go#L76

metricbeat/module/kubernetes/events/eventwatcher.go Outdated

    
              )

              // EventWatcher is a controller that synchronizes Pods.

              type EventWatcher struct {

Member

ruflin May 16, 2017

You could call this just Watcher as it is already in the event package.

metricbeat/module/kubernetes/events/eventwatcher.go Outdated

    
              		if err != nil {

              			//if listing fails try again after sometime

              			logp.Err("kubernetes: List API error %v", err)

              			time.Sleep(time.Second)

Member

ruflin May 16, 2017

Based on what did you select this timeout? Perhaps a comment could be helpful here.

metricbeat/module/kubernetes/events/eventwatcher.go Outdated

    
              			//watch events failures should be logged and gracefully failed over as metadata retrieval

              			//should never stop.

              			logp.Err("kubernetes: Watching API eror %v", err)

              			time.Sleep(time.Second)

Member

ruflin May 16, 2017

see above

vjsamuel force-pushed the kubernetes_events branch from c87167f to 463299c Compare

May 16, 2017 17:52

ruflin reviewed

View reviewed changes

metricbeat/metricbeat.full.yml Outdated

    
              - module: kubernetes

                metricsets:

                  - event

                kube_config: ${HOME}/.kube/config

Member

ruflin May 17, 2017

Is ${HOME} always set in these environments?
You left out the hosts: [...] part. Is that on purpose?

Contributor Author

vjsamuel May 17, 2017

as per @exekias we can default in_cluster to true. this metricset doesnt rely on hosts by default. we can add documentation on various ways to configure it similar to the kubernetes processor.

metricbeat/module/kubernetes/event/_meta/data.json Outdated

    
                          },

                          "message": "Created pod: prometheus-2552087900-9fxh6",

                          "metadata": {

                              "annotations": null,

Member

ruflin May 17, 2017

Would be nice to not send fields if they are null

Contributor Author

vjsamuel May 17, 2017

done

metricbeat/module/kubernetes/_meta/config.yml

    
              # Kubernetes events

              - module: kubernetes

                metricsets:

                  - event

Member

ruflin May 17, 2017

I remember @exekias added somewhere that all state_* prefixed metricset must be installed once in contrast to the other ones. If this is installed global (my assumption) should this also have a state_ prefix?

NOTE: In general I think we need to find a better solution for this in the long run but in the short term we should try to be consistent.

Contributor

exekias May 17, 2017

I think event (or events) is ok in this case, this is something unrelated to kube-state-metrics, state_ metricsets get their name from it

exekias reviewed

View reviewed changes

metricbeat/module/kubernetes/_meta/config.yml Outdated

    
                metricsets:

                  - event

                kube_config: ${HOME}/.kube/config

                in_cluster: false

Contributor

exekias May 17, 2017

perhaps we should leave in_cluster: true as default? I guess it will be the most common case

Contributor Author

vjsamuel May 17, 2017

will default to true

metricbeat/module/kubernetes/_meta/config.yml

    
              # Kubernetes events

              - module: kubernetes

                metricsets:

                  - event

Contributor

exekias May 17, 2017

I think event (or events) is ok in this case, this is something unrelated to kube-state-metrics, state_ metricsets get their name from it

metricbeat/module/kubernetes/event/event.go Outdated

    
              				"deleted": eve.Metadata.DeletionTimestamp,

              			},

              			"name":             eve.Metadata.Name,

              			"namespace":        eve.Metadata.Namespace,

Contributor

exekias May 17, 2017

this one can ve moved to kubernetes.namespace, using mb.ModuleKey. WDYT?

Contributor Author

vjsamuel May 17, 2017

we would need to do this across all metricsets to establish consistency. lets do it in a follow-up PR all in one go.

vjsamuel force-pushed the kubernetes_events branch from 463299c to a5e3f48 Compare

May 17, 2017 16:50

vjsamuel added 3 commits

May 17, 2017 23:23


          Adding events metricset for kubernetes metricbeat module

2e6453b


          Moving k8s client to top level vendor

59227c8


          Incorporating review comments

vjsamuel force-pushed the kubernetes_events branch from a5e3f48 to 3267969 Compare

May 18, 2017 06:23

Member

ruflin commented May 18, 2017

jenkins, test it

elasticmachine reviewed

View reviewed changes

metricbeat/module/kubernetes/event/config.go

+              	SyncPeriod time.Duration `config:"sync_period"`
+              }
+              type Enabled struct {

Collaborator

elasticmachine May 18, 2017

[golint] _{reported by reviewdog 🐶}
exported type Enabled should have comment or be unexported

metricbeat/module/kubernetes/event/config.go

+              	Enabled bool `config:"enabled"`
+              }
+              type PluginConfig []map[string]common.Config

Collaborator

elasticmachine May 18, 2017

[golint] _{reported by reviewdog 🐶}
exported type PluginConfig should have comment or be unexported

metricbeat/module/kubernetes/event/types.go


		import "time"

		type ObjectMeta struct {

Collaborator

elasticmachine May 18, 2017

[golint] _{reported by reviewdog 🐶}
exported type ObjectMeta should have comment or be unexported

metricbeat/module/kubernetes/event/types.go

+              	UID             string `json:"uid"`
+              }
+              type Event struct {

Collaborator

elasticmachine May 18, 2017

[golint] _{reported by reviewdog 🐶}
exported type Event should have comment or be unexported

metricbeat/module/kubernetes/event/watcher.go


		}

		func (w *Watcher) Run() {

Collaborator

elasticmachine May 18, 2017

[golint] _{reported by reviewdog 🐶}
exported method Watcher.Run should have comment or be unexported

metricbeat/module/kubernetes/event/watcher.go


		}

		func (w *Watcher) Stop() {

Collaborator

elasticmachine May 18, 2017

[golint] _{reported by reviewdog 🐶}
exported method Watcher.Stop should have comment or be unexported

exekias merged commit cc0f165 into elastic:master

exekias mentioned this pull request

Follow up k8s events metricset #4347

Closed

3 tasks

monicasarbu added review and removed in progress labels

vjsamuel deleted the kubernetes_events branch

March 5, 2018 07:25

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Metricbeat review