Refactor metadata generator to support adding metadata across resources #14875

vjsamuel · 2019-12-02T08:55:14Z

This PR attempts to refactor metadata generator to support enrichment of resources metadata on top of other resources.

Ex: enriching a pod object with information from a node and namespace object.

This PR does the same only for the autodiscover provider of kubernetes. It adds a configuration add_resource_metadata to allow specifying what resources should be used for enrichment.

Sample configuration:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      kube_config: ${HOME}/.kube/config
      resource: service
      annotations.dedot: false
      add_resource_metadata:
        namespace.enabled: true
      builders:
        - type: hints

output.console.pretty: true

this would ensure that we additionally add namespace metadata along with the service metadata.

pods can be enriched with node and namespace metadata. the namespace and node parameters of add_resource_metadata can take include_*, exlude_* configs similar to the parent resource.

closes #13873

elasticmachine · 2019-12-02T08:55:16Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2019-12-02T08:55:16Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

houndci-bot · 2019-12-02T08:55:17Z

libbeat/autodiscover/providers/kubernetes/kubernetes.go

 	"github.com/elastic/beats/libbeat/logp"
 )

 func init() {
 	autodiscover.Registry.AddProvider("kubernetes", AutodiscoverBuilder)
 }

+type Eventer interface {


exported type Eventer should have comment or be unexported

houndci-bot · 2019-12-02T08:55:18Z

libbeat/autodiscover/providers/kubernetes/kubernetes.go

 	"github.com/elastic/beats/libbeat/logp"
 )

 func init() {
 	autodiscover.Registry.AddProvider("kubernetes", AutodiscoverBuilder)
 }

+type Eventer interface {


exported type Eventer should have comment or be unexported

houndci-bot · 2019-12-02T08:55:18Z

libbeat/autodiscover/providers/kubernetes/kubernetes.go

 	"github.com/elastic/beats/libbeat/logp"
 )

 func init() {
 	autodiscover.Registry.AddProvider("kubernetes", AutodiscoverBuilder)
 }

+type Eventer interface {


exported type Eventer should have comment or be unexported

houndci-bot · 2019-12-02T08:55:20Z

libbeat/common/kubernetes/metadata/resource.go

+	config *Config
+}
+
+func NewResourceMetadataGenerator(config *Config) *resource {


exported func NewResourceMetadataGenerator returns unexported type *metadata.resource, which can be annoying to use
exported function NewResourceMetadataGenerator should have comment or be unexported

houndci-bot · 2019-12-02T08:55:20Z

libbeat/common/kubernetes/metadata/pod.go

+		}
+
+		return p.Generate(po, opts...)
+	} else {


if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)

houndci-bot · 2019-12-02T08:55:20Z

libbeat/common/kubernetes/metadata/pod.go

+	resource   *resource
+}
+
+func NewPodMetadataGenerator(cfg *Config, acfg *AddResourceMetadataConfig, client k8s.Interface, options kubernetes.WatchOptions) (MetaGen, error) {


exported function NewPodMetadataGenerator should have comment or be unexported

houndci-bot · 2019-12-02T08:55:20Z

libbeat/common/kubernetes/metadata/metadata.go

+
+type FieldOptions func(common.MapStr) error
+
+func WithFields(key string, value interface{}) func(common.MapStr) {


exported function WithFields should have comment or be unexported

houndci-bot · 2019-12-02T08:55:21Z

libbeat/common/kubernetes/metadata/metadata.go

+	Stop()
+}
+
+type FieldOptions func(common.MapStr) error


exported type FieldOptions should have comment or be unexported

houndci-bot · 2019-12-02T08:55:21Z

libbeat/common/kubernetes/metadata/config.go

+	}
+}
+
+func (c *Config) ApplyConfig(cfg *common.Config) error {


exported method Config.ApplyConfig should have comment or be unexported

houndci-bot · 2019-12-02T08:55:22Z

libbeat/common/kubernetes/metadata/config.go

+	Namespace *common.Config `config:"config"`
+}
+
+func DefaultConfig() Config {


exported function DefaultConfig should have comment or be unexported

libbeat/common/kubernetes/metadata/config.go

houndci-bot · 2019-12-02T08:55:22Z

libbeat/common/kubernetes/metadata.go

@@ -38,6 +38,10 @@ type MetaGenerator interface {
 	ContainerMetadata(pod *Pod, container string, image string) common.MapStr
 }

+type MetaGen interface {


exported type MetaGen should have comment or be unexported

houndci-bot · 2019-12-02T08:55:22Z

libbeat/common/kubernetes/informer.go

+	}
+}
+
+func NewInformer(client kubernetes.Interface, resource Resource, opts WatchOptions) (cache.SharedInformer, string, error) {


exported function NewInformer should have comment or be unexported

houndci-bot · 2019-12-02T23:14:12Z

libbeat/common/kubernetes/metadata/service.go

+		}
+
+		return s.Generate(po, opts...)
+	} else {


if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)

libbeat/common/kubernetes/metadata/service.go

houndci-bot · 2019-12-02T23:14:13Z

libbeat/common/kubernetes/metadata/resource.go

+
+type resource = Config
+
+func NewResourceMetadataGenerator(cfg *common.Config) *resource {


exported function NewResourceMetadataGenerator should have comment or be unexported

libbeat/common/kubernetes/metadata/pod.go

houndci-bot · 2019-12-02T23:14:14Z

libbeat/common/kubernetes/metadata/node.go

+		}
+
+		return n.Generate(no, opts...)
+	} else {


if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)

libbeat/common/kubernetes/metadata/namespace.go

libbeat/common/kubernetes/metadata/config.go

houndci-bot · 2019-12-02T23:14:15Z

libbeat/common/kubernetes/informer.go

+
+	if indexers != nil {
+		return cache.NewSharedIndexInformer(listwatch, resource, opts.SyncTimeout, indexers), objType, nil
+	} else {


if block ends with a return statement, so drop this else and outdent its block

houndci-bot · 2019-12-02T23:14:15Z

libbeat/common/kubernetes/eventhandler.go

+
+}
+
+// OnAdd does a no-op on a delete event


comment on exported method NoOpEventHandlerFuncs.OnDelete should be of the form "OnDelete ..."

houndci-bot · 2019-12-02T23:14:16Z

libbeat/common/kubernetes/eventhandler.go

+
+}
+
+// OnAdd does a no-op on an update event


comment on exported method NoOpEventHandlerFuncs.OnUpdate should be of the form "OnUpdate ..."

houndci-bot · 2019-12-03T06:11:39Z

libbeat/common/kubernetes/metadata/resource.go

+	return &config
+}
+
+func (r *Resource) Generate(obj kubernetes.Resource, options ...FieldOptions) common.MapStr {


exported method Resource.Generate should have comment or be unexported

houndci-bot · 2019-12-03T06:11:40Z

libbeat/common/kubernetes/metadata/resource.go

+	"github.com/elastic/beats/libbeat/common/safemapstr"
+)
+
+type Resource = Config


exported type Resource should have comment or be unexported

houndci-bot · 2019-12-03T06:11:40Z

libbeat/common/kubernetes/metadata/resource.go

+	return &config
+}
+
+func (r *Resource) Generate(obj kubernetes.Resource, options ...FieldOptions) common.MapStr {


exported method Resource.Generate should have comment or be unexported

houndci-bot · 2019-12-03T06:11:40Z

libbeat/common/kubernetes/metadata/resource.go

+	"github.com/elastic/beats/libbeat/common/safemapstr"
+)
+
+type Resource = Config


exported type Resource should have comment or be unexported

libbeat/common/kubernetes/metadata/resource.go

houndci-bot · 2019-12-03T08:11:10Z

libbeat/common/kubernetes/metadata/namespace.go

+	resource *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for namespace resources


comment on exported function NewNamespaceMetadataGenerator should be of the form "NewNamespaceMetadataGenerator ..."

houndci-bot · 2019-12-03T08:11:11Z

libbeat/common/kubernetes/metadata/node.go

+	resource *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for service resources


comment on exported function NewNodeMetadataGenerator should be of the form "NewNodeMetadataGenerator ..."

houndci-bot · 2019-12-03T08:11:11Z

libbeat/common/kubernetes/metadata/namespace.go

+	resource *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for namespace resources


comment on exported function NewNamespaceMetadataGenerator should be of the form "NewNamespaceMetadataGenerator ..."

houndci-bot · 2019-12-03T08:11:12Z

libbeat/common/kubernetes/metadata/node.go

+	resource *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for service resources


comment on exported function NewNodeMetadataGenerator should be of the form "NewNodeMetadataGenerator ..."

houndci-bot · 2019-12-03T08:11:12Z

libbeat/common/kubernetes/metadata/service.go

+	resource  *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for service resources


comment on exported function NewServiceMetadataGenerator should be of the form "NewServiceMetadataGenerator ..."

houndci-bot · 2019-12-03T08:11:13Z

libbeat/common/kubernetes/metadata/node.go

+	resource *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for service resources


comment on exported function NewNodeMetadataGenerator should be of the form "NewNodeMetadataGenerator ..."

houndci-bot · 2019-12-03T08:11:14Z

libbeat/common/kubernetes/metadata/service.go

+	resource  *Resource
+}
+
+// NewPodMetadataGenerator creates a metagen for service resources


comment on exported function NewServiceMetadataGenerator should be of the form "NewServiceMetadataGenerator ..."

vjsamuel · 2019-12-03T08:11:55Z

#14738 needs to be merged first

vjsamuel · 2019-12-03T19:10:36Z

@exekias since this breaks the schema can you let me know places where that information need to be updated on Beats side?

vjsamuel · 2020-01-09T23:05:35Z

@odacremolbap i have made the necessary changes. i will create the git issue at the time of merge if that is ok?

odacremolbap · 2020-01-13T11:45:58Z

lgtm but there is still this request by @exekias #14875 (comment)

would you mindadding docs on the parameter?

odacremolbap · 2020-01-13T11:50:45Z

would you mindadding docs on the parameter?

I missed that you already added those docs.
/lgtm

exekias · 2020-01-13T14:25:44Z

ok to test

…es (elastic#14875) * Refactor metagen to allow multiple resources to be enriched (cherry picked from commit dba8f74)

…es (#14875) (#15528) * Refactor metagen to allow multiple resources to be enriched (cherry picked from commit dba8f74) Co-authored-by: Vijay Samuel <[email protected]>

houndci-bot reviewed Dec 2, 2019

View reviewed changes

houndci-bot reviewed Dec 3, 2019

View reviewed changes

vjsamuel changed the title ~~Meta refactor~~ Refactor metadata generator to support adding metadata across resources Dec 3, 2019

vjsamuel force-pushed the meta_refactor branch from fa3ef89 to 617fd17 Compare December 3, 2019 08:04

houndci-bot reviewed Dec 3, 2019

View reviewed changes

libbeat/common/kubernetes/metadata/resource.go Show resolved Hide resolved

houndci-bot reviewed Dec 3, 2019

View reviewed changes

libbeat/common/kubernetes/metadata/resource.go Show resolved Hide resolved

houndci-bot reviewed Dec 3, 2019

View reviewed changes

libbeat/common/kubernetes/metadata/resource.go Show resolved Hide resolved

vjsamuel force-pushed the meta_refactor branch from 617fd17 to 5a59671 Compare December 3, 2019 08:11

houndci-bot reviewed Dec 3, 2019

View reviewed changes

vjsamuel marked this pull request as ready for review December 3, 2019 08:11

vjsamuel requested a review from a team as a code owner December 3, 2019 08:11

vjsamuel force-pushed the meta_refactor branch from 5a59671 to 46bdf3b Compare December 3, 2019 08:12

vjsamuel force-pushed the meta_refactor branch 3 times, most recently from 5871578 to c433691 Compare December 3, 2019 22:03

exekias added autodiscovery review labels Dec 5, 2019

vjsamuel added 5 commits January 9, 2020 14:16

Fix missing key on metadata

eadab09

Fixing test cases

b6821c5

Incorporate review comments

f5fe8be

Add docs

feeff7f

Incorporate review comments

fcb2025

vjsamuel force-pushed the meta_refactor branch from e51dfff to fcb2025 Compare January 9, 2020 22:17

odacremolbap self-requested a review January 13, 2020 11:50

odacremolbap approved these changes Jan 13, 2020

View reviewed changes

Merge branch 'master' into meta_refactor

0b65027

exekias merged commit dba8f74 into elastic:master Jan 14, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Jan 14, 2020

exekias pushed a commit to exekias/beats that referenced this pull request Jan 14, 2020

Refactor metadata generator to support adding metadata across resourc…

7188bc8

…es (elastic#14875) * Refactor metagen to allow multiple resources to be enriched (cherry picked from commit dba8f74)

exekias mentioned this pull request Jan 14, 2020

Cherry-pick #14875 to 7.x: Refactor metadata generator to support adding metadata across resources #15528

Merged

exekias added the v7.6.0 label Jan 14, 2020

andresrc added the needs testing notes label Jan 14, 2020

andresrc unassigned exekias and odacremolbap Jan 14, 2020

odacremolbap mentioned this pull request Jan 15, 2020

Breaking changes for 8.0 #13911

Closed

19 tasks

vjsamuel deleted the meta_refactor branch January 16, 2020 18:35

andresrc removed the [zube]: Done label Jan 17, 2020

andresrc assigned odacremolbap Jan 20, 2020

odacremolbap added the test-plan-ok This PR passed manual testing label Feb 4, 2020

ChrsMark mentioned this pull request Feb 25, 2020

Define mapping for new k8s resources #16558

Closed

ChrsMark mentioned this pull request Sep 13, 2021

Add complete k8s metadata through composable provider #27691

Merged

7 tasks


		type FieldOptions func(common.MapStr) error

		func WithFields(key string, value interface{}) func(common.MapStr) {


		type resource = Config

		func NewResourceMetadataGenerator(cfg common.Config) resource {

Refactor metadata generator to support adding metadata across resources #14875

Refactor metadata generator to support adding metadata across resources #14875

Conversation

vjsamuel commented Dec 2, 2019 • edited by exekias Loading

elasticmachine commented Dec 2, 2019

elasticmachine commented Dec 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vjsamuel commented Dec 3, 2019

vjsamuel commented Dec 3, 2019

vjsamuel commented Jan 9, 2020

odacremolbap commented Jan 13, 2020

odacremolbap commented Jan 13, 2020

exekias commented Jan 13, 2020

vjsamuel commented Dec 2, 2019 •

edited by exekias

Loading