[ resource generation processor] add new processor skeleton and configuration examples

[hossain-rayhan]

Signed-off-by: Rayhan Hossain [email protected]

This PR adds the skeleton and configuration examples of a new processor metricsgenerationprocessor.

Description:
The metrics generation processor (metricsgenerationprocessor) can be used to create new metrics using existing metrics following a given rule. Currently it supports following two approaches for creating a new metric.

1. It can create a new metric using two existing metrics applying one of the follwing arithmatic operations: add, subtract, multiply, divide and percent. One use case is to calculate the pod.memory.utilization metric like the following equation-
   pod.memory.utilization = (pod.memory.usage.bytes / node.memory.limit)
2. It can create a new metric by scaling the value of an existing metric with a given constant number. One use case is to convert pod.memory.usage metric values from Megabytes to Bytes (multiply the existing metric's value by 1000000)

Configuration

Configuration is specified through a list of generation rules. Generation rules find the metrics which
matche the given metric names and apply the operation to those metrics.

processors:
    # processor name: metricsgeneration
    metricsgeneration:

        # specify the metric generation rules
        generation_rules:
              # name of the new metric. this is a required field
            - new_metric_name: <new_metric_name>

              # generation_type describes how the metric will be generated. it can either be calculate or scale calculate generates a metric applying the given operation on two operand metrics. scale operates only on  operand1 metric to generate the new metric.
              generation_type: {calculate, scale}

              # this is a required field
              operand1_metric: <first_operand_metric>

              # this field is required only if the generation_type is calculate
              operand2_metric: <second_operand_metric>

              # operation specifies which atrithmatic operation to apply. it can be one of the five supported operations.
              operation: {add, subtract, multiply, divide, percent}

              # scale_by is only required if the generation_type is scale
              scale_by: 1000

Example Configurations

Create a new metric using two existing metrics

# create pod.cpu.utilized following (pod.cpu.usage / node.cpu.limit)
generation_rules:
    - new_metric_name: pod.cpu.utilized
      generation_type: calculate
      operand1_metric: pod.cpu.usage
      operand2_metric: node.cpu.limit
      operation: divide

Create a new metric scaling the value of an existing metric

# create pod.memory.usage.bytes from pod.memory.usage.megabytes
generation_rules:
    - new_metric_name: pod.memory.usage.bytes
      generation_type: scale
      operand1_metric: pod.memory.usage.megabytes
      operation: multiply
      scale_by: 1000000

Link to tracking Issue:
#2722

Testing:
Added unit tests. This is the skeleton only. Will add more test later.

Documentation:
README updated.

[dashpole]

Related PR, which just wants to add the ability to scale metrics to the existing metricstransformprocessor: #3177

[hossain-rayhan]

@bogdandrutu Would you please take a look on this?

[hossain-rayhan]

Hi @bogdandrutu a review is really appreciated.

[Aneurysm9]

Would simply rules be a better name for this field? I imagine it's always going to follow the declaration of the metricsgeneration processor and seems to stutter in that way. Similarly, some of the rule field names could be simplified by making use of their context:

processors:
  metricsgeneration:
    rules:
      - name: <new metric name>
        type: {calculate, scale}
        metric1: <first metric>
        metric2: <second metric>
        operation: ...

[hossain-rayhan]

Makes sense. will update.

[hossain-rayhan]

updated.

[Aneurysm9]

Suggested change

	processorConfig := cfg.(*Config)
	processorConfig, ok := cfg.(*Config)
	if !ok {
	// handle this error
	}

This coercion should be checked to avoid a potential panic.

[hossain-rayhan]

Updated.

[Aneurysm9]

Why is there a new type that mirrors the exported GenerationRule type?

[hossain-rayhan]

Updated.

[Aneurysm9]

generationTypes is now a map[GenerationType]struct{}, so is this output still appropriate? Same with operationTypes below.

[hossain-rayhan]

I don't think I understand what you said.

[Aneurysm9]

https://play.golang.org/p/B6CLhHkdkDj

Map:	map["bar":{} "foo":{}]
Slice:	["foo" "bar"]
MapKeys:	["foo" "bar"]

[hossain-rayhan]

Oh I see. Thanks. will update.

[Aneurysm9]

If addition is a valid operation should we allow 0? I'm not sure whether it makes sense to use this to copy a metric.

[hossain-rayhan]

For generation type calculate or scale I think this seems to be OK. For renaming, we may introduce another generation type rename which will filter based on metric labels and create a new one. The rename was part of our first proposal. But right now this is out of scope and we can achieve it with metricstransformprocessor.

[Aneurysm9]

I still think the rest of these field names should be simplified as well:

processors:
  metricsgeneration:
    rules:
      - name: <new metric name>
        type: {calculate, scale}
        metric1: <first metric>
        metric2: <second metric>
        operation: ...

[hossain-rayhan]

I prefer generation_type so that people don't mix it up with metric type. Also, I saw other examples where we used the prefix operand1_ which gives them clear idea about numerator and denominator. This design was discussed and I feel like current namings are OK. Expecting a second thought.

[Aneurysm9]

v0v

I feel that the current names are overly verbose and somewhat awkward, but it's not a hill I want to fight on.

[anuraaga]

Let's go with @Aneurysm9's suggested names, they look much better to me

[mxiamxia]

pls rebase

[mxiamxia]

LGTM!

[anuraaga]

Let's go with @Aneurysm9's suggested names, they look much better to me

[anuraaga]

Just a couple more renames, thanks

[hossain-rayhan]

Rebased again to see if we can pass the unexpected windows and trace test failure. Now the load-test for logs is failing which is not related to this change.

[alolita]

@bogdandrutu this PR has been reviewed by approved and should be ready-to-merge.