Filter Processor Proposal

[anniefu]

Filter Processor

A processor for dropping (AKA filtering) metrics and spans from the Collector pipeline based on specified string properties, for example, metric name or span name.

Motivation

From #297, "Processing unnecessary metrics causes undue burden on resources in otelsvc pipeline. It also increase the backend cost."

A filter processor combined with multiple otelvc pipeline's also enables increased versatility from a single collector service. Each pipeline can filter for a certain subset of metrics and apply very different processing on those subsets. This is a use case for Knative, which aims to support multiple types of metrics from both knative/serving and knative/eventing with the same collector service.

Proposed Configuration

Intent

• Keep consistency with attribute/span processor's filtering configuration

processors:
    filter:
        # spans is the configuration for filtering spans
        spans:
            # include and/or exclude can be specified. However, the include properties
            # are always checked before the exclude properties.
            {include, exclude}:
                # At least one of services, span_names or attributes must be specified.
                # It is supported to have more than one specified, but all of the specified
                # conditions must evaluate to true for a match to occur.

                # match_type controls how items in "services" and "span_names" arrays are
                # interpreted. Possible values are "regexp" or "strict".
                # This is a required field.
                match_type: {strict, regexp}

                # Configuration section for any additional filtering options for
                # a particular match_type. The match_type name is the name of the section.
                # This is an optional field.
                {strict, regexp}:
                    < see "match configuration" below >

                # services specify an array of items to match the service name against.
                # A match occurs if the span service name matches at least of the items.
                # This is an optional field.
                services: [<item1>, ..., <itemN>]

                # The span name must match at least one of the items.
                # This is an optional field.
                span_names: [<item1>, ..., <itemN>]

                # Attributes specifies the list of attributes to match against.
                # All of these attributes must match exactly for a match to occur.
                # Only match_type=strict is allowed if "attributes" are specified.
                # This is an optional field.
                attributes:
                    # Key specifies the attribute to match against.
                - key: <key>
                    # Value specifies the exact value to match against.
                    # If not specified, a match occurs if the key is present in the attributes.
                    value: {value}
        # metrics is the configuration for filtering metrocs
        metrics:
            # include and/or exclude can be specified. However, the include properties
            # are always checked before the exclude properties.
            {include, exclude}:
                # match_type controls how items in "metric_names" arrays are
                # interpreted. Possible values are "regexp" or "strict".
                # This is a required field.
                match_type: {strict, regexp}

                # Configuration section for any additional filtering options for
                # a particular match_type. The match_type name is the name of the section.
                # This is an optional field.
                {strict, regexp}:
                    < see "match configuration" below >

                # The metric name must match at least one of the items.
                # This is an optional field.
                meric_names: [<item1>, ..., <itemN>]

match configuration

Depending on the match_type, the optional additional filtering configuration are:

# additional configuration for match_type regexp
regexp:
    # when true, filter matches are cached to speed up future matches
    cacheenabled: <bool>
    # the size of the cache, specified by the number of matches to save; ignored if cacheenabled is false
    cachesize: <int>
    
# additional configuration for match_type strict
strict:
    # no options, but included for completeness

Implementation Interface

Goals

• Easy to add new string properties for filtering
• Easy to add new string match patterns schemes (i.e. regexp, glob, etc.)
• Support metrics and traces
• Re-usable across different processors

Metric Interface

Similar to the attribute/span processor interface, MatchSpan, a MatchMetric interface will be implemented so that different MetricProcessor's can re-use the same filtering logic for consistency.

// internal/processor/metric/metric.go
package metric
import (
        metricspb "github.com/census-instrumentation/opencensus-proto/gen-go/metrics/v1"
)
type Matcher interface {
	MatchMetric(*metricspb.Metric) bool
}

Shared Interface

// FilterSet is an interface for matching strings against a set of filters.
type FilterSet interface {
	// Matches returns true if the given string matches at least one
	// of the filters encapsulated by the FilterSet.
	Matches(string) bool
}

// FilterType indicates the type of filter pattern used for matching strings.
type FilterType string

const (
	// REGEXP filters use re2 regexp pattern matching.
	REGEXP FilterType = "regexp"
	// STRICT filters use exact string matching.
	STRICT FilterType = "strict"
)

Both the MatchSpan implementation and the MatchMetric implementation can use the same implementation of the FilterSet interface to match strings properties.

The Span/Metric implementations can maintain one FilterSet per property to filter against. To add new string properties for filtering, a new FilterSet is added.

MatchSpan and MatchMetric implementor's handle parsing the span or metric data for the string input to FilterSet.Matches(string). This helps keep metric & trace specific logic independent while sharing the core filtering logic.

To add a new string match pattern, implement a new FilterSet and add it to the FilterType enum. Add fields for match_type specific configuration to the config yaml as appropriate.

[anniefu]

@tigrannajaryan, @rghetia, @annanay25

[rghetia]

@anniefu thanks for writing the proposal. Couple of questions/comments

• This proposal states that it will drop the entire trace. Few things to consider for dropping entire trace.
  • spans would have to be grouped by traceid (like it is done for tail sampling) prior to applying the filter.
  • It would require same horizontal scaling solution as it is required for tail sampling.
  • It would require some other configuration such as wait time for grouping, max number of traces to hold, etc. From that perspective it is different from metrics filtering. Hence the two should be separate.
• Instead of dropping entire trace, only matching spans could be dropped but not sure how valuable it is.
• If there are multiple properties specified then is the match operation 'AND' or 'OR'? For example, label, name and resource label are specified. Is it included/excluded if all of them matches or only one of them matches?

[annanay25]

Agree with @rghetia. For traces, I think the right place to add these features would be as part of the filters in the tailsamplingprocessor. That would allow you to truly drop full traces.

[tigrannajaryan]

@anniefu thank you for the detailed description.

This would be a very reasonable proposal if we did not already have the include/exclude filters in existing processors. Unfortunately because they exist we have to take that into account and aim for uniform configuration syntax across all processors in this repo.

The include/exclude order is implementation-specific and users must be aware of the order when using both include and exclude.

This is true, but I do not think it is significant enough to warrant introducing a new way of filtering.

Allows users to specify as many include/exclude filters as desired by chaining multiple filter processors together
A similar chaining cannot be done with the include/exclude

This can be done by defining multiple filter processors, each specifying a pair. Processors chaining is already a feature.

The filter type for attribute/span processors is specified at the same level as the property being used to filter data. This means that each property must use the same filtering match type, since there is no way to specify a different one per property.

It is possible to specify different match types for each processor. This may be less flexible than allowing match type per property but probably good enough for simple cases.

Generally, the declarative approach to specifying filters is going to remain limited one way or another unless we end up re-creating an SQL-like language that is turing-complete.

My view on how the filters can be made more powerful and flexible is to introduce a new match type "eval", which allows to specify arbitrary expression (e.g. using something like https://github.com/antonmedv/expr or similar). This would also answer the question raised by @rghetia - AND and OR would be explicit operators in the eval expression (e.g. SpanName=="auth" OR Attributes["http.host"]="login.example.com"). The "eval" basically will be the escape hatch, anything that cannot be expressed via declarative pattern-match syntax will use the "eval" approach that can support arbitrarily nested expressions and any matching functions we desire to have. I do not believe that YAML-based syntax can be reasonably usable for complex cases, the explicit expressions are much simpler and more flexible for the power users.

Implementing "eval" in a way that is performant is non-trivial though and will require a significant upfront design and thought to do correctly.

How I would suggest to move forward with this proposal:

1. Try to refactor it in a way that makes the proposed config a superset of existing include/exclude filter pairs. Any valid include/exclude config should remain valid. We do not want to break existing configs and we do not want to introduce a new, slightly different way of filtering. For example see if a config like this is good enough for your needs:

processors:
  filter/allow-auth:
    include:
      match_type: regexp
      services: ["auth.*", "login.*"]
  filter/drop-authenticator:
    exclude:
      match_type: strict
      services: ["authenticator"]
service:
  pipelines:
    traces:
      processors: [filter/allow-auth, filter/drop-authenticator]

Note how you do not have to use both include and exclude in each processor and how using only one allows you to control the order of their application.

2. If it is not possible to do (1) then consider keeping the filter processor that you suggested in a separate repo that you own and use a custom build of Collector for your needs. The Collector is designed to make this easy. See for example how https://github.com/open-telemetry/opentelemetry-collector-contrib/ is done.

[anniefu]

Thanks for the quick feedback, everyone! Full disclosure, I have not worked with traces/spans much :)

@rghetia I meant filter spans instead of traces to be parallel with filtering the metrics array. I'm not very familiar with traces/spans, but it seems like being able to filter by span name or perhaps Resource just to reduce unnecessary processing could be worth implementing.

On the AND/OR, to match the existing attribute/span processor, it would be an OR between different metric/span properties.

@rghetia, @annanay25 do you think it's worth implementing a filterprocessor for spans or should that just be omitted and left to the tailsamplingprocessor? (I don't think I would be the one to implement that).

@tigrannajaryan Good point that different match types can be achieved already by chaining.

I agree that having two filter configs is not ideal, so I am happy to match the filter format of the attribute/span processor. Knative's use case is actually very simple, just one include filter of strict matches. But for consistency, I'll implement the components necessary to match the full attribute/span filter config.

I thought I would put this one out there just in case we were willing to diverge between processor, since the mutating nature of attribute/span processors have a few more restrictions to meet than one that's just dropping entire metrics/spans.

The eval match type would be very powerful, but I agree with you that it is unlikely that most users would need such flexibility,

Based off the feedback, I will update the design proposal:

• Replace filtering for traces with filtering for spans
• Match filter config used by span/attribute processor

[tigrannajaryan]

@anniefu I think a filter for individual spans can still be useful. For example there may be a particularly chatty service or part of service that generates large volume of useless or even incorrect spans (as a result of a bug) that we want to suppress until the bug is fixed.

I will update the design proposal
Sounds good. Please try to make the configuration (and implementation) of filters for spans and metrics a uniform as possible.

Let's see the modified proposal here before you proceed to implementation.

[anniefu]

Proposal updated, please take another look when you get a chance. Thanks!

[anniefu]

Bump on this. I'm happy to get the implementation done as soon as we reach a consensus.

[tigrannajaryan]

@anniefu Thank you for updating, sorry for the delay.

A few questions:

Q1.

attributes:
                    # Key specifies the attribute to match against.
                - key: <key>
                    # Value specifies the exact value to match against.
                    # If not specified, a match occurs if the key is present in the attributes.
                    value: {value}

Is this how we currently specify attributes in include/exclude filters? The syntax appears a bit different (perhaps it denotes the same thing in YAML, I am not sure).

Q2. No support to filter metrics by attributes/labels? This is fine, we can add it later, just double checking.

Q3.

# additional configuration for match_type regexp
regexp:
    # when true, filter matches are cached to speed up future matches
    cacheenabled: <bool>
    # the size of the cache, specified by the number of matches to save; ignored if cacheenabled is false
    cachesize: <int>

Can you clarify how the cache is intended to work?

Q4.

MatchMetric(*metricspb.Metric) bool

Why is serviceName not included like it is done for MatchSpan?

[anniefu]

@tigrannajaryan No worries, thanks for taking another look!

Q1.
Is this how we currently specify attributes in include/exclude filters? The syntax appears a bit different (perhaps it denotes the same thing in YAML, I am not sure).

I believe so (I just copy-pasted that part from the processor README).

Q2. No support to filter metrics by attributes/labels? This is fine, we can add it later, just double checking.

I think first iteration will just be setting up the interfaces/helper classes and the name field, which should set an example for adding more fields. The amount of code/test for that will already be large, so I am not planning on including attributes/labels at this time.

Q3.
Can you clarify how the cache is intended to work?

The cache will key off the string field being compared to (like name) and return true/false to whether it previously matched the filter. That way the cost of a regex match is replaced with the cost of a dictionary lookup.

This might be an over optimization, so it is a setting that can be used at will.

I do know for Knative we have a fixed set of metric names that will be repeatedly used, so it's unnecessary to repeatedly regexp match the same names and the cache of unique metric names will be relatively small (maybe 1k strings).

Q4.

MatchMetric(*metricspb.Metric) bool

Why is serviceName not included like it is done for MatchSpan?

Oh, this is because I was not planning on implementing support for serviceName yet, and it would just be dead code.

I guess we have a 1:1 interface:implementation match here, so maybe the best option is to remove the interface and just call MatchMetric implementation directly. That way in the future when people want to add more match fields, they don't have to update the interface AND the implementation stub.

We probably don't want different processors implementing different MatchMetric's anyway as that would lead to inconsistency in behavior across processors.

[anniefu]

Any additional thoughts or concern? I would like to get an implementation out before I get caught up in other projects :)

[ConverJens]

Very nice initiative! I'm looking exactly this functionality in order to setup several pipeline that needs to drop spans based on an attribute on them.
@anniefu My time is a bit limited but if you need help with this, perhaps I can assist you.

[vikrambe]

@tigrannajaryan I working on composite policy as discussed earlier. Looks like the functionality proposed here overlaps with composite policy.
#413

[tigrannajaryan]

@anniefu sorry for the delay. The proposal mostly looks good to me. Let's go ahead with implementation and we can discuss/decide the rest of the details in the PR.

[anniefu]

@ConverJens Great! I'm only planning on setting up the interfaces and name filtering right now, so maybe adding attributes filtering for spans would be something you'd be interested in adding after? I will CC you on the PRs to keep you in the loop.

@vikrambe @tigrannajaryan it does seem like there's some overlap in configuration with the composite tail filtering proposal, but functionally it is different enough that I don't see a problem with both of them existing.

I'm planning to split the PRs into smaller chunks as follows:

1. Add FilterSet interface, Regexp FilterSet implementation, Strict FilterSet implementation
2. Introduce new MatchMetric interface & swap MatchSpan to use FilterSet
3. In parallel
   3a. Metric filter processor config
   3b. Trace filter processor config
4. In parallel
   4a. Metric filter processor implementation
   4b. Trace filter processor implementation

[ConverJens]

@anniefu Sounds good! Let me know when you get going!

@vikrambe @tigrannajaryan I agree with @anniefu that the functionality is different enough to warrant both processors.

[william-tran]

Sorry for the scope creep, but I'll throw this out there. I'd like to apply filtering to span events as well as just spans. I'm playing with the idea of application loggers logging to OpenTracing span logs as a way to unify tracing and logging, and take advantage of trace sampling decisions. I want to apply the concept of enabling certain log levels to span events using this filter, e.g. filter out all DEBUG and TRACE level logs for com.acme loggers by default, but then in an incident to help with debugging, I'd remove those filters. I'm sure these features will become available when this project starts to take on straight-up logs, but I'd like a shortcut to that nirvana.

[anniefu]

@william-tran I'm more familiar with metrics than spans, but I will put out traceprocessor that filters spans as part of completing this proposal. My intent is to use the MatchSpan implementation, so the processor will be able to filter on anything that implementation can match on.

I will be updating MatchSpan to consolidate the implementation with the MatchMetric one proposed in this proposal, but I wasn't personally intending on adding additional span matching properties.

But in general, if the MatchSpan implementation is updated to work with span events too, the filterprocessor will pick up the ability to filter on that too.

[anniefu]

Hello, I have not forgotten about this, just got a bit busy with other projects. I'll have a filter metrics processor PR out sometime this week.

[jrcamp]

I'm a bit late to the party but would make one UX suggestion for this. Would suggest not having match_type and have it based on the match text instead. For example:

metrics:
    # include and/or exclude can be specified. However, the include properties
    # are always checked before the exclude properties.
    exclude:
        # Configuration section for any additional filtering options for
        # a particular match_type. The match_type name is the name of the section.
        # This is an optional field.
        {strict, regexp}:
            < see "match configuration" below >

        # The metric name must match at least one of the items.
        # This is an optional field.
        meric_names: [
            "host/cpu/*",       # glob matching
            "host/cpu/usage",   # exact matching
            "/cpu/",            # regex matching
            "!host/cpu/total"   # inverse match so that host/cpu/total isn't dropped
        ]

Many times a regex isn't needed and a glob will suffice. This allows the user to use whichever form of matching is more convenient for each entry.

[jrcamp]

I've made a dedicated proposal for this here: #1081

[FPAX]

I would like that rules (also in other processors) had an enable attribute (true by default) so configuration can be easily modified by turning on/off different rules and not deleting/inserting them. This would allow for quick and flexible reconfiguration when needed, without losing the declarations of rules that are not temporarily being removed. The true by default value would make it backwards compatible with existing config files while allowing for the new feature to be used.

[anniefu]

Given that both the attribute and span processor can filter spans already as is documented, do we see it as valuable to add the same functionality for spans to the filterprocessor?

As previously discussed, filtering entire traces is a bit more complex and would need design that is out of scope of the filterprocessor proposed here.

[flands]

@anniefu what is the latest on this?

[anniefu]

This is complete and available to use. For metrics, it's the filter processor. For spans, it's the attribute or span processor.

[johanbrandhorst]

Is it possible to drop spans without setting any attribute/span mutations? I tried the following and it errors:

span:
  exclude:
    match_type: regexp
    span_names:
      - opencensus\.proto\.agent\.+

The error is:

Error: cannot setup pipelines: cannot build pipelines: error creating processor "span" in pipeline "traces": error creating "span" processor: either "from_attributes" or "to_attributes" must be specified in "name:"

Am I doing something wrong? My hope is to drop these traces/spans altogether.

[sribalakumar]

Is it possible to drop spans without setting any attribute/span mutations? I tried the following and it errors:

span:
  exclude:
    match_type: regexp
    span_names:
      - opencensus\.proto\.agent\.+

The error is:

Error: cannot setup pipelines: cannot build pipelines: error creating processor "span" in pipeline "traces": error creating "span" processor: either "from_attributes" or "to_attributes" must be specified in "name:"

Am I doing something wrong? My hope is to drop these traces/spans altogether.

@johanbrandhorst Any luck with dropping spans? I have a similar use case and not able to drop spans.

processors:
  span:
    exclude:
      match_type: strict
      services: ["Sidekiq"]
      span_names: ["BRPOP"]
      attributes:
        - key: "db.type"
          value: "redis"

the error observed is

application run finished with error: cannot setup pipelines: cannot build pipelines: error creating processor "span" in pipeline "traces": error creating "span" processor: either "from_attributes" or "to_attributes" must be specified in "name:"

[johanbrandhorst]

Nope, not had any luck.

[ipaxos]

@sribalakumar @johanbrandhorst

Any updates regarding the above? I am facing the same issue.

[johanbrandhorst]

I still don't have a solution for this.

[pasali]

hi @anniefu and @tigrannajaryan
I couldn't find out how to drop spans with current functionality. In the docs, it says:

The attribute processor and the span processor expose the option to provide a set of properties of a span to match against to determine if the span should be included or excluded from the processor.

What i understand from this is its only work for processors. But i didn't see any drop-like action for attribute or span processor. Can you give an example to how to do this ?

[albertteoh]

Looking at the attributes processor and span processor, the exclude filters are used to exclude spans' attributes/name from being mutated, not dropping them. All spans will continue to be passed through the pipeline.

The tail sampling processor adopts an inclusion filter approach, based on span attributes and does not support exclusion at the moment. There are also plans to deprecate the processor.

The filterprocessor currently only supports metrics, and it does have the desired behaviour of dropping metrics; just not spans.

What I understand from this #560 (comment) is that it would still be useful to have a span filter. If so, I feel it would make most sense to have that logic within the filterprocessor (so it supports filtering both metrics and spans). What do folks think @tigrannajaryan @anniefu ? I'd be happy to contribute to this effort if you're both happy to go ahead with this proposal.