Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[processor/filter] Filters out all logs if logBodies used and logs aren't strings #22736

Closed
sumo-drosiek opened this issue May 24, 2023 · 4 comments
Labels
bug Something isn't working needs triage New item requiring triage processor/filter Filter processor

Comments

@sumo-drosiek
Copy link
Member

Component(s)

processor/filter

What happened?

Description

All logs are filtered out by filter processor if LogBodies is being used and logs are not a string

Steps to Reproduce

Run configuration with example log

Expected Result

Logs are not filtered out

Actual Result

Logs are filtered out

Collector version

latest

Environment information

Environment

N/A

OpenTelemetry Collector configuration

receivers:
  filelog:
    include:
      - ./tmp/2.log
    start_at: beginning
processors:
  filter/exclude-phi:
    logs:
      exclude:
        bodies:
        - not matches anything
  transform/containers_parse_json:
    error_mode: ignore
    log_statements:
    - context: log
      statements:
      - set(body, ParseJSON(body)) where IsMatch(body, "^{") == true
exporters:
  logging:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
        - transform/containers_parse_json
        - filter/exclude-phi
      exporters:
        - logging

Log output

No response

Additional context

./tmp/2.log:

{"level":"info"}
@sumo-drosiek sumo-drosiek added bug Something isn't working needs triage New item requiring triage labels May 24, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the processor/filter Filter processor label May 24, 2023
@sumo-drosiek

This comment was marked as outdated.

@sumo-drosiek
Copy link
Member Author

The following configuration may be use as alternative:

  filter:
    logs:
      log_record:
      - 'IsMatch(body, "not matches anything")'

@TylerHelmuth
Copy link
Member

I believe #22102 solves this issue.

Another option is to swap the order of the filterprocessor and transformprocessor and do the filtering on the string before the transformprocessor turns the body into a map.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs triage New item requiring triage processor/filter Filter processor
Projects
None yet
Development

No branches or pull requests

2 participants