Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The severity_parser in filelogreceiver parses the level of log as 'debug' and not 'DEBUG' #26718

Closed
ghost opened this issue Sep 16, 2023 · 3 comments · Fixed by #27004
Closed

The severity_parser in filelogreceiver parses the level of log as 'debug' and not 'DEBUG' #26718

ghost opened this issue Sep 16, 2023 · 3 comments · Fixed by #27004
Labels
enhancement New feature or request receiver/filelog

Comments

@ghost
Copy link

ghost commented Sep 16, 2023 •
edited by ghost
Loading

Component(s)

receiver/filelog

What happened?

Description

I am trying to read logs from a file "gg.log" using the filelogreceiver.
There is only one entry in the file which looks something like this.
{"level":"DEBUG","additionalData":"bytes":1887389,"file":"BSE_CM.csv"},"requestId":"","fyId":"","location":"data_rest/develop/controllers.Downloadfroms3.func1[symbol_master_s3.go:110]","time":"2023-09-01T02:32:38Z","message":"Downloaded file"}

Steps to Reproduce

Please check the otel collector config.

Expected Result

Sep 16 13:50:21 LMAO otelcol-contrib[192989]: 2023-09-16T13:50:21.071+0530        info        ResourceLog #0
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Resource SchemaURL:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Resource attributes:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> service.name: Str(indus-history)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> service.version: Str(v1)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: ScopeLogs #0
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: ScopeLogs SchemaURL:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: InstrumentationScope
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: LogRecord #0
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: ObservedTimestamp: 2023-09-16 08:20:20.874016581 +0000 UTC
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Timestamp: 2023-09-01 02:32:38 +0000 UTC
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: SeverityText: DEBUG
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: SeverityNumber: Debug(5)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Body: Str({"level":"debug","additionalData":{"bytes":1887389,"file":"BSE_CM.csv"},"requestId":"","fyId":"","location":"data_rest/develop/controllers.Downloadfroms3.func1[symbol_master_s3.go:110]","time":"2023-09-01T02:32:38Z","message":"Downloaded file"})
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Attributes:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> message: Str(Downloaded file)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> level: Str(debug)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> additionalData: Map({"bytes":1887389,"file":"BSE_CM.csv"})
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> requestId: Str()
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> log.file.name: Str(gg.log)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> fyId: Str()
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> location: Str(data_rest/develop/controllers.Downloadfroms3.func1[symbol_master_s3.go:110])
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> time: Str(2023-09-01T02:32:38Z)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Trace ID:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Span ID:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Flags: 0

Actual Result

Sep 16 13:50:21 LMAO otelcol-contrib[192989]: 2023-09-16T13:50:21.071+0530        info        ResourceLog #0
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Resource SchemaURL:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Resource attributes:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> service.name: Str(indus-history)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> service.version: Str(v1)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: ScopeLogs #0
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: ScopeLogs SchemaURL:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: InstrumentationScope
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: LogRecord #0
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: ObservedTimestamp: 2023-09-16 08:20:20.874016581 +0000 UTC
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Timestamp: 2023-09-01 02:32:38 +0000 UTC
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: SeverityText: debug
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: SeverityNumber: Debug(5)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Body: Str({"level":"debug","additionalData":{"bytes":1887389,"file":"BSE_CM.csv"},"requestId":"","fyId":"","location":"data_rest/develop/controllers.Downloadfroms3.func1[symbol_master_s3.go:110]","time":"2023-09-01T02:32:38Z","message":"Downloaded file"})
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Attributes:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> message: Str(Downloaded file)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> level: Str(debug)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> additionalData: Map({"bytes":1887389,"file":"BSE_CM.csv"})
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> requestId: Str()
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> log.file.name: Str(gg.log)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> fyId: Str()
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> location: Str(data_rest/develop/controllers.Downloadfroms3.func1[symbol_master_s3.go:110])
Sep 16 13:50:21 LMAO otelcol-contrib[192989]:      -> time: Str(2023-09-01T02:32:38Z)
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Trace ID:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Span ID:
Sep 16 13:50:21 LMAO otelcol-contrib[192989]: Flags: 0

The field I am pointing at is the SeverityText field.

The result I am getting is SeverityText: debug

But according to this opentelemetry doc
The result should have been SeverityText: DEBUG

i.e log level in uppercase ('DEBUG' and not 'debug').

If I am wrong with the assumption that "the level needs to be always uppercase", then let me know, I will close this.

Apologies if I might have missed something or misinterpret anything.
Thanks & Regards

Collector version

v0.83.0

Environment information

Environment

OS: (e.g., "Ubuntu 22.04")

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ /var/log/gg.log ]
    start_at: beginning
    operators:
    - type: add
      field: resource["service.name"]
      value: indus-history
    - type: json_parser
    - type: severity_parser
      parse_from: attributes.level
      preset: default
    - type: time_parser
      parse_from: attributes.time
      layout: "%Y-%m-%dT%H:%M:%S.%LZ"

processors:
  batch:

exporters:
  logging:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [logging]

Log output

No response

Additional context

No response
@ghost ghost added bug Something isn't working needs triage New item requiring triage labels Sep 16, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Sep 19, 2023
Open
@djaglowski
Copy link
Member

Thanks for reporting about this @shiva-fyers.

I think technically the behavior is correct. However you are not the first person to expect something else here, so I'm happy to report I think I have a solution for you.

First, regarding the reason why I think this is correct behavior. It's not called out very prominently, but in our specification, we describe severity text as the original string representation of the severity as it is known at the source. So basically, we preserve the text exactly as is, but also can interpret it into a number.

Very recently a similar question was raised, regarding whether it should be possible to overwrite the severity text with the official strings that correspond to the numbers. See #26671. I believe this same solution would provide the behavior you expect. I will try to get a PR out for it this week.

@djaglowski djaglowski added enhancement New feature or request and removed needs triage New item requiring triage bug Something isn't working labels Sep 19, 2023
@djaglowski djaglowski mentioned this issue Sep 19, 2023
Merged
@ghost
Copy link
Author

ghost commented Sep 19, 2023 •
edited by ghost
Loading

I understand @djaglowski .

I couldn't be more grateful for your response and time.
I will be closing this.

Thanks and Regards

@ghost ghost closed this as completed Sep 19, 2023
djaglowski added a commit that referenced this issue Sep 21, 2023
@djaglowski
[pkg/stanza] Add 'overwrite_text' option to severity parser (#27004)
ccfd8b1 
Resolves #26671
Resolves #26718
jmsnll pushed a commit to jmsnll/opentelemetry-collector-contrib that referenced this issue Nov 12, 2023
@djaglowski @jmsnll
[pkg/stanza] Add 'overwrite_text' option to severity parser (open-tel…
b01ee67 
…emetry#27004)

Resolves open-telemetry#26671
Resolves open-telemetry#26718
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request receiver/filelog
Projects
None yet
Milestone
No milestone
1 participant
@djaglowski