[WIP] Filebeat module issues found during ECS conversion

[webmat]

Please ignore for now.

This is just a collection of all the small problems we notice could be improved while performing the ECS transition (#8655). Some things are small enough that we can tackle them right in the ECS PRs.

Some things are out of scope, though. Here they are:

Usage of http.response.elapsed_time

This field is not in ECS and should be deprecated in favour of event.duration. Modules using it:

• kibana

Full version strings vs breakdown fields

• Ingest Node's UA parser gives us the version number in broken up in 3 fields (major/minor/patch), but no full version string (e.g. 10.14.1). We should populate user_agent.version and user_agent.os.version with full version strings. Modules affected:
  • apache2.access
  • iis.access
  • nginx.access
  • traefik.access

system.syslog

• Add integration test for the convert_timezone option

system.auth

• Add integration tests for message formats other than sshd, sudo, groupadd and useradd. Right now it's only testing for those, and no test for the last GROK pattern that just populates the message field.
• Patterns specific to sshd, sudo, groupadd and useradd
  • Do not set process.name. They should.
  • Do not set message. Although this point will be solved by Filebeat modules: keep raw message #8083.
• After performing field renames on auth messages, I wonder if the resulting events are missing something. For example the first pattern uses only ECS fields, if it wasn't for one last field that hasn't been renamed (system.auth.ssh.method, line 29), I would never know that its ECS event is a message about SSH.

nginx.access

• Current parsing of access log assumes people are using $http_x_forwarded_for rather than the default $remote_addr for their remote address. This means even though nginx can receive connections from unix sockets (e.g. common when polling for nginx stats), this module doesn't support these log events.

Message in error filesets

E.g. apache.error, nginx.error

• Currently outputs to message rather than error.message, since all Ingest Node documentation currently suggests outputting Ingest Node error messages at error.message.
  • I'm not 100% convinced about using error.message in any case for userland error messages, by the way. We've been talking about always populating message for the timeline viewer.

[ruflin]

Thanks for this issue, let's keep adding things. I think if possible we should resolve these issues for 7.0.

[alvarolobato]

@ruflin will break this down into individual issues after all the ECS conversion is processed.

[ruflin]

• user_agent change will be address by ingest processor change in Elasticsearch
• Follow up for convert_timezone test can be found here: Add ability to read config file in test_modules.py for filebeat #10182
• Follow up issue for nginx.access change can be found here: Nginx.access fileset improvement #10309

Closing this issue.