Cherry-pick #17253 to 7.7: Ignore trailing spaces in CEF messages

[adriansr]

Cherry-pick of PR #17253 to 7.7 branch. Original message:

What does this PR do?

This patch updates the Ragel state machine to skip trailing space at the end of CEF messages.

Some CEF exporters, Check Point for example, have been observed to add a trailing space to CEF messages:

"CEF:0:| [...] src=127.0.0.1 "

Currently, this space character is interpreted as part of the last field's value, which can cause decoding errors if the value is an integer or an IP address:

"error": {
    "message": "sourceAddress: value is not a valid IP address"
  }

For maximizing compatibility, we also want to ignore other kinds of white-space characters (newline, carriage return, tab). For example we can get a trailing newline when processing CEF messages from UDP input instead of syslog, which removes newlines.

Trailing space in other extensions' values is preserved, as the CEF standard permits (but discourages) it's use in non-final extensions.

Why is it important?

We've observed users experiencing this problem, trying to fix it (unsuccessfully) with a processor:

- dissect:
      when:
        equals:
          event.module: "cef"
      tokenizer: "%{sourceAddress} "
      field: "cef.extensions.sourceAddress"
      target_prefix: "cef.extensions"

Why a draft?

While the current solution is complete, I'm not satisfied of the changes that introduces to the Ragel SM definition.

Originally I wanted to get something more elegant like this to work:

extensions = (extension " ")* final_extension space*

So that we can keep the extension machine as-is, and add a specialized final_extension machine that will disallow trailing space. However, I didn't manage to get this kind of pattern to work. It requires rewriting a lot of the capture actions to allow for the necessary backtracking, and I wasn't confident enough to get that right without introducing more problems than I was solving, or dedicating too many hours to this fix, due to my limited experience with ragel.

The current solution works by accident. I decided to try a different approach starting by disallowing trailing space in all extensions, and found out that it works as I wanted and captures white-space as value in all extensions but the last. This is a side effect of how an extension value is captured differently in extension_key vs extension_eof. The former captures the previous value up to mark-1, that is the start of the current key minus the space separator. The later captures up to extValueEnd, which is not incremented for trailing space.

The current machine definition is an ugly mix of " " and space usage, because we want to have the space character as the only separator, but trim al trailing space: [\t\v\f\n\r ]

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)