libbeat/reader/syslog: relax timestamp parsing to allow leading zero

[efd6]

What does this PR do?

This change relaxes the RFC3164 timestamp grammar to allow dates with a leading
zero to be parsed as valid syslog timestamps, bringing the parser's behaviour into
line with the parser in filebeat/input/syslog.

Why is it important?

There are non-standard syslog generating implementations that we were previously unable to consume. With this change we also gain the possibility of reducing the number of syslog parser implementations that we have.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Confirm that the equivalent test in filebeat/input/syslog matches (there is one difference in that the hostname in that package's test does not conform to the RFC).
• This is arguably a bug fix even though it is a break from the strict definition in the RFC. Opinions on this are welcome.

How to test this PR locally

go test github.com/elastic/beats/v7/libbeat/reader/syslog

Related issues

• Relates [FileBeat - syslog input] More lenient parsing of RFC3164 dates #16824
• Relates Make Syslog parser/processor handle errors more gracefully #31246

Use cases

Screenshots

Logs

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-04-15T02:27:52.451+0000

• Duration: 70 min 11 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22301
Skipped	1935
Total	24236

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b syslogrelaxdate upstream/syslogrelaxdate
git merge upstream/main
git push upstream syslogrelaxdate

[taylor-swanson]

LGTM

[taylor-swanson]

This is arguably a bug fix even though it is a break from the strict definition in the RFC

As much as I'd like to strictly adhere to RFC, it doesn't help us trying to ingest from as many sources as possible. The more I look at it, the more I realize it is not our job to be enforcers of RFC, there are better tools out there if a user really wants to do that. I think for us, leaning towards best effort rather than strict RFC adherence is the way to go. Another example would be my latest change that makes the priority field optional, which is a blatant violation of RFC (yet many syslog providers do just that).

I have been looking at making the parser a lot more tolerant and being able to recover from certain errors, but it's proving to be more difficult and time consuming than I'd like. @efd6 I know you recently did some work with the cef decoder and recovery. Feel free to look at it if you've got time. The problem I'm running into is getting the state machine to transition to the "next state" (easier said than done). Further discussion on that topic should probably be done on the main issue, #31246

[taylor-swanson]

Should this be backported to 8.2.0?

[efd6]

Happy to take a look @taylor-swanson — I'm not surprised you are having difficulties; starting a robust machine with throwing out your invariants is hard/foolish (coming from someone who has spent time doing that foolishness).

This should be back ported if it's considered a bug. @andrewkroh?

[andrewkroh]

IIRC we documented the time parsing behavior of the syslog processor/parser and deviations from the RFC. Please check the docs and update them accordingly.

As for backports, I think this should go into 8.2.

[efd6]

@andrewkroh Please take a look at the doc wording. Also, since the issue that this fixes has been around for two years, should I back port to 7.17, and 8.1(too late for 8.1)?

[andrewkroh]

The syslog processor being changed is new and only exists in main/8.2. So users that want this fix will need to upgrade to take advantage of the new processor.