Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix syslog RFC 5424 parsing in CheckPoint module #21854

Merged

Conversation

andrewkroh
Copy link
Member

What does this PR do?

Change the input type in the CheckPoint module to udp from syslog
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.

Why is it important?

The module was causing warnings while parsing data.

Checklist

  • My code follows the style guidelines of this project
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

  1. Enable module.
  2. echo -n '<134>1 2020-03-29T13:19:21Z gw-da58d3 CheckPoint 1930 - [flags:"133440"; ifdir:"inbound"; ifname:"daemon"; loguid:"{0x5e80a059,0x2,0x6401a8c0,0x3c7878a}"; origin:"192.168.1.100"; sequencenum:"2"; version:"5"; product:"System Monitor"; sys_message::"installed Standard"]' | nc -4u -w1 localhost 9001
  3. Check Elasticsearch.

Related issues

Logs

I manually tested the syslog parsing using the steps above and this is the event that landed in ES.

{
          "checkpoint" : {
            "sys_message" : "installed Standard"
          },
          "agent" : {
            "name" : "mac",
            "id" : "a92a047c-c6c6-4025-8933-1672a1b99ae1",
            "type" : "filebeat",
            "ephemeral_id" : "658300b1-a9a5-45b9-b9ab-14c09edcee16",
            "version" : "8.0.0"
          },
          "log" : {
            "source" : {
              "address" : "127.0.0.1:60816"
            }
          },
          "fileset" : {
            "name" : "firewall"
          },
          "tags" : [
            "checkpoint-firewall",
            "forwarded"
          ],
          "network" : {
            "direction" : "inbound"
          },
          "input" : {
            "type" : "udp"
          },
          "observer" : {
            "ingress" : {
              "interface" : {
                "name" : "daemon"
              }
            },
            "product" : "System Monitor",
            "vendor" : "Checkpoint",
            "name" : "192.168.1.100",
            "type" : "firewall"
          },
          "@timestamp" : "2020-03-29T13:19:21.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "checkpoint"
          },
          "event" : {
            "sequence" : 2,
            "ingested" : "2020-10-15T13:41:59.409604500Z",
            "timezone" : "-04:00",
            "created" : "2020-10-15T13:41:55.836Z",
            "kind" : "event",
            "module" : "checkpoint",
            "id" : "{0x5e80a059,0x2,0x6401a8c0,0x3c7878a}",
            "category" : [
              "network"
            ],
            "dataset" : "checkpoint.firewall"
          }
        }

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 15, 2020
@andrewkroh andrewkroh force-pushed the bugfix/fb/checkpoint-syslog-rfc5424 branch from b4265dc to 4135c42 Compare October 15, 2020 14:24
@andrewkroh andrewkroh marked this pull request as ready for review October 15, 2020 14:25
Change the input type in the CheckPoint module to `udp` from `syslog`
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.
@andrewkroh andrewkroh force-pushed the bugfix/fb/checkpoint-syslog-rfc5424 branch from 4135c42 to cc73439 Compare October 15, 2020 14:28
@andrewkroh andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Oct 15, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 15, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21854 updated]

  • Start Time: 2020-10-15T14:28:54.969+0000

  • Duration: 62 min 29 sec

Test stats 🧪

Test Results
Failed 0
Passed 4435
Skipped 561
Total 4996

@andrewkroh andrewkroh merged commit f2e161f into elastic:master Oct 15, 2020
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Oct 15, 2020
Change the input type in the CheckPoint module to `udp` from `syslog`
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.

(cherry picked from commit f2e161f)
@andrewkroh andrewkroh added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Oct 15, 2020
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Oct 15, 2020
Change the input type in the CheckPoint module to `udp` from `syslog`
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.

(cherry picked from commit f2e161f)
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Oct 15, 2020
Change the input type in the CheckPoint module to `udp` from `syslog`
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.

(cherry picked from commit f2e161f)
v1v added a commit to v1v/beats that referenced this pull request Oct 19, 2020
* upstream/master: (23 commits)
  [Ingest Manager] Prevent reporting ecs version twice (elastic#21616)
  [CI] Use google storage to keep artifacts (elastic#21910)
  Update docs.asciidoc (elastic#21849)
  Kubernetes leaderelection improvements (elastic#21896)
  Apply name changes to elastic agent docs (elastic#21549)
  Add 7.7.1 relnotes to 7.8 docs (elastic#21937) (elastic#21941)
  [libbeat] Fix potential deadlock in the disk queue + add more unit tests (elastic#21930)
  Refactor docker watcher to fix flaky test and other small issues (elastic#21851)
  [CI] Add stage name in the step (elastic#21887)
  [docs] Remove extra word in autodiscover docs (elastic#21871)
  [CI] lint stage doesn't produce test reports (elastic#21888)
  Add tests of reader of filestream input (elastic#21814)
  [Ingest Manager] Use local temp instead of system one (elastic#21883)
  chore: delegate variant pushes to the right method (elastic#21861)
  [CI] kind setup fails sometimes (elastic#21857)
  Fix panic on add_docker_metadata close (elastic#21882)
  Add tests for fileProspector in filestream input (elastic#21712)
  [Filebeat][okta] Fix okta pagination (elastic#21797)
  Add cloud.account.id into add_cloud_metadata for gcp (elastic#21776)
  Fix syslog RFC 5424 parsing in CheckPoint module (elastic#21854)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 19, 2020
* upstream/master: (23 commits)
  [Ingest Manager] Prevent reporting ecs version twice (elastic#21616)
  [CI] Use google storage to keep artifacts (elastic#21910)
  Update docs.asciidoc (elastic#21849)
  Kubernetes leaderelection improvements (elastic#21896)
  Apply name changes to elastic agent docs (elastic#21549)
  Add 7.7.1 relnotes to 7.8 docs (elastic#21937) (elastic#21941)
  [libbeat] Fix potential deadlock in the disk queue + add more unit tests (elastic#21930)
  Refactor docker watcher to fix flaky test and other small issues (elastic#21851)
  [CI] Add stage name in the step (elastic#21887)
  [docs] Remove extra word in autodiscover docs (elastic#21871)
  [CI] lint stage doesn't produce test reports (elastic#21888)
  Add tests of reader of filestream input (elastic#21814)
  [Ingest Manager] Use local temp instead of system one (elastic#21883)
  chore: delegate variant pushes to the right method (elastic#21861)
  [CI] kind setup fails sometimes (elastic#21857)
  Fix panic on add_docker_metadata close (elastic#21882)
  Add tests for fileProspector in filestream input (elastic#21712)
  [Filebeat][okta] Fix okta pagination (elastic#21797)
  Add cloud.account.id into add_cloud_metadata for gcp (elastic#21776)
  Fix syslog RFC 5424 parsing in CheckPoint module (elastic#21854)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 19, 2020
…laky-test-analyser

* upstream/master: (22 commits)
  [Ingest Manager] Prevent reporting ecs version twice (elastic#21616)
  [CI] Use google storage to keep artifacts (elastic#21910)
  Update docs.asciidoc (elastic#21849)
  Kubernetes leaderelection improvements (elastic#21896)
  Apply name changes to elastic agent docs (elastic#21549)
  Add 7.7.1 relnotes to 7.8 docs (elastic#21937) (elastic#21941)
  [libbeat] Fix potential deadlock in the disk queue + add more unit tests (elastic#21930)
  Refactor docker watcher to fix flaky test and other small issues (elastic#21851)
  [CI] Add stage name in the step (elastic#21887)
  [docs] Remove extra word in autodiscover docs (elastic#21871)
  [CI] lint stage doesn't produce test reports (elastic#21888)
  Add tests of reader of filestream input (elastic#21814)
  [Ingest Manager] Use local temp instead of system one (elastic#21883)
  chore: delegate variant pushes to the right method (elastic#21861)
  [CI] kind setup fails sometimes (elastic#21857)
  Fix panic on add_docker_metadata close (elastic#21882)
  Add tests for fileProspector in filestream input (elastic#21712)
  [Filebeat][okta] Fix okta pagination (elastic#21797)
  Add cloud.account.id into add_cloud_metadata for gcp (elastic#21776)
  Fix syslog RFC 5424 parsing in CheckPoint module (elastic#21854)
  ...
@crisdarocha
Copy link

@andrewkroh just to confirm. Looking into the just released 7.9.3 Filebeat, it seems the change didn't make to this release.

I still see

type: syslog
protocol.udp:
  host: "{{.syslog_host}}:{{.syslog_port}}"

Could you please confirm if this is correct, or am I missing something?

Thanks a lot.

andrewkroh added a commit that referenced this pull request Oct 27, 2020
Change the input type in the CheckPoint module to `udp` from `syslog`
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.

(cherry picked from commit f2e161f)
andrewkroh added a commit that referenced this pull request Oct 27, 2020
Change the input type in the CheckPoint module to `udp` from `syslog`
so the syslog parsing happens in the ingest node pipeline rather than
in the Filebeat syslog input that only support RFC 3164.

(cherry picked from commit f2e161f)
@andrewkroh andrewkroh removed the v7.9.3 label Oct 27, 2020
andrewkroh added a commit to andrewkroh/integrations that referenced this pull request Oct 28, 2020
Sync the change from elastic/beats#21854 to use the UDP input instead of syslog input
to allow for RFC 5424 syslog parsing.
andrewkroh added a commit to andrewkroh/integrations that referenced this pull request Oct 28, 2020
Sync the change from elastic/beats#21854 to use the UDP input instead of syslog input
to allow for RFC 5424 syslog parsing.
andrewkroh added a commit to elastic/integrations that referenced this pull request Oct 29, 2020
* Update CheckPoint package with RFC 5424 parsing fix

Sync the change from elastic/beats#21854 to use the UDP input instead of syslog input
to allow for RFC 5424 syslog parsing.

* paths is required for logfile
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
* Update CheckPoint package with RFC 5424 parsing fix

Sync the change from elastic/beats#21854 to use the UDP input instead of syslog input
to allow for RFC 5424 syslog parsing.

* paths is required for logfile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants