Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changed input from syslog to tcp/udp due to unsupported RFC #18447

Merged
merged 3 commits into from
May 14, 2020

Conversation

P1llus
Copy link
Member

@P1llus P1llus commented May 12, 2020

What does this PR do?

Changes beat input in docs and configuration file due to unsupported RFC syslog pattern.

Why is it important?

Module needs this fix to work properly.

Checklist

Added the changes, updated the documentation, ran nosetests and confirmed by testing with netcat to simulate the syslog input.

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 12, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented May 12, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 1
Passed 2780
Skipped 418
Total 3199

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat Mac OS X / test_clean_removed_with_clean_inactive – test_registrar.Test
    • Status: FAILED
    • Age: 1
    • Duration: 0.93
    • Error Details:
      -------------------- >> begin captured stdout << ---------------------
      registry size: 2
      registry size after remove: 2

--------------------- >> end captured stdout << ----------------------

Steps errors

Expand to view the steps failures

  • Name: Mage build unitTest
    • Description: mage build unitTest

    • Result: FAILURE

    • Duration: 6 min 26 sec

    • Start Time: 2020-05-14T11:18:41.514+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-14T11:46:48.403Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-05-14T11:46:48.403Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-05-14T11:46:48.403Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-05-14T11:46:48.403Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-05-14T11:46:48.403Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-05-14T11:46:48.403Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-05-14T11:46:48.784Z] + curl -sSLo codecov https://codecov.io/bash
[2020-05-14T11:46:49.044Z] + FILE=auditbeat/build/coverage/full.cov
[2020-05-14T11:46:49.044Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-05-14T11:46:49.044Z] + FILE=filebeat/build/coverage/full.cov
[2020-05-14T11:46:49.044Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-05-14T11:46:49.044Z] + bash codecov -f filebeat/build/coverage/full.cov
[2020-05-14T11:46:49.044Z] 
[2020-05-14T11:46:49.044Z]   _____          _
[2020-05-14T11:46:49.044Z]  / ____|        | |
[2020-05-14T11:46:49.044Z] | |     ___   __| | ___  ___ _____   __
[2020-05-14T11:46:49.044Z] | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
[2020-05-14T11:46:49.044Z] | |___| (_) | (_| |  __/ (_| (_) \ V /
[2020-05-14T11:46:49.044Z]  \_____\___/ \__,_|\___|\___\___/ \_/
[2020-05-14T11:46:49.044Z]                               Bash-tbd
[2020-05-14T11:46:49.044Z] 
[2020-05-14T11:46:49.044Z] 
[2020-05-14T11:46:49.044Z] ==> Jenkins CI detected.
[2020-05-14T11:46:49.044Z]     project root: .
[2020-05-14T11:46:49.044Z]     Fixing merge commit SHA
[2020-05-14T11:46:49.044Z]     Yaml found at: codecov.yml
[2020-05-14T11:46:49.044Z]     -> Found 1 reports
[2020-05-14T11:46:49.044Z] ==> Detecting git/mercurial file structure
[2020-05-14T11:46:49.303Z] ==> Reading reports
[2020-05-14T11:46:49.303Z]     + filebeat/build/coverage/full.cov bytes=261000
[2020-05-14T11:46:49.303Z] ==> Appending adjustments
[2020-05-14T11:46:49.303Z]     http://docs.codecov.io/docs/fixing-reports
[2020-05-14T11:46:57.435Z]     + Found adjustments
[2020-05-14T11:46:57.436Z] ==> Gzipping contents
[2020-05-14T11:46:57.695Z] ==> Uploading reports
[2020-05-14T11:46:57.695Z]     url: https://codecov.io
[2020-05-14T11:46:57.695Z]     query: branch=PR-18447&commit=fe12fdd1c9219791d69541fa18306666b7e1bd04&build=5&build_url=https%3A%2F%2Fbeats-ci.elastic.co%2Fjob%2FBeats%2Fjob%2Fbeats-beats-mbp%2Fjob%2FPR-18447%2F5%2F&name=&tag=&slug=elastic%2Fbeats&service=jenkins&flags=&pr=18447&job=
[2020-05-14T11:46:57.695Z]     -> Pinging Codecov
[2020-05-14T11:46:57.695Z] https://codecov.io/upload/v4?package=bash-tbd&token=secret&branch=PR-18447&commit=fe12fdd1c9219791d69541fa18306666b7e1bd04&build=5&build_url=https%3A%2F%2Fbeats-ci.elastic.co%2Fjob%2FBeats%2Fjob%2Fbeats-beats-mbp%2Fjob%2FPR-18447%2F5%2F&name=&tag=&slug=elastic%2Fbeats&service=jenkins&flags=&pr=18447&job=
[2020-05-14T11:46:57.955Z] HTTP 400
[2020-05-14T11:46:57.955Z] Please provide the repository token to upload reports via `-t :repository-token`
[2020-05-14T11:46:57.955Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-14T11:46:57.955Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-05-14T11:46:57.955Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-14T11:46:57.955Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-05-14T11:46:57.955Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-14T11:46:57.955Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-05-14T11:46:57.955Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-14T11:46:57.955Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-14T11:46:57.955Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-14T11:46:57.955Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-14T11:46:57.955Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-14T11:46:57.955Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-14T11:46:59.509Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats
[2020-05-14T11:46:59.821Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-14T11:46:59.835Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats/Lint
[2020-05-14T11:46:59.902Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-05-14T11:46:59.973Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats/Filebeat-Windows
[2020-05-14T11:47:00.039Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats/Filebeat-x-pack
[2020-05-14T11:47:00.105Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats/Filebeat-oss
[2020-05-14T11:47:00.460Z] + cat
[2020-05-14T11:47:00.460Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-14T11:47:00.460Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-14T11:47:08.632Z] runbld>>> runbld started
[2020-05-14T11:47:08.632Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-14T11:47:09.208Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18447' in order of occurrence in the config (last value wins).
[2020-05-14T11:47:10.598Z] runbld>>> Debug logging enabled.
[2020-05-14T11:47:10.598Z] runbld>>> Storing result
[2020-05-14T11:47:10.865Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-14T11:47:10.865Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200514114710-238F117E
[2020-05-14T11:47:10.865Z] runbld>>> Adding system facts.
[2020-05-14T11:47:11.820Z] runbld>>> Adding vcs info for the latest commit:  af744bdc7156f84f5d5894a7a96fe8eb67814040
[2020-05-14T11:47:12.083Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-14T11:47:12.083Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-14T11:47:12.083Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-14T11:47:12.083Z] Processing JUnit reports with runbld...
[2020-05-14T11:47:12.344Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-14T11:47:12.345Z] runbld>>> DURATION: 9ms
[2020-05-14T11:47:12.345Z] runbld>>> STDOUT: 40 bytes
[2020-05-14T11:47:12.345Z] runbld>>> STDERR: 49 bytes
[2020-05-14T11:47:12.345Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-14T11:47:12.345Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats
[2020-05-14T11:47:13.737Z] runbld>>> Storing build metadata: 
[2020-05-14T11:47:13.737Z] runbld>>> Adding test report.
[2020-05-14T11:47:13.737Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447/src/github.com/elastic/beats
[2020-05-14T11:47:15.133Z] runbld>>> Found 9 test output files
[2020-05-14T11:47:16.083Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 3199 Skipped: 408
[2020-05-14T11:47:16.084Z] runbld>>> Storing result
[2020-05-14T11:47:16.084Z] runbld>>> FAILURES: 1
[2020-05-14T11:47:16.656Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-14T11:47:16.657Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200514114710-238F117E
[2020-05-14T11:47:16.657Z] runbld>>> Email notification disabled by environment variable.
[2020-05-14T11:47:16.657Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-14T11:47:22.551Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18447
[2020-05-14T11:47:22.862Z] [INFO] getVaultSecret: Getting secrets
[2020-05-14T11:47:22.917Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-14T11:47:23.695Z] + chmod 755 generate-build-data.sh
[2020-05-14T11:47:23.695Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18447/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18447/runs/5 FAILURE 3535947
[2020-05-14T11:47:24.246Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18447/runs/5/steps/?limit=10000 -o steps-info.json
[2020-05-14T11:47:24.797Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18447/runs/5/tests/?status=FAILED -o tests-errors.json

@P1llus P1llus removed the needs_team Indicates that the issue/PR needs a Team:* label label May 12, 2020
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you manually check the udp input with a netcat message?


The interface to listen to UDP based syslog traffic. Defaults to localhost.
The protocol to use, can be either the value `tcp` or `udp`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about mentioning file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you manually check the udp input with a netcat message?

I tested with both TCP and UDP. TCP works with the same amount of messages and 0 error.messages filled, UDP also works but I see a few GROK missmatch messages (with the same logs). This seems to be because it sends the whole 300 messages at the same time, and all messages that gets through gets parsed correctly, while the other few gets a grok error message.

Any idea here? Testing seems fine, and If I tested with a smaller set of logs it should still work fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some new testing, UDP is also fine.

@@ -1,14 +1,14 @@
module_version: 1.0

var:
- name: syslog_host
- name: host
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you might still want to call this variables syslog_host and syslog_port to stay aligned with the rest of the modules. The underlying protocol is still syslog (more or less), right?

@P1llus P1llus requested review from andrewkroh and adriansr May 14, 2020 06:59
Copy link
Contributor

@adriansr adriansr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All review comments have been addressed and CI failure is unrelated.

@adriansr adriansr merged commit 140cc11 into elastic:master May 14, 2020
adriansr pushed a commit to adriansr/beats that referenced this pull request May 14, 2020
…18447)

Changes beat input in docs and configuration file due to unsupported RFC syslog pattern.

(cherry picked from commit 140cc11)
adriansr pushed a commit to adriansr/beats that referenced this pull request May 14, 2020
…18447)

Changes beat input in docs and configuration file due to unsupported RFC syslog pattern.

(cherry picked from commit 140cc11)
v1v added a commit to v1v/beats that referenced this pull request May 15, 2020
…w-oss

* upstream/master: (27 commits)
  Disable host fields for "cloud", panw, cef modules (elastic#18223)
  [docs] Rename monitoring collection from legacy internal collection to legacy collection (elastic#18504)
  Introduce auto detection of format (elastic#18095)
  Add additional fields to address issue elastic#18465 for googlecloud audit log (elastic#18472)
  Fix libbeat import path in seccomp policy template (elastic#18418)
  Address Okta input issue elastic#18530 (elastic#18534)
  [Ingest Manager] Avoid Chown on windows (elastic#18512)
  Fix Cisco ASA/FTD msgs that use a host name as NAT address (elastic#18376)
  [CI] Optimise stash/unstash performance (elastic#18473)
  Libbeat: Remove global loggers from libbeat/metric and libbeat/cloudid (elastic#18500)
  Fix PANW bad mapping of client/source and server/dest packets and bytes (elastic#18525)
  Add a file lock to the data directory on startup to prevent multiple agents. (elastic#18483)
  Followup to 12606 (elastic#18316)
  changed input from syslog to tcp/udp due to unsupported RFC (elastic#18447)
  Improve ECS field mappings in Sysmon module. (elastic#18381)
  [Elastic Agent] Cleaner output of inspect command  (elastic#18405)
  [Elastic Agent] Pick up version from libbeat (elastic#18350)
  Update communitybeats.asciidoc (elastic#18470)
  [Metricbeat] Change visualization interval from 15m to >=15m (elastic#18466)
  docs: Fix typo in kerberos docs (elastic#18503)
  ...
adriansr added a commit that referenced this pull request May 15, 2020
…18528)

Changes beat input in docs and configuration file due to unsupported RFC syslog pattern.

(cherry picked from commit 140cc11)

Co-authored-by: Marius Iversen <[email protected]>
adriansr added a commit that referenced this pull request May 15, 2020
…18527)

Changes beat input in docs and configuration file due to unsupported RFC syslog pattern.

(cherry picked from commit 140cc11)

Co-authored-by: Marius Iversen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants