Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #24763 to 7.x: Add support for parsers in filestream input #25224

Merged
merged 1 commit into from
Apr 22, 2021

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Apr 22, 2021

Cherry-pick of PR #24763 to 7.x branch. Original message:

This PR adds support for pasers in the filestream input.

Example configuration that aggregates fives lines into a single event and parses the JSON contents:

- type: filestream
  enabled: true
  paths:
  - test.log
  parsers:
  - multiline:
      type: count
      count_lines: 5
      skip_newline: true
  - ndjson:
      fields_under_root: true

This PR adds support for pasers in the `filestream` input.

Example configuration that aggregates fives lines into a single event and parses the JSON contents:
```yaml
- type: filestream
  enabled: true
  paths:
  - test.log
  parsers:
  - multiline:
      type: count
      count_lines: 5
      skip_newline: true
  - ndjson:
      fields_under_root: true
```

(cherry picked from commit 30331bc)
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 22, 2021
@elasticmachine
Copy link
Collaborator

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25224 opened

  • Start Time: 2021-04-22T08:34:14.369+0000

  • Duration: 105 min 25 sec

  • Commit: 3b40e99

Test stats 🧪

Test Results
Failed 1
Passed 47163
Skipped 5190
Total 52354

Trends 🧪

Image of Build Times

Image of Tests

Test errors 1

Expand to view the tests failures

Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test
    Expand to view the error details

     AssertionError: Element counts were not equal: First has 1, Second has 0:  'logstash' First has 0, Second has 1:  'error' 
    

    Expand to view the stacktrace

     self = <test_logstash.Test testMethod=test_node_stats>
    
        @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, "integration test")
        def test_node_stats(self):
            """
            logstash node_stats metricset test
            """
    >       self.check_metricset("logstash", "node_stats", self.get_hosts(), self.FIELDS)
    
    module/logstash/test_logstash.py:30: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    tests/system/metricbeat.py:105: in check_metricset
        self.assertCountEqual(self.de_dot(fields), evt.keys())
    E   AssertionError: Element counts were not equal:
    E   First has 1, Second has 0:  'logstash'
    E   First has 0, Second has 1:  'error' 
    

Steps errors 4

Expand to view the steps failures

generator-metricbeat-test - make -C generator/_templates/metricbeat test test-package
  • Took 5 min 10 sec . View more details on here
  • Description: make -C generator/_templates/metricbeat test test-package
generator-beat-test - make -C generator/_templates/beat test test-package
  • Took 7 min 19 sec . View more details on here
  • Description: make -C generator/_templates/beat test test-package
metricbeat-pythonIntegTest - mage pythonIntegTest
  • Took 31 min 0 sec . View more details on here
  • Description: mage pythonIntegTest
Error signal
  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 2'

Log output

Expand to view the last 100 lines of log output

[2021-04-22T10:18:54.631Z] 9.66s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_022_zscaler
[2021-04-22T10:18:54.631Z] 9.53s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_096_gcp
[2021-04-22T10:18:54.631Z] 9.51s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_118_squid
[2021-04-22T10:18:54.631Z] 9.49s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_315_cisco
[2021-04-22T10:18:54.631Z] 9.44s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_370_oracle
[2021-04-22T10:18:54.631Z] ================ 402 passed, 389 warnings in 1810.26s (0:30:10) ================
[2021-04-22T10:18:54.889Z] >> python test: Integration Testing Complete
[2021-04-22T10:18:58.745Z] Cleaning up /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae
[2021-04-22T10:18:58.745Z] Client: Docker Engine - Community
[2021-04-22T10:18:58.745Z]  Version:           20.10.3
[2021-04-22T10:18:58.745Z]  API version:       1.41
[2021-04-22T10:18:58.745Z]  Go version:        go1.13.15
[2021-04-22T10:18:58.745Z]  Git commit:        48d30b5
[2021-04-22T10:18:58.745Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-22T10:18:58.745Z]  OS/Arch:           linux/amd64
[2021-04-22T10:18:58.745Z]  Context:           default
[2021-04-22T10:18:58.745Z]  Experimental:      true
[2021-04-22T10:18:58.745Z] 
[2021-04-22T10:18:58.745Z] Server: Docker Engine - Community
[2021-04-22T10:18:58.745Z]  Engine:
[2021-04-22T10:18:58.745Z]   Version:          20.10.3
[2021-04-22T10:18:58.745Z]   API version:      1.41 (minimum version 1.12)
[2021-04-22T10:18:58.745Z]   Go version:       go1.13.15
[2021-04-22T10:18:58.745Z]   Git commit:       46229ca
[2021-04-22T10:18:58.745Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-22T10:18:58.745Z]   OS/Arch:          linux/amd64
[2021-04-22T10:18:58.745Z]   Experimental:     false
[2021-04-22T10:18:58.745Z]  containerd:
[2021-04-22T10:18:58.745Z]   Version:          1.4.4
[2021-04-22T10:18:58.745Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-22T10:18:58.745Z]  runc:
[2021-04-22T10:18:58.745Z]   Version:          1.0.0-rc93
[2021-04-22T10:18:58.745Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-22T10:18:58.745Z]  docker-init:
[2021-04-22T10:18:58.745Z]   Version:          0.19.0
[2021-04-22T10:18:58.745Z]   GitCommit:        de40ad0
[2021-04-22T10:18:58.745Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-22T10:18:58.745Z] Unable to find image 'alpine:3.4' locally
[2021-04-22T10:18:59.313Z] 3.4: Pulling from library/alpine
[2021-04-22T10:18:59.881Z] c1e54eec4b57: Pulling fs layer
[2021-04-22T10:19:00.141Z] c1e54eec4b57: Verifying Checksum
[2021-04-22T10:19:00.141Z] c1e54eec4b57: Download complete
[2021-04-22T10:19:00.400Z] c1e54eec4b57: Pull complete
[2021-04-22T10:19:00.400Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-04-22T10:19:00.400Z] Status: Downloaded newer image for alpine:3.4
[2021-04-22T10:19:02.319Z] Change permissions with write access of all files inside the specific folder
[2021-04-22T10:19:03.737Z] Running in /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae/src/github.com/elastic/beats/build
[2021-04-22T10:19:04.085Z] + rm -rf ve
[2021-04-22T10:19:04.085Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-04-22T10:19:04.524Z] + python .ci/scripts/pre_archive_test.py
[2021-04-22T10:19:07.059Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-04-22T10:19:07.112Z] Running in /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae/src/github.com/elastic/beats/build
[2021-04-22T10:19:07.177Z] Recording test results
[2021-04-22T10:19:10.090Z] [Checks API] No suitable checks publisher found.
[2021-04-22T10:19:10.664Z] + go clean -modcache
[2021-04-22T10:19:14.280Z] Cleaning up /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae
[2021-04-22T10:19:14.281Z] Client: Docker Engine - Community
[2021-04-22T10:19:14.281Z]  Version:           20.10.3
[2021-04-22T10:19:14.281Z]  API version:       1.41
[2021-04-22T10:19:14.281Z]  Go version:        go1.13.15
[2021-04-22T10:19:14.281Z]  Git commit:        48d30b5
[2021-04-22T10:19:14.281Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-04-22T10:19:14.281Z]  OS/Arch:           linux/amd64
[2021-04-22T10:19:14.281Z]  Context:           default
[2021-04-22T10:19:14.281Z]  Experimental:      true
[2021-04-22T10:19:14.281Z] 
[2021-04-22T10:19:14.281Z] Server: Docker Engine - Community
[2021-04-22T10:19:14.281Z]  Engine:
[2021-04-22T10:19:14.281Z]   Version:          20.10.3
[2021-04-22T10:19:14.281Z]   API version:      1.41 (minimum version 1.12)
[2021-04-22T10:19:14.281Z]   Go version:       go1.13.15
[2021-04-22T10:19:14.281Z]   Git commit:       46229ca
[2021-04-22T10:19:14.281Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-04-22T10:19:14.281Z]   OS/Arch:          linux/amd64
[2021-04-22T10:19:14.281Z]   Experimental:     false
[2021-04-22T10:19:14.281Z]  containerd:
[2021-04-22T10:19:14.281Z]   Version:          1.4.4
[2021-04-22T10:19:14.281Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-04-22T10:19:14.281Z]  runc:
[2021-04-22T10:19:14.281Z]   Version:          1.0.0-rc93
[2021-04-22T10:19:14.281Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-04-22T10:19:14.281Z]  docker-init:
[2021-04-22T10:19:14.281Z]   Version:          0.19.0
[2021-04-22T10:19:14.281Z]   GitCommit:        de40ad0
[2021-04-22T10:19:14.281Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-04-22T10:19:22.401Z] Change permissions with write access of all files inside the specific folder
[2021-04-22T10:19:23.019Z] Running in /var/lib/jenkins/workspace/PR-25224-1-e33f09b7-a4f0-4677-83f2-e2924a22c4ae
[2021-04-22T10:19:29.416Z] + gsutil --version
[2021-04-22T10:19:30.848Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-04-22T10:19:31.213Z] + gcloud auth activate-service-account --key-file ****
[2021-04-22T10:19:31.780Z] Activated service account credentials for: [[email protected]]
[2021-04-22T10:19:32.212Z] + gsutil -m -q cp -a public-read eC1wYWNrL2ZpbGViZWF0LWJ1aWxkM2I0MGU5OTJlYzRlYWY2MjAwM2ExN2M0NjM1MjI3YTAyZTQ2NjQ4Nw gs://beats-ci-temp/ci/cache/
[2021-04-22T10:19:38.125Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-04-22T10:19:38.269Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25224/src/github.com/elastic/beats
[2021-04-22T10:19:38.622Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25224
[2021-04-22T10:19:38.785Z] [INFO] getVaultSecret: Getting secrets
[2021-04-22T10:19:38.882Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-22T10:19:39.882Z] + chmod 755 generate-build-data.sh
[2021-04-22T10:19:39.882Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25224/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25224/runs/1 FAILURE 6325241
[2021-04-22T10:19:40.133Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25224/runs/1/steps/?limit=10000 -o steps-info.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 1
Passed 47163
Skipped 5190
Total 52354

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / metricbeat-pythonIntegTest / test_node_stats – metricbeat.module.logstash.test_logstash.Test

@kvch kvch merged commit e3ebe2c into elastic:7.x Apr 22, 2021
@zube zube bot removed the [zube]: Done label Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants