Cherry-pick to 7.11: CI: add resilience when ephemeral windows workers are reused (#24316) #24326

v1v · 2021-03-03T18:37:49Z

Backports the following commits to 7.11:

CI: add resilience when ephemeral windows workers are reused (CI: add resilience when ephemeral windows workers are reused #24316)

…#24316) # Conflicts: # Jenkinsfile

botelastic · 2021-03-03T18:37:55Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-03-03T18:51:37Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: v1v commented: /test
Start Time: 2021-03-04T08:42:05.239+0000
Duration: 50 min 53 sec
Commit: 260e151

Test stats 🧪

Test	Results
Failed	1
Passed	48559
Skipped	5311
Total	53871

Trends 🧪

Test errors

Expand to view the tests failures

`Build&Test / filebeat-macos-macosx / test_shutdown – filebeat.tests.system.test_shutdown.Test`

Expand to view the error details

 AssertionError: Expected exit code to be 0, but it was -15

Expand to view the stacktrace

 self = <test_shutdown.Test testMethod=test_shutdown>

    @unittest.skipIf(platform.platform().startswith("Windows-7"),
                     "Flaky test: https://github.com/elastic/beats/issues/22795")
    def test_shutdown(self):
        """
        Test starting and stopping Filebeat under load.
        """
    
        self.nasa_logs()
    
        self.render_config_template(
            path=os.path.abspath(self.working_dir) + "/log/*",
            ignore_older="1h"
        )
        for i in range(1, 5):
            proc = self.start_beat(logging_args=["-e", "-v"])
            time.sleep(.5)
>           proc.check_kill_and_wait()

tests/system/test_shutdown.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../libbeat/tests/system/beat/beat.py:105: in check_kill_and_wait
    return self.check_wait(exit_code=exit_code)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <beat.beat.Proc object at 0x103e1a710>, exit_code = 0

    def check_wait(self, exit_code=0):
        actual_exit_code = self.wait()
        assert actual_exit_code == exit_code, "Expected exit code to be %d, but it was %d" % (
>           exit_code, actual_exit_code)
E       AssertionError: Expected exit code to be 0, but it was -15

../libbeat/tests/system/beat/beat.py:94: AssertionError

Steps errors

Expand to view the steps failures

`auditbeat-windows-10-windows-10 - Install Go/Mage/Python 1.14.15`

Took 3 min 59 sec . View more details on here
Description: .ci/scripts/install-tools.bat

`filebeat-macos-macosx - mage build unitTest`

Took 10 min 47 sec . View more details on here
Description: mage build unitTest

`Prepare test output`

Took 0 min 5 sec . View more details on here
Description: python .ci/scripts/pre_archive_test.py

`Prepare test output`

Took 0 min 5 sec . View more details on here
Description: python .ci/scripts/pre_archive_test.py

`Prepare test output`

Took 0 min 2 sec . View more details on here
Description: python .ci/scripts/pre_archive_test.py

`Prepare test output`

Took 0 min 6 sec . View more details on here
Description: python .ci/scripts/pre_archive_test.py

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-03-04T09:31:29.219Z] 16.77s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_150_snort
[2021-03-04T09:31:29.219Z] 16.15s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_046_o365
[2021-03-04T09:31:29.219Z] 11.61s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_055_panw
[2021-03-04T09:31:29.219Z] 11.50s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_001_ibmmq
[2021-03-04T09:31:29.219Z] 11.43s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_053_panw
[2021-03-04T09:31:29.219Z] 10.75s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_057_panw
[2021-03-04T09:31:29.219Z] 10.55s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_152_cisco
[2021-03-04T09:31:29.219Z] 10.46s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_170_cisco
[2021-03-04T09:31:29.219Z] 10.28s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_006_gcp
[2021-03-04T09:31:29.219Z] 10.21s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_059_f5
[2021-03-04T09:31:29.219Z] 10.20s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_095_oracle
[2021-03-04T09:31:29.219Z] 10.18s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_061_microsoft
[2021-03-04T09:31:29.220Z] 9.99s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_098_oracle
[2021-03-04T09:31:29.220Z] 9.87s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_288_fortinet
[2021-03-04T09:31:29.220Z] 9.87s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_051_imperva
[2021-03-04T09:31:29.220Z] 9.87s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_052_bluecoat
[2021-03-04T09:31:29.220Z] 9.80s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_007_cyberark
[2021-03-04T09:31:29.220Z] ================ 306 passed, 293 warnings in 1472.27s (0:24:32) ================
[2021-03-04T09:31:29.220Z] >> python test: Integration Testing Complete
[2021-03-04T09:31:31.691Z] Client: Docker Engine - Community
[2021-03-04T09:31:31.691Z]  Version:           20.10.3
[2021-03-04T09:31:31.691Z]  API version:       1.41
[2021-03-04T09:31:31.691Z]  Go version:        go1.13.15
[2021-03-04T09:31:31.691Z]  Git commit:        48d30b5
[2021-03-04T09:31:31.691Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-04T09:31:31.691Z]  OS/Arch:           linux/amd64
[2021-03-04T09:31:31.691Z]  Context:           default
[2021-03-04T09:31:31.691Z]  Experimental:      true
[2021-03-04T09:31:31.691Z] 
[2021-03-04T09:31:31.691Z] Server: Docker Engine - Community
[2021-03-04T09:31:31.691Z]  Engine:
[2021-03-04T09:31:31.691Z]   Version:          20.10.3
[2021-03-04T09:31:31.691Z]   API version:      1.41 (minimum version 1.12)
[2021-03-04T09:31:31.691Z]   Go version:       go1.13.15
[2021-03-04T09:31:31.691Z]   Git commit:       46229ca
[2021-03-04T09:31:31.691Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-04T09:31:31.691Z]   OS/Arch:          linux/amd64
[2021-03-04T09:31:31.691Z]   Experimental:     false
[2021-03-04T09:31:31.691Z]  containerd:
[2021-03-04T09:31:31.691Z]   Version:          1.4.3
[2021-03-04T09:31:31.691Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-03-04T09:31:31.691Z]  runc:
[2021-03-04T09:31:31.691Z]   Version:          1.0.0-rc92
[2021-03-04T09:31:31.691Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-03-04T09:31:31.691Z]  docker-init:
[2021-03-04T09:31:31.691Z]   Version:          0.19.0
[2021-03-04T09:31:31.691Z]   GitCommit:        de40ad0
[2021-03-04T09:31:31.691Z] Unable to find image 'alpine:3.4' locally
[2021-03-04T09:31:32.258Z] 3.4: Pulling from library/alpine
[2021-03-04T09:31:32.518Z] c1e54eec4b57: Pulling fs layer
[2021-03-04T09:31:32.796Z] c1e54eec4b57: Download complete
[2021-03-04T09:31:33.055Z] c1e54eec4b57: Pull complete
[2021-03-04T09:31:33.056Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-03-04T09:31:33.056Z] Status: Downloaded newer image for alpine:3.4
[2021-03-04T09:31:35.319Z] + python .ci/scripts/pre_archive_test.py
[2021-03-04T09:31:37.243Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-03-04T09:31:37.263Z] Running in /var/lib/jenkins/workspace/PR-24326-2-cd6e9007-c64f-4916-9a54-b3cde6696820/src/github.com/elastic/beats/build
[2021-03-04T09:31:37.689Z] + rm -rf ve
[2021-03-04T09:31:37.689Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-03-04T09:31:37.711Z] Recording test results
[2021-03-04T09:31:41.907Z] [Checks API] No suitable checks publisher found.
[2021-03-04T09:31:42.272Z] + go clean -modcache
[2021-03-04T09:31:45.539Z] Client: Docker Engine - Community
[2021-03-04T09:31:45.539Z]  Version:           20.10.3
[2021-03-04T09:31:45.539Z]  API version:       1.41
[2021-03-04T09:31:45.539Z]  Go version:        go1.13.15
[2021-03-04T09:31:45.539Z]  Git commit:        48d30b5
[2021-03-04T09:31:45.539Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-04T09:31:45.539Z]  OS/Arch:           linux/amd64
[2021-03-04T09:31:45.539Z]  Context:           default
[2021-03-04T09:31:45.539Z]  Experimental:      true
[2021-03-04T09:31:45.539Z] 
[2021-03-04T09:31:45.539Z] Server: Docker Engine - Community
[2021-03-04T09:31:45.539Z]  Engine:
[2021-03-04T09:31:45.539Z]   Version:          20.10.3
[2021-03-04T09:31:45.539Z]   API version:      1.41 (minimum version 1.12)
[2021-03-04T09:31:45.539Z]   Go version:       go1.13.15
[2021-03-04T09:31:45.539Z]   Git commit:       46229ca
[2021-03-04T09:31:45.539Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-04T09:31:45.539Z]   OS/Arch:          linux/amd64
[2021-03-04T09:31:45.539Z]   Experimental:     false
[2021-03-04T09:31:45.539Z]  containerd:
[2021-03-04T09:31:45.539Z]   Version:          1.4.3
[2021-03-04T09:31:45.539Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-03-04T09:31:45.539Z]  runc:
[2021-03-04T09:31:45.539Z]   Version:          1.0.0-rc92
[2021-03-04T09:31:45.539Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-03-04T09:31:45.539Z]  docker-init:
[2021-03-04T09:31:45.539Z]   Version:          0.19.0
[2021-03-04T09:31:45.539Z]   GitCommit:        de40ad0
[2021-03-04T09:31:52.138Z] Running in /var/lib/jenkins/workspace/PR-24326-2-cd6e9007-c64f-4916-9a54-b3cde6696820
[2021-03-04T09:31:57.154Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-04T09:31:57.503Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24326/src/github.com/elastic/beats
[2021-03-04T09:31:57.865Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-24326
[2021-03-04T09:31:57.990Z] [INFO] getVaultSecret: Getting secrets
[2021-03-04T09:31:58.140Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-04T09:31:58.936Z] + chmod 755 generate-build-data.sh
[2021-03-04T09:31:58.936Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24326/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24326/runs/2 FAILURE 2993434
[2021-03-04T09:31:59.187Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24326/runs/2/steps/?limit=10000 -o steps-info.json
[2021-03-04T09:32:04.446Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24326/runs/2/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	48559
Skipped	5311
Total	53871

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / filebeat-macos-macosx / test_shutdown – filebeat.tests.system.test_shutdown.Test

v1v · 2021-03-04T08:41:43Z

/test

CI: add resilience when ephemeral windows workers are reused (elastic…

260e151

…#24316) # Conflicts: # Jenkinsfile

v1v self-assigned this Mar 3, 2021

v1v added automation backport labels Mar 3, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2021

v1v added ci skip-ci Skip the build in the CI but linting Team:Automation Label for the Observability productivity team labels Mar 3, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 3, 2021

v1v removed the skip-ci Skip the build in the CI but linting label Mar 4, 2021

v1v requested a review from a team March 4, 2021 08:41

v1v added the macOS Enable builds in the CI for darwin testing label Mar 4, 2021

kuisathaverat approved these changes Mar 4, 2021

View reviewed changes

v1v merged commit 1d9cced into elastic:7.11 Mar 4, 2021

v1v deleted the backport/7.11/pr-24316 branch March 4, 2021 12:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick to 7.11: CI: add resilience when ephemeral windows workers are reused (#24316) #24326

Cherry-pick to 7.11: CI: add resilience when ephemeral windows workers are reused (#24316) #24326

v1v commented Mar 3, 2021

botelastic bot commented Mar 3, 2021

elasticmachine commented Mar 3, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

`Build&Test / filebeat-macos-macosx / test_shutdown – filebeat.tests.system.test_shutdown.Test`

`auditbeat-windows-10-windows-10 - Install Go/Mage/Python 1.14.15`

`filebeat-macos-macosx - mage build unitTest`

`Prepare test output`

`Prepare test output`

`Prepare test output`

`Prepare test output`

`Error signal`

Test stats 🧪

Genuine test errors

v1v commented Mar 4, 2021

Cherry-pick to 7.11: CI: add resilience when ephemeral windows workers are reused (#24316) #24326

Cherry-pick to 7.11: CI: add resilience when ephemeral windows workers are reused (#24316) #24326

Conversation

v1v commented Mar 3, 2021

botelastic bot commented Mar 3, 2021

elasticmachine commented Mar 3, 2021 • edited by jenkins-beats-ci bot Loading

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build&Test / filebeat-macos-macosx / test_shutdown – filebeat.tests.system.test_shutdown.Test

Steps errors

auditbeat-windows-10-windows-10 - Install Go/Mage/Python 1.14.15

filebeat-macos-macosx - mage build unitTest

Prepare test output

Prepare test output

Prepare test output

Prepare test output

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

v1v commented Mar 4, 2021

elasticmachine commented Mar 3, 2021 •

edited by jenkins-beats-ci bot

Loading

`Build&Test / filebeat-macos-macosx / test_shutdown – filebeat.tests.system.test_shutdown.Test`

`auditbeat-windows-10-windows-10 - Install Go/Mage/Python 1.14.15`

`filebeat-macos-macosx - mage build unitTest`

`Prepare test output`

`Prepare test output`

`Prepare test output`

`Prepare test output`

`Error signal`