Pre-build docker images used for filebeat integration testing #18969

jsoriano · 2020-06-04T11:15:52Z

What does this PR do?

Pre-build docker images used for filebeat integration testing.

Why is it important?

To reduce flakiness of tests because of having to build images.

Related issues

Tag googlepubsub image used in filebeat tests Tag googlepubsub docker image #18773
Move this job to beats-ci Migrate job to build test docker images to beats repo #18785
Enable JJBB for this job ci: enable JJBB #18812

elasticmachine · 2020-06-04T12:34:43Z

💔 Tests Failed

Expand to view the summary

Build stats

Build Cause: [Pull request #18969 updated]
Start Time: 2020-06-04T16:14:41.288+0000
Duration: 79 min 37 sec

Test stats 🧪

Test	Results
Failed	1
Passed	9316
Skipped	1574
Total	10891

Test errors

Expand to view the tests failures

Name: Build and Test / Filebeat Windows / test_tail_files – test_crawler.Test
- Age: 1
- Duration: 16.239
- Error Details: Timeout waiting for 'cond' to be true. Waited 15 seconds.

Steps errors

Expand to view the steps failures

Name: General Build Step
- Description: [2020-06-04T16:36:49.132Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs:
- Duration: 0 min 5 sec
- Start Time: 2020-06-04T16:36:49.086+0000
- log
Name: Mage build unitTest
- Description: mage build unitTest
- Duration: 11 min 44 sec
- Start Time: 2020-06-04T16:41:49.318+0000
- log
Name: Install Go 1.13.10
- Description: .ci/scripts/install-go.sh
- Duration: 2 min 37 sec
- Start Time: 2020-06-04T16:37:04.318+0000
- log
Name: Install Go 1.13.10
- Description: .ci/scripts/install-go.sh
- Duration: 2 min 22 sec
- Start Time: 2020-06-04T16:37:05.147+0000
- log
Name: Install Go 1.13.10
- Description: .ci/scripts/install-go.sh
- Duration: 2 min 22 sec
- Start Time: 2020-06-04T16:37:09.145+0000
- log
Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 2 min 22 sec
- Start Time: 2020-06-04T16:56:04.722+0000
- log

Log output

Expand to view the last 100 lines of log output

[2020-06-04T17:32:50.031Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-06-04T17:32:50.031Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-06-04T17:32:50.031Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-06-04T17:32:50.031Z] + FILE=journalbeat/build/coverage/full.cov
[2020-06-04T17:32:50.031Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-06-04T17:32:50.756Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats
[2020-06-04T17:32:51.069Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-06-04T17:32:51.082Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Lint
[2020-06-04T17:32:51.167Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Winlogbeat-oss
[2020-06-04T17:32:51.241Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-06-04T17:32:51.327Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Dockerlogbeat
[2020-06-04T17:32:51.416Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-06-04T17:32:51.498Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-06-04T17:32:51.582Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Journalbeat-oss
[2020-06-04T17:32:51.675Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-06-04T17:32:51.761Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-06-04T17:32:51.846Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-06-04T17:32:51.932Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-06-04T17:32:52.017Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-06-04T17:32:52.105Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-06-04T17:32:52.189Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-06-04T17:32:52.278Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-06-04T17:32:52.369Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Heartbeat-oss
[2020-06-04T17:32:52.461Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-06-04T17:32:52.553Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-06-04T17:32:52.641Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-06-04T17:32:52.718Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-06-04T17:32:52.798Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Filebeat-Windows
[2020-06-04T17:32:52.875Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Libbeat-x-pack
[2020-06-04T17:32:52.962Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-06-04T17:32:53.042Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-06-04T17:32:53.128Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-06-04T17:32:53.218Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Packetbeat-oss
[2020-06-04T17:32:53.319Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-06-04T17:32:53.420Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-Windows
[2020-06-04T17:32:53.511Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-06-04T17:32:53.604Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-06-04T17:32:53.694Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-06-04T17:32:53.783Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-06-04T17:32:53.863Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Filebeat-x-pack
[2020-06-04T17:32:53.948Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-06-04T17:32:54.027Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Filebeat-oss
[2020-06-04T17:32:54.104Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Heartbeat-Windows
[2020-06-04T17:32:54.180Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-06-04T17:32:54.266Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-06-04T17:32:54.340Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Libbeat-oss
[2020-06-04T17:32:54.417Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Functionbeat-Windows
[2020-06-04T17:32:54.501Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-06-04T17:32:54.574Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-06-04T17:32:54.645Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-06-04T17:32:54.719Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-06-04T17:32:54.793Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-06-04T17:32:55.157Z] + cat
[2020-06-04T17:32:55.157Z] + /usr/local/bin/runbld ./runbld-script
[2020-06-04T17:32:55.157Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-06-04T17:33:01.816Z] runbld>>> runbld started
[2020-06-04T17:33:01.816Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-06-04T17:33:03.201Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-18969' in order of occurrence in the config (last value wins).
[2020-06-04T17:33:04.591Z] runbld>>> Debug logging enabled.
[2020-06-04T17:33:04.591Z] runbld>>> Storing result
[2020-06-04T17:33:04.852Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-06-04T17:33:04.852Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200604173304-C567770E
[2020-06-04T17:33:04.852Z] runbld>>> Adding system facts.
[2020-06-04T17:33:05.797Z] runbld>>> Adding vcs info for the latest commit:  d9885fb326ddcf44569a1d17dea4482b63b3a998
[2020-06-04T17:33:05.797Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-06-04T17:33:05.797Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-06-04T17:33:05.797Z] Processing JUnit reports with runbld...
[2020-06-04T17:33:05.797Z] + echo 'Processing JUnit reports with runbld...'
[2020-06-04T17:33:06.373Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-06-04T17:33:06.373Z] runbld>>> DURATION: 14ms
[2020-06-04T17:33:06.373Z] runbld>>> STDOUT: 40 bytes
[2020-06-04T17:33:06.373Z] runbld>>> STDERR: 49 bytes
[2020-06-04T17:33:06.373Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-06-04T17:33:06.373Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats
[2020-06-04T17:33:07.756Z] runbld>>> Storing build metadata: 
[2020-06-04T17:33:07.756Z] runbld>>> Adding test report.
[2020-06-04T17:33:07.757Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats
[2020-06-04T17:33:08.704Z] runbld>>> Found 115 test output files
[2020-06-04T17:33:08.966Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-06-04T17:33:08.966Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-06-04T17:33:09.228Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-06-04T17:33:09.228Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-06-04T17:33:09.228Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-06-04T17:33:09.228Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-06-04T17:33:09.228Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-06-04T17:33:09.228Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-06-04T17:33:11.144Z] runbld>>> Test output logs contained: Errors: 1 Failures: 0 Tests: 10741 Skipped: 1335
[2020-06-04T17:33:11.144Z] runbld>>> Storing result
[2020-06-04T17:33:11.144Z] runbld>>> FAILURES: 1
[2020-06-04T17:33:11.716Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-06-04T17:33:11.716Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200604173304-C567770E
[2020-06-04T17:33:11.716Z] runbld>>> Email notification disabled by environment variable.
[2020-06-04T17:33:11.716Z] runbld>>> Slack notification disabled by environment variable.
[2020-06-04T17:33:17.237Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18969
[2020-06-04T17:33:17.503Z] [INFO] getVaultSecret: Getting secrets
[2020-06-04T17:33:17.560Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-04T17:33:18.306Z] + chmod 755 generate-build-data.sh
[2020-06-04T17:33:18.306Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18969/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18969/runs/3 FAILURE 4716758
[2020-06-04T17:33:18.306Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18969/runs/3/steps/?limit=10000 -o steps-info.json
[2020-06-04T17:33:20.789Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18969/runs/3/tests/?status=FAILED -o tests-errors.json

jsoriano · 2020-06-04T15:29:30Z

Testing the new pipeline locally I have seen that the script cannot be found, I wonder why this doesn't happen in beats-ci/apm-ci.
I have added a change to explicitly change to the base directory.

jsoriano · 2020-06-09T13:01:46Z

@kuisathaverat could you take another look after my latest changes? Thanks!

…c#18969)

Pre-build docker images used for integration testing

b4a8454

jsoriano added :Testing Team:Automation Label for the Observability productivity team labels Jun 4, 2020

jsoriano requested a review from kuisathaverat June 4, 2020 11:15

jsoriano self-assigned this Jun 4, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 4, 2020

kuisathaverat approved these changes Jun 4, 2020

View reviewed changes

Explicitly change directory

b2a58d1

Use full path for the directory where the script is executed

2ffbaf4

jsoriano requested a review from kuisathaverat June 4, 2020 16:15

kuisathaverat approved these changes Jun 9, 2020

View reviewed changes

jsoriano merged commit 247a00b into elastic:master Jun 9, 2020

jsoriano deleted the add-filebeat-docker-builds branch June 9, 2020 14:53

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020

Pre-build docker images used for filebeat integration testing (elasti…

0f85568

…c#18969)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pre-build docker images used for filebeat integration testing #18969

Pre-build docker images used for filebeat integration testing #18969

jsoriano commented Jun 4, 2020 •

edited

Loading

elasticmachine commented Jun 4, 2020 •

edited

Loading

Build stats

Test stats 🧪

jsoriano commented Jun 4, 2020

jsoriano commented Jun 9, 2020

Pre-build docker images used for filebeat integration testing #18969

Pre-build docker images used for filebeat integration testing #18969

Conversation

jsoriano commented Jun 4, 2020 • edited Loading

What does this PR do?

Why is it important?

Related issues

elasticmachine commented Jun 4, 2020 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Steps errors

Log output

jsoriano commented Jun 4, 2020

jsoriano commented Jun 9, 2020

jsoriano commented Jun 4, 2020 •

edited

Loading

elasticmachine commented Jun 4, 2020 •

edited

Loading