[build][functionbeat][packetbeat][darwin][7.9] mage build unitTest failed

[v1v]

Command

$ cd functionbeat
$ mage build unitTest

Stacktrace

Expand to view

[2020-10-21T11:00:48.522Z] + mage build unitTest
[2020-10-21T11:00:57.215Z] >> build: Building functionbeat
[2020-10-21T11:01:40.697Z] >> build: Building functionbeat-aws
[2020-10-21T11:02:16.656Z] >> go test: Unit Testing
[2020-10-21T11:03:43.026Z] SUMMARY:
[2020-10-21T11:03:43.026Z]   Fail:     0
[2020-10-21T11:03:43.026Z]   Skip:     1
[2020-10-21T11:03:43.026Z]   Pass:     142
[2020-10-21T11:03:43.026Z]   Packages: 13
[2020-10-21T11:03:43.026Z]   Duration: 1m13.156194714s
[2020-10-21T11:03:43.026Z]   Coverage Report: /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/x-pack/functionbeat/build/TEST-go-unit.html
[2020-10-21T11:03:43.026Z]   JUnit Report:    /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-10-21T11:03:43.026Z]   Output File:     /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/x-pack/functionbeat/build/TEST-go-unit.out
[2020-10-21T11:03:43.026Z] >> go test: Unit Test Passed
[2020-10-21T11:04:00.966Z] >> python test: Unit Testing
[2020-10-21T11:04:13.553Z]     ERROR: Command errored out with exit status 1:
[2020-10-21T11:04:13.553Z]      command: /private/var/lib/jenkins/workspace/Beats_beats_7.9/python-env/build/ve/darwin/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-qmtq0wh1/markupsafe/setup.py'"'"'; __file__='"'"'/private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-qmtq0wh1/markupsafe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-pip-egg-info-aujphqj1
[2020-10-21T11:04:13.553Z]          cwd: /private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-qmtq0wh1/markupsafe/
[2020-10-21T11:04:13.553Z]     Complete output (5 lines):
[2020-10-21T11:04:13.553Z]     Traceback (most recent call last):
[2020-10-21T11:04:13.553Z]       File "<string>", line 1, in <module>
[2020-10-21T11:04:13.553Z]       File "/private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-qmtq0wh1/markupsafe/setup.py", line 6, in <module>
[2020-10-21T11:04:13.553Z]         from setuptools import setup, Extension, Feature
[2020-10-21T11:04:13.553Z]     ImportError: cannot import name 'Feature' from 'setuptools' (/private/var/lib/jenkins/workspace/Beats_beats_7.9/python-env/build/ve/darwin/lib/python3.8/site-packages/setuptools/__init__.py)
[2020-10-21T11:04:13.553Z]     ----------------------------------------
[2020-10-21T11:04:13.553Z] ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
[2020-10-21T11:04:13.553Z] Error: running "/var/lib/jenkins/workspace/Beats_beats_7.9/python-env/build/ve/darwin/bin/pip install --quiet -Ur /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/libbeat/tests/system/requirements.txt" failed with exit code 1
script returned exit code 1

[2020-10-21T10:56:22.660Z] + mage build unitTest
[2020-10-21T10:56:29.905Z] >> build: Building packetbeat
[2020-10-21T10:57:00.421Z] >> go test: Unit Testing
[2020-10-21T10:58:01.506Z] SUMMARY:
[2020-10-21T10:58:01.506Z]   Fail:     0
[2020-10-21T10:58:01.506Z]   Skip:     0
[2020-10-21T10:58:01.507Z]   Pass:     348
[2020-10-21T10:58:01.507Z]   Packages: 23
[2020-10-21T10:58:01.507Z]   Duration: 1m2.13963747s
[2020-10-21T10:58:01.507Z]   Coverage Report: /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/packetbeat/build/TEST-go-unit.html
[2020-10-21T10:58:01.507Z]   JUnit Report:    /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/packetbeat/build/TEST-go-unit.xml
[2020-10-21T10:58:01.507Z]   Output File:     /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/packetbeat/build/TEST-go-unit.out
[2020-10-21T10:58:01.507Z] >> go test: Unit Test Passed
[2020-10-21T10:58:01.993Z] Generated fields.yml for packetbeat to /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/packetbeat/fields.yml
[2020-10-21T10:58:27.548Z] >> python test: Unit Testing
[2020-10-21T10:58:40.531Z]     ERROR: Command errored out with exit status 1:
[2020-10-21T10:58:40.531Z]      command: /private/var/lib/jenkins/workspace/Beats_beats_7.9/python-env/build/ve/darwin/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-5pgixwsi/markupsafe/setup.py'"'"'; __file__='"'"'/private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-5pgixwsi/markupsafe/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-pip-egg-info-62tt63tl
[2020-10-21T10:58:40.531Z]          cwd: /private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-5pgixwsi/markupsafe/
[2020-10-21T10:58:40.531Z]     Complete output (5 lines):
[2020-10-21T10:58:40.531Z]     Traceback (most recent call last):
[2020-10-21T10:58:40.531Z]       File "<string>", line 1, in <module>
[2020-10-21T10:58:40.531Z]       File "/private/var/folders/30/v8fqcvss33z_vy3cqhxrz8c80000md/T/pip-install-5pgixwsi/markupsafe/setup.py", line 6, in <module>
[2020-10-21T10:58:40.531Z]         from setuptools import setup, Extension, Feature
[2020-10-21T10:58:40.531Z]     ImportError: cannot import name 'Feature' from 'setuptools' (/private/var/lib/jenkins/workspace/Beats_beats_7.9/python-env/build/ve/darwin/lib/python3.8/site-packages/setuptools/__init__.py)
[2020-10-21T10:58:40.531Z]     ----------------------------------------
[2020-10-21T10:58:40.531Z] ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
[2020-10-21T10:58:40.531Z] Error: running "/var/lib/jenkins/workspace/Beats_beats_7.9/python-env/build/ve/darwin/bin/pip install --quiet -Ur /private/var/lib/jenkins/workspace/Beats_beats_7.9/src/github.com/elastic/beats/libbeat/tests/system/requirements.txt" failed with exit code 1
script returned exit code 1

Worker -> worker-c07mq1u7dy3h

Question

• Is there any specific mage goal/task to prepare the ve context before running them? If so, we could add a retry step for the ve prepare context to avoid this kind of issue as much as possible, since so far if it happens then the builds get broken

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[v1v]

This issue seems to be happening in different workers quite often only for the 7.9 branch

[v1v]

For instance:

• https://beats-ci.elastic.co/job/Beats/job/beats/job/7.9/245/execution/node/2555/log/ used https://beats-ci.elastic.co/computer/worker-c07mq25jdy3h
• https://beats-ci.elastic.co/job/Beats/job/beats/job/7.9/250/execution/node/1804/log/ used https://beats-ci.elastic.co/computer/worker-c07mq1u7dy3h/
• https://beats-ci.elastic.co/job/Beats/job/beats/job/7.9/250/execution/node/2954/log/ used https://beats-ci.elastic.co/computer/worker-c07mq1u7dy3h/

[fearful-symmetry]

It looks like this is related to this: pypa/setuptools#2017

We either need to pin an older version of setup tools or update our script. I've never touched these, so not sure what the best route is here.

[urso]

Is this still an issue with 7.11?

[jlind23]

Backlog grooming: Closing for now.