Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CreatePex attempts to write to the default pex cache (~/.pex). #9361

Closed
jsirois opened this issue Mar 20, 2020 · 3 comments · Fixed by #9401
Closed

CreatePex attempts to write to the default pex cache (~/.pex). #9361

jsirois opened this issue Mar 20, 2020 · 3 comments · Fixed by #9401
Assignees
@jsirois
Labels
bug

Comments

@jsirois
Copy link
Contributor

jsirois commented Mar 20, 2020

Seen in a remote build execution platform that only presents a read-only filesystem except for the EPR chroot:

$ ./pants --no-v1 --v2-ui --pants-config-files=pants.local-bf.toml --remote-execution-extra-platform-properties=execution-policy=debug test src/python/toolchain/satresolver:tests
20:38:20 [WARN] Error loading access token: AuthError('Failed to load auth token (no default file or environment variable). Run `./tc auth-acquire` to set up authentication.')
20:38:20 [INFO] Starting tests: src/python/toolchain/satresolver:tests
20:38:23 [WARN] /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/1.27.0.dev0_py38/lib/python3.8/site-packages/pants/base/exception_sink.py:431: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),
ERROR: process 'Create a requirements PEX: ansicolors==1.1.8, coloredlogs, pex==2.1.6, pytest>=5.4.1, termcolor' failed with exit code 129.
stdout:
stderr:
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Enabling debug
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Called in PWD=/worker/operations/default_memory_instance/operations/0843afa8-7036-4f43-a954-791a37846457 with: network=true container-image=283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0 debug=true python ./pex --disable-cache --output-file requirements.pex --interpreter-constraint CPython>=3.8,<4 --not-zip-safe --manylinux manylinux2014 --constraints 3rdparty/python/constraints.txt --sources-directory=source_files ansicolors==1.1.8 coloredlogs pex==2.1.6 pytest>=5.4.1 termcolor
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	total 2564
drwxr-xr-x 3 worker worker    4096 Mar 20 20:38 3rdparty
-rwxr--r-- 3 worker worker 2614280 Mar 20 19:58 pex
drwxr-xr-x 2 worker worker    4096 Mar 20 20:38 source_files
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Performing runt.sh exec on image 283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0
2020-03-20T20:38:21+00:00 executor.sh DEBUG
	Enabling network
2020-03-20T20:38:21+00:00 runt.sh DEBUG
	Wrapping process with waitnet: /usr/bin/waitnet -d /slirp4netns/run python ./pex --disable-cache --output-file requirements.pex --interpreter-constraint CPython>=3.8,<4 --not-zip-safe --manylinux manylinux2014 --constraints 3rdparty/python/constraints.txt --sources-directory=source_files ansicolors==1.1.8 coloredlogs pex==2.1.6 pytest>=5.4.1 termcolor
2020-03-20T20:38:21+00:00 runt.sh DEBUG
	Executing /worker/exec/283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0/283194185447_dkr_ecr_us_east_1_amazonaws_com_ci_2020_03_18_00_32_05_4255fb8f57f0_4c3fa4a1_d153_4694_9c85_1ee4cbd5f40f/config.json as uid=1000(worker) gid=1000(worker) groups=1000(worker)
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Called as /worker/bin/start-network.sh -d true -s /usr/bin/slirp4netns /worker/exec/283194185447.dkr.ecr.us-east-1.amazonaws.com/ci:2020-03-18.00-32-05-4255fb8f57f0/283194185447_dkr_ecr_us_east_1_amazonaws_com_ci_2020_03_18_00_32_05_4255fb8f57f0_4c3fa4a1_d153_4694_9c85_1ee4cbd5f40f/slirp4netns/run
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Starting slirp4netns daemon for container process 452...
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Started slirp4netns daemon @ 465.
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Starting quit daemon for slirp4netns daemon @ 465...
2020-03-20T20:38:21+00:00 start-network.sh DEBUG
	Started quit daemon @ 473.
waitnet DEBUG
	Received 1 on /slirp4netns/run/ready
waitnet DEBUG
	Executing python...
Traceback (most recent call last):
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 393, in execute
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 325, in _wrap_coverage
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 356, in _wrap_profiling
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 438, in _execute
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 535, in execute_entry
  File "/worker/operations/pex/.bootstrap/pex/pex.py", line 550, in execute_pkg_resources
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/bin/pex.py", line 661, in main
    pex_builder = build_pex(reqs, options)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/bin/pex.py", line 578, in build_pex
    resolveds = resolve_multi(requirements=reqs,
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 802, in resolve_multi
    return list(resolve_request.resolve_distributions(ignore_errors=ignore_errors))
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 497, in resolve_distributions
    for resolve_result in self._run_parallel(inputs=to_resolve,
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 366, in _run_parallel
    for result in execute_parallel(
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/jobs.py", line 219, in execute_parallel
    raise error
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/jobs.py", line 202, in spawn_jobs
    result = spawn_func(item)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/resolver.py", line 376, in _spawn_resolve
    download_job = get_pip().spawn_download_distributions(
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/pip.py", line 256, in get_pip
    _PIP = Pip.create()
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/pip.py", line 30, in create
    isolated_pip_environment = third_party.pkg_resources.Environment(search_path=pythonpath)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1016, in __init__
    self.scan(search_path)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1047, in scan
    for item in search_path:
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/third_party/__init__.py", line 442, in expose
    for path in VendorImporter.expose(dists, root=isolated()):
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/third_party/__init__.py", line 363, in isolated
    with atomic_directory(isolated_dir) as chroot:
  File "/usr/local/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/common.py", line 303, in atomic_directory
    safe_mkdir(atomic_dir.work_dir)
  File "/worker/operations/pex_root/installed_wheels/297b14b117c1355307b3d1c38f87a9dbd9a11900/pex-2.1.6-py2.py3-none-any.whl/pex/common.py", line 185, in safe_mkdir
    os.makedirs(directory)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/toolchain/.pex'
waitnet DEBUG
	Sent quit on /slirp4netns/run/quit
(Use --print-exception-stacktrace to see more error details.)
@jsirois
Copy link
Contributor Author

jsirois commented Mar 20, 2020

It looks like newer Pex is what is broken here, #8335 via #8283 fixed this by always passing --disable-cache - which no longer appears to fully work. A workaround would be to set the PEX_ROOT like HermeticPex does.

@jsirois
Copy link
Contributor Author

jsirois commented Mar 20, 2020

OK - this is a problem unique to PEXed Pex. It, and only it, extracts itself to $PEX_ROOT/isolated, but all PEXes have PEX_ env vars stripped before running, which leaves PEXed Pex using the default cache even though Pants tries to set that to ./pex_root.

@jsirois jsirois self-assigned this Mar 21, 2020
@jsirois
Copy link
Contributor Author

jsirois commented Mar 21, 2020

This will need to be solved over in Pex. Forked off this issue which Pants will block on via a Pex 2.1.7 release: pex-tool/pex#926

jsirois added a commit to jsirois/pants that referenced this issue Mar 25, 2020
@jsirois
Upgrade to Pex 2.1.7.
029c7ec 
This brings in fixes that support our use of `--disable-cache` with the
Pex PEX binary that had regressed in the Pex 2.x series.

Also leverage the new `--no-strip-pex-env` option when building the
pants PEX we deploy.

Further improvements including better error messages in some cases are
detailed in the release notes:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.7

Fixes pantsbuild#9361

[ci skip-jvm-tests]  # No JVM changes made.
@jsirois jsirois mentioned this issue Mar 25, 2020
Merged
jsirois added a commit that referenced this issue Mar 26, 2020
@jsirois
Upgrade to Pex 2.1.7. (#9401)
b5c7ca8 
This brings in fixes that support our use of `--disable-cache` with the
Pex PEX binary that had regressed in the Pex 2.x series.

Also leverage the new `--no-strip-pex-env` option when building the
pants PEX we deploy.

Further improvements including better error messages in some cases are
detailed in the release notes:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.7

Fixes #9361
stuhood pushed a commit that referenced this issue Apr 7, 2020
@jsirois @stuhood
Upgrade to Pex 2.1.7. (#9401)
0697d3a 
This brings in fixes that support our use of `--disable-cache` with the
Pex PEX binary that had regressed in the Pex 2.x series.

Also leverage the new `--no-strip-pex-env` option when building the
pants PEX we deploy.

Further improvements including better error messages in some cases are
detailed in the release notes:
  https://github.com/pantsbuild/pex/releases/tag/v2.1.7

Fixes #9361
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upgrade to Pex 2.1.7. jsirois/pants
1 participant
@jsirois