Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pex requirement parsing is tripped up by files in the CWD with the same name as requirements' project names. #1188

Closed
jsirois opened this issue Jan 22, 2021 · 1 comment · Fixed by #1190
Closed

Pex requirement parsing is tripped up by files in the CWD with the same name as requirements' project names. #1188

jsirois opened this issue Jan 22, 2021 · 1 comment · Fixed by #1190
Assignees
@jsirois
Labels
bug

Comments

@jsirois
Copy link
Member

jsirois commented Jan 22, 2021

For example:

$ touch requests && pex requests==2.0.0 -- -c 'import requests; print(requests.__file__)'
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/bin/pex.py", line 1068, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/bin/pex.py", line 946, in build_pex
    resolveds = resolve_multi(
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/resolver.py", line 1068, in resolve_multi
    direct_requirements = _parse_reqs(requirements, requirement_files, network_configuration)
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/resolver.py", line 945, in _parse_reqs
    parsed_requirements.extend(parse_requirement_strings(requirements))
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/requirements.py", line 774, in parse_requirement_strings
    yield _parse_requirement_line(
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/requirements.py", line 615, in _parse_requirement_line
    requirement = parse_requirement_from_dist(
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/requirements.py", line 300, in parse_requirement_from_dist
    project_name_and_version = dist_metadata.project_name_and_version(dist)
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/dist_metadata.py", line 200, in project_name_and_version
    return ProjectNameAndVersion.from_filename(dist)
  File "/home/jsirois/.venv/pex/lib/python3.9/site-packages/pex/dist_metadata.py", line 180, in from_filename
    raise MetadataError(
pex.dist_metadata.MetadataError: The distribution at path '/home/jsirois/dev/pantsbuild/jsirois-pex/requests' does not have a file name matching known sdist or wheel file name formats.

This only happens for files. Directories are handled just fine as is often exercised in the Pex repo where there is a top-level pex dir; yet pex pex -cpex -opex.pex works just fine.
@jsirois jsirois added the bug label Jan 22, 2021
@jsirois
Copy link
Member Author

jsirois commented Jan 22, 2021

This was spotted over in Pants trying to upgrade to Pex 2.1.25. Pants is notable here in that it uses the Pex PEX (a file named pex) to do requirement resolves. When it tries to resolve the requirement string pex==2.1.25 it hits:

$ ./pants test src/python/pants/init/load_backends_integration_test.py 
16:13:03.88 [INFO] initializing pantsd...
16:13:04.71 [INFO] pantsd initialized.
16:13:10.53 [INFO] Completed: Resolving 3rdparty/python/constraints.txt
16:13:10.53 [INFO] Canceled: Building pytest.pex with 6 requirements: ipdb, pygments, pytest-cov>=2.10.1,<2.11, pytest-icdiff, pytest>=6.0.1,<6.3, zipp==2.1.0
16:13:10.53 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 265, in run
    engine_result = self._perform_run(goals)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 188, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 210, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/init/engine_initializer.py", line 132, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.core.goals.test.enrich_test_result (src/python/pants/init/load_backends_integration_test.py:integration)
  in pants.backend.python.goals.pytest_runner.run_python_test (src/python/pants/init/load_backends_integration_test.py:integration)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/process.py", line 259, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Resolving 3rdparty/python/constraints.txt' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 477, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 394, in _wrap_coverage
    return runner(*args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 425, in _wrap_profiling
    return runner(*args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 533, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 635, in execute_entry
    return runner(entry_point)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 651, in execute_pkg_resources
    return runner()
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/bin/pex.py", line 1068, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/bin/pex.py", line 964, in build_pex
    ignore_errors=options.ignore_errors,
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/resolver.py", line 1068, in resolve_multi
    direct_requirements = _parse_reqs(requirements, requirement_files, network_configuration)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/resolver.py", line 945, in _parse_reqs
    parsed_requirements.extend(parse_requirement_strings(requirements))
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/requirements.py", line 780, in parse_requirement_strings
    end_line=1,
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/requirements.py", line 616, in _parse_requirement_line
    archive_or_project_path, extras=extras, marker=marker
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/requirements.py", line 300, in parse_requirement_from_dist
    project_name_and_version = dist_metadata.project_name_and_version(dist)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/dist_metadata.py", line 200, in project_name_and_version
    return ProjectNameAndVersion.from_filename(dist)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/dist_metadata.py", line 182, in from_filename
    "file name formats.".format(path)
pex.dist_metadata.MetadataError: The distribution at path '/tmp/process-executionqRWad1/pex' does not have a file name matching known sdist or wheel file name formats.


Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 265, in run
    engine_result = self._perform_run(goals)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 188, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 210, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/init/engine_initializer.py", line 132, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.core.goals.test.enrich_test_result (src/python/pants/init/load_backends_integration_test.py:integration)
  in pants.backend.python.goals.pytest_runner.run_python_test (src/python/pants/init/load_backends_integration_test.py:integration)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/process.py", line 259, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Resolving 3rdparty/python/constraints.txt' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 477, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 394, in _wrap_coverage
    return runner(*args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 425, in _wrap_profiling
    return runner(*args)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 533, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 635, in execute_entry
    return runner(entry_point)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.bootstrap/pex/pex.py", line 651, in execute_pkg_resources
    return runner()
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/bin/pex.py", line 1068, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/bin/pex.py", line 964, in build_pex
    ignore_errors=options.ignore_errors,
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/resolver.py", line 1068, in resolve_multi
    direct_requirements = _parse_reqs(requirements, requirement_files, network_configuration)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/resolver.py", line 945, in _parse_reqs
    parsed_requirements.extend(parse_requirement_strings(requirements))
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/requirements.py", line 780, in parse_requirement_strings
    end_line=1,
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/requirements.py", line 616, in _parse_requirement_line
    archive_or_project_path, extras=extras, marker=marker
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/requirements.py", line 300, in parse_requirement_from_dist
    project_name_and_version = dist_metadata.project_name_and_version(dist)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/dist_metadata.py", line 200, in project_name_and_version
    return ProjectNameAndVersion.from_filename(dist)
  File "/home/jsirois/.cache/pants/named_caches/pex_root/unzipped_pexes/139c4021d7418da8f4255e16de6a69aa46c7d82c/.deps/pex-2.1.25-py2.py3-none-any.whl/pex/dist_metadata.py", line 182, in from_filename
    "file name formats.".format(path)
pex.dist_metadata.MetadataError: The distribution at path '/tmp/process-executionqRWad1/pex' does not have a file name matching known sdist or wheel file name formats.

@jsirois jsirois mentioned this issue Jan 22, 2021
Closed
1 task
@jsirois jsirois self-assigned this Jan 22, 2021
jsirois added a commit to jsirois/pex that referenced this issue Jan 22, 2021
@jsirois
Requirement parsing handles local non-dist files.
49f345a 
Previously a file (not directory) in the CWD with the same name as the
project name of a requirement being resolved would lead to an error.
Modify the requirement parsing stress test to cause this error and fix
parsing so the test passes.

Fixes pex-tool#1188
@jsirois jsirois mentioned this issue Jan 22, 2021
Merged
jsirois added a commit that referenced this issue Jan 22, 2021
@jsirois
Requirement parsing handles local non-dist files. (#1190)
eec3591 
Previously a file (not directory) in the CWD with the same name as the
project name of a requirement being resolved would lead to an error.
Modify the requirement parsing stress test to cause this error and fix
parsing so the test passes.

Fixes #1188
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Requirement parsing handles local non-dist files. jsirois/pex
1 participant
@jsirois