Using relative path fails for commands with --project argument

[gmichaeljaison]

• I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

When pdm add command is run outside the project directory using --project argument, it fails to recognize path relative to current directory

❯ pdm add --project packages/greet packages/translate-gmj -v
Adding packages to default dependencies: translate-gmj @ file:///${PROJECT_ROOT}/packages/translate-gmj
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pytest
pdm.termui:   translate-gmj @ file:///${PROJECT_ROOT}/packages/translate-gmj
pdm.termui:   pytest
pdm.termui:   pytest-cov
pdm.termui:   black
pdm.termui:   pylint
pdm.termui:   pytest
pdm.termui:   pytest-cov
pdm.termui:   python>=3.8
pdm.termui:   Adding requirement pytest
pdm.termui:   Adding requirement translate-gmj @ file:///${PROJECT_ROOT}/packages/translate-gmj
Traceback (most recent call last):
  File "/Users/michaeljaison/.local/bin/pdm", line 10, in <module>
    sys.exit(main())
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 256, in main
    return Core().main(args)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 189, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 184, in main
    f(options.project, options)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/cli/commands/add.py", line 58, in handle
    actions.do_add(
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/cli/actions.py", line 276, in do_add
    resolved = do_lock(
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/cli/actions.py", line 98, in do_lock
    mapping, dependencies = resolve(
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/resolver/core.py", line 31, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/resolvelib/resolvers.py", line 348, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/resolvelib/structs.py", line 126, in __bool__
    next(self._factory())
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/resolver/providers.py", line 252, in matches_gen
    yield from super_find()
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/resolver/providers.py", line 146, in matches_gen
    candidates = self._find_candidates(reqs[0])
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/resolver/providers.py", line 125, in _find_candidates
    can.prepare(self.repository.environment).metadata
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 461, in metadata
    result = self.prepare_metadata()
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 405, in prepare_metadata
    self.obtain(allow_all=True)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 395, in obtain
    result = finder.download_and_unpack(
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/unearth/finder.py", line 361, in download_and_unpack
    file = unpack_link(
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/unearth/preparer.py", line 294, in unpack_link
    validator.validate_path(artifact)
  File "/Users/michaeljaison/Library/Application Support/pdm/venv/lib/python3.10/site-packages/unearth/preparer.py", line 115, in validate_path
    with path.open("rb") as f:
  File "/Users/michaeljaison/.pyenv/versions/3.10.2/lib/python3.10/pathlib.py", line 1117, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/Users/michaeljaison/Sync/python-repo-skeleton/packages/greet/packages/translate-gmj'

Obviously, the path '/Users/michaeljaison/Sync/python-repo-skeleton/packages/greet/packages/translate-gmj' does not exist, because it is relative to '/Users/michaeljaison/Sync/python-repo-skeleton/' directory.

But the same command also fails if the path is specified relative to the actual project location.

❯ pdm add --project packages/greet ../translate-gmj -v
[RequirementError]: The local path ../translate-gmj does not exist.

Actual behavior

Not able to use --project argument to use pdm outside of project directory

Expected behavior

pdm should convert relative paths relative to current directory.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
❯ pdm info -p packages/greet
PDM version:
  2.0.0b2
Python Interpreter:
  /Users/michaeljaison/.pyenv/versions/py-skeleton-pdm/bin/python3 (3.10)
Project Root:
  /Users/michaeljaison/Sync/python-repo-skeleton/packages/greet
Project Packages:
  None

❯ pdm info --env -p packages/greet
{
  "implementation_name": "cpython",
  "implementation_version": "3.10.2",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "21.5.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 21.5.0: Tue Mar 15 01:02:30 PDT 2022; 
root:xnu-8020.120.35.0.1~8/DEVELOPMENT_ARM64_T6000",
  "python_full_version": "3.10.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "darwin"
}

[frostming]

We don't allow adding packages beyond the project root, because it will break the reproducibility(You can't make it work on another machine where the path is changed), and it also makes the package not distributable.

I notice you are implementing a monorepo, you may be interested in Monas

[JohnHardy]

@frostming Is there any chance you could reconsider the point about referencing projects beyond ${PROJECT_ROOT}?

It seems to me like resolving the dep to an absolute URL doesn't protect reproducibility against unintentional mistakes any more than allowing a relative URL?

For instance, a dependency at the absolute URL file:///C:/Users/John/Desktop/myproject/components/mylib/python works fine, but breaks the workflow for the rest of the team. However, if the team checks out into the C:/ drive then PDM can be used to find the same dependency at: file:///C:/myproject/components/mylib/python.

To me this feels like an artificial limit of PDM to certain project structures, since reproducibility can be broken with the absolute URL too. Unless I am missing something obvious? :)

Perhaps PDM could allow the relative URL, but generate a warning so that the user knows they are doing something potentially harmful?

p.s. I saw your link to https://github.com/frostming/monas it looks nice but it's too much of a change for us at the moment since we are not using a monorepo.

p.p.s I am a HUGE fan of PDM! 🚀 very nice work!

[carl-ellis]

There is a valid edge case here, where you have a wider constellation of packages defined by git submodules, which then, for development purposes, need to be side loaded in as editable packages.

This is something the package manager should allow, with a warning if need be proclaiming that if the directory out of the project root is unmanaged, then this will affect reproducibility.