Drop #egg fragment on pdm export if supplied a --no-egg-fragments

[yyolk]

Is your feature/enhancement proposal related to a problem? Please describe.

This is my first feature/enhancement proposal. This relates to a problem reconciling #egg fragments not being supported with uv. astral-sh/uv#46

pip is also dropping support for egg distributions: pypa/pip#12330

There has been work to remove it already: https://discuss.python.org/t/killing-off-the-egg-fragment-once-and-for-all/21660/ with a PR pypa/pip#11617

In the first referenced astral-sh/uv issue; a reply mentions the use of "editable wheels", which I think is the only case for it being used from what I can see in the documentation.

At usage/dependency/#vcs-dependencies

# Or use the #egg fragment
pdm add "git+https://github.com/pypa/[email protected]#egg=pip"
# Install from a subdirectory
pdm add "git+https://github.com/owner/repo.git@master#egg=pkg&subdirectory=subpackage"

I'm currently working on a proof of concept with pdm for use in a monorepo lift. Following the documentation on this and referring to the example repo - the dependencies at the root of the project are written like:

[tool.pdm.dev-dependencies]
dev = [
    "-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
]

This works correctly for installing and working with the project in my testing for the proof of concept.

When attempting to export the dependencies to be read by a tool like uv; the export includes an egg fragment (probably for compatibility).

Here's the output of the pdm-example-monorepo from running pdm export:

# This file is @generated by PDM.
# Please do not edit it manually.

click==8.1.7 \
    --hash=sha256:ae74fb96c20a0277a1d615f1e4d73c8414f5a98db8b799a7931d1582f3390c28 \
    --hash=sha256:ca9853ad459e787e2192211578cc907e7594e294c7ccc834310722b41b9ca6de
colorama==0.4.6; platform_system == "Windows" \
    --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \
    --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6
-e file:///home/python/pdm-example-monorepo/packages/pkg-core#egg=pkg-core
-e file:///home/python/pdm-example-monorepo/packages/pkg-first#egg=pkg-first
-e file:///home/python/pdm-example-monorepo/packages/pkg-second#egg=pkg-second

This will fail uv pip install -r requirements.txt

A workaround is to use sed to strip any #egg fragments from the editable installs from pdm export with a pipe through sed

$ pdm export | sed 's/#egg.*$//' | tee requirements.txt 
STATUS: Resolving packages from lockfile...
STATUS: Fetching hashes for resolved packages...
# This file is @generated by PDM.
# Please do not edit it manually.

click==8.1.7 \
    --hash=sha256:ae74fb96c20a0277a1d615f1e4d73c8414f5a98db8b799a7931d1582f3390c28 \
    --hash=sha256:ca9853ad459e787e2192211578cc907e7594e294c7ccc834310722b41b9ca6de
colorama==0.4.6; platform_system == "Windows" \
    --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \
    --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6
-e file:///home/python/pdm-example-monorepo/packages/pkg-core
-e file:///home/python/pdm-example-monorepo/packages/pkg-first
-e file:///home/python/pdm-example-monorepo/packages/pkg-second

Describe the solution you'd like

An option to drop the egg fragments like --no-hashes, perhaps --no-egg-fragments.
With my own unfamiliarity with code, I able to go as far as attempting to instantiate an instance of the export command in a repl, with a pdm.core.Core() instance, but I ended up falling back to my sed workaround for my proof of concept.

Feel free to modify or drop the suggestion. I appreciate your hard work on pdm!

[yyolk]

I was able to look at this a little deeper today, and determined that this is being set as a result of the requirements from a pyproject.toml or lock file being read in as a pdm.models.requirements.FileRequirement. Specifically its a result of its as_line():

pdm/src/pdm/models/requirements.py

Lines 355 to 372 in 07918b8

	def as_line(self) -> str:
	project_name = f"{self.project_name}" if self.project_name else ""
	extras = f"[{','.join(sorted(self.extras))}]" if self.extras and self.project_name else ""
	marker = self._format_marker()
	if marker:
	marker = f" {marker}"
	url = self.get_full_url()
	fragments = []
	if self.subdirectory:
	fragments.append(f"subdirectory={self.subdirectory}")
	if self.editable:
	if project_name:
	fragments.insert(0, f"egg={project_name}{extras}")
	fragment_str = ("#" + "&".join(fragments)) if fragments else ""
	return f"-e {url}{fragment_str}{marker}"
	delimiter = " @ " if project_name else ""
	fragment_str = ("#" + "&".join(fragments)) if fragments else ""
	return f"{project_name}{extras}{delimiter}{url}{fragment_str}{marker}"

I was able to also determine that for some reason the logic for will result in the FileRequirements which expand with the egg# fragment

pdm/src/pdm/cli/commands/export.py

Lines 59 to 68 in 07918b8

	def handle(self, project: Project, options: argparse.Namespace) -> None:
	if options.pyproject:
	options.hashes = False
	selection = GroupSelection.from_options(project, options)
	requirements: dict[str, Requirement] = {}
	packages: Iterable[Requirement] | Iterable[Candidate]
	for group in selection:
	requirements.update(project.get_dependencies(group))
	if options.pyproject:
	packages = requirements.values()

In my own testing with mocking this code directly, I originally thought that I was following the same logic, but you can see I'm only getting the default dependencies of my modified pdm-example-monorepo which includes this pyproject.toml:

[project]
name = "monorepo"
version = "0.1.0"
dependencies = [
        #none of these work ! they're not supposed to
	#"-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
	#"-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
	#"-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
	#"-e pkg-third @ file:///${PROJECT_ROOT}/packages/pkg-third",
	#"-e pkg-fourth @ file:///${PROJECT_ROOT}/packages/pkg-fourth",
	#"./packages/pkg-core",
	#"./packages/pkg-first",
	#"./packages/pkg-second",
	#"./packages/pkg-third",
	#"./packages/pkg-fourth",
        # these will!
	"pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
	"pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
	"pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
	"pkg-third @ file:///${PROJECT_ROOT}/packages/pkg-third",
	"pkg-fourth @ file:///${PROJECT_ROOT}/packages/pkg-fourth",
]
requires-python = ">=3.8"

[tool.pdm.dev-dependencies]
dev = [
    "-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
    "-e pkg-third @ file:///${PROJECT_ROOT}/packages/pkg-third",
    "-e pkg-fourth @ file:///${PROJECT_ROOT}/packages/pkg-fourth",
]


[tool.pdm]
distribution = false

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

>>> from pdm.core import Core
>>> from pdm.project.core import Project
>>> core = Core()
>>> Project(core, '.')
<Project '/home/python/pdm-example-monorepo'>
>>> proj = Project(core, '.')

>>> packages = requirements.values()
>>> packages
dict_values([FileRequirement(name='pkg-core', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-core', path=PosixPath('packages/pkg-core'), subdirectory=None), FileRequirement(name='pkg-first', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-first', path=PosixPath('packages/pkg-first'), subdirectory=None), FileRequirement(name='pkg-second', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-second', path=PosixPath('packages/pkg-second'), subdirectory=None), FileRequirement(name='pkg-third', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-third', path=PosixPath('packages/pkg-third'), subdirectory=None), FileRequirement(name='pkg-fourth', marker=None, extras=set(), specifier=<SpecifierSet('')>, editable=False, prerelease=None, groups=['default'], url='file:///${PROJECT_ROOT}/packages/pkg-fourth', path=PosixPath('packages/pkg-fourth'), subdirectory=None)])

>>> content = FORMATS["requirements"].export(proj, packages, options)
>>> content
'# This file is @generated by PDM.\n# Please do not edit it manually.\n\npkg-core @ file:///home/python/pdm-example-monorepo/packages/pkg-core\npkg-first @ file:///home/python/pdm-example-monorepo/packages/pkg-first\npkg-fourth @ file:///home/python/pdm-example-monorepo/packages/pkg-fourth\npkg-second @ file:///home/python/pdm-example-monorepo/packages/pkg-second\npkg-third @ file:///home/python/pdm-example-monorepo/packages/pkg-third\n'
>>> print(content)
# This file is @generated by PDM.
# Please do not edit it manually.

pkg-core @ file:///home/python/pdm-example-monorepo/packages/pkg-core
pkg-first @ file:///home/python/pdm-example-monorepo/packages/pkg-first
pkg-fourth @ file:///home/python/pdm-example-monorepo/packages/pkg-fourth
pkg-second @ file:///home/python/pdm-example-monorepo/packages/pkg-second
pkg-third @ file:///home/python/pdm-example-monorepo/packages/pkg-third

Then when running pdm export --pyproject, it picks up the dev dependencies with #egg= fragments as they're FileRequirements:

pdm export --pyroject

# This file is @generated by PDM.
# Please do not edit it manually.

-e file:///home/python/pdm-example-monorepo/packages/pkg-core#egg=pkg-core
-e file:///home/python/pdm-example-monorepo/packages/pkg-first#egg=pkg-first
-e file:///home/python/pdm-example-monorepo/packages/pkg-fourth#egg=pkg-fourth
-e file:///home/python/pdm-example-monorepo/packages/pkg-second#egg=pkg-second
-e file:///home/python/pdm-example-monorepo/packages/pkg-third#egg=pkg-third

As I dug deeper, FileRequirement is what will resolve to, and the -e is always prefixed and #egg is always attached if the package is --editable.

I will get the same results of my repl session above if I run pdm export--pyproject --prod, which was using only the default dependencies and was explicitly set to dev=False with my hack using a NamedTuple instead of a argparse.Namespace:

pdm export --pyproject --prod      
# This file is @generated by PDM.
# Please do not edit it manually.

pkg-core @ file:///home/python/pdm-example-monorepo/packages/pkg-core
pkg-first @ file:///home/python/pdm-example-monorepo/packages/pkg-first
pkg-fourth @ file:///home/python/pdm-example-monorepo/packages/pkg-fourth
pkg-second @ file:///home/python/pdm-example-monorepo/packages/pkg-second
pkg-third @ file:///home/python/pdm-example-monorepo/packages/pkg-third

I think in order for this to be an acceptable feature, there would need to be proof in what happens with a vanilla pip install -e for comparison. Which I've done, and will x-post the relevant bits to here from astral-sh/uv#313 (comment)

In a vanila pip install -e 'file://...', there's a quirk if the path is relative inside a git directory, and will produce something like:

# replace .venv
$ uv v --seed
# install the package as editable with relative path
$ .venv/bin/pip install -e 'file://'$PWD'packages/pkg-second'

# ...

# show pip installation
$ .venv/bin/pip freeze | grep pkg-core

-e git+https://github.com/pdm-project/pdm-example-monorepo@b85261555ea063e68eaf744e225804149c27e64f#egg=pkg_core&subdirectory=packages/pkg-core

However, when not in a git directory, the same way of installation will result in what I'm after:

# replace .venv
$ uv v --seed
# get rid of .git, so it's not detected as a git repository
$ mv .git .gitold
# install the package as editable with relative installation, again
$ .venv/bin/pip install -e 'file://'$PWD'/packages/pkg-core'

# ...

# show pip installation
$ .venv/bin/pip freeze | grep pkg-core

# Editable install with no version control (pkg-core==0.1.0+editable)
-e /home/python/pdm-example-monorepo/packages/pkg-core

Ultimately I can see how I could implement this feature. Through FileRequirement, since there's some open loops on uv with their handling of git+http://... urls which do include egg= in my testing, it may be revisited.

[yyolk]

I'm close to closing this issue as I've come to refactor my thoughts considerably, with my feature request being an edge case and the egg quirk being part of a much larger, and ongoing conversation from 2013, (with it mostly dropped in [email protected] hopefully to drop completely in 2025. And with it only being supported within a VCS repository url that's editable to name the package 🙄

I was originally dealing with theoretical scenarios and have recently moved to concrete implementation which as of now I'm comfortable with needing to use local editable when needed with a sed in place. Since this will only likely occur during a build / CI step it's not that bad.

I found this one other comment today that I stumbled as I'm actively investigating, where mitsuhiko recommends using pdm's method of handling local editables which is also very topical as I'm also currently leveraging pdm for a refactor of a medium sized codebase to a monorepo - and I've looked a lot at the page mentioned a lot 😆

astral-sh/uv#592 (comment)

[yyolk]

I am closing this issue feature request.

Thank you for letting me dump my thoughts here!

The primary reason is that this feature wouldn't be useful to many people, the output of pdm export ... can easily be stream edited to remove the problem parts when using the exported requirements.txt.

Another reason is there's a better option (in my case at least) to use non-editable package paths and rely on project.dependencies in a production deployment setting where other tools that understand PEP-621 have no problems installing those packages and dependencies.

---

Some more context:

I have gone through many scenarios with boilerplate project structures. I've since discovered the best approach for my use case is to not rely on pdm export but instead use PEP-621, with project.dependencies for production deployments. Whereas before I wasn't considering that aspect of it, I was trying to use uv in the context of editable development, where egg would be used.

Now, in this specific use case, I am using both project.dependencies and tool.pdm.dev-dependencies (no need for project.optional-dependencies at this time, which are the equivalent to the extras=[...] a la setup-tools).

During development, I will use pdm. For deployment, I can use pdm sync --no-editables --prod or I can use uv pip install -r pyproject.toml. Since uv is very new, this seems like an OK place to evaluate it for now.

I could also still convert the requirements over through pdm export ..., stripping off the egg=... with sed, but with the approach of having production ("non-editable") project.dependencies, uv can read it directly. With every sub-package within the monorepo, which is referenced at the root of this project also using PEP-621, and project.packages for describing its dependencies, I don't foresee issues at this point, and the option to fallback now that I've adopted pdm into this project, with pdm run ... helpers or other project scripts in place that can staple those commands I don't see this as a desirable feature.

I expect in the future, egg=... will be a distant memory, and including this as a feature will make another thing to remove and make it somewhat more difficult to migrate any other users that would happen to adopt this command, naively¹.

For posterity and anyone that might stumble across this, the monorepo is laid out almost identical to pdm-project/pdm-example-monorepo.

To then use it for uv pip install -r pyproject.toml, the root pyproject.toml would look like this:

[project]
dependencies = [
    "pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
]
requires-python = ">=3.8"

[tool.pdm.dev-dependencies]
dev = [
    "-e pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
    "-e pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
    "-e pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
]

or as a diff:

2c2,6
< dependencies = []
---
> dependencies = [
>     "pkg-core @ file:///${PROJECT_ROOT}/packages/pkg-core",
>     "pkg-first @ file:///${PROJECT_ROOT}/packages/pkg-first",
>     "pkg-second @ file:///${PROJECT_ROOT}/packages/pkg-second",
> ]

Thanks again for your excellent work and this excellent project!

Footnotes

1. Naive in the sense of exploring PEP-621 or use a simple workaround like a sed replacement in the export if they don't go so far as to write their own pdm plugin. ↩