Exception when trying to add packages: [ChunkedEncodingError]: ('Connection broken: IncompleteRead

[Mjboothaus]

• I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

This is a brand new project. Although I have used pdm many times before without any errors.

I have run pdm init and then am trying to add my first package using (in this case for the notebook package - same behaviour with other attempted package adds):

pdm add --dev notebook

Actual behavior

username@My-Laptop working % pdm add -v --dev notebook
Adding group dev to lockfile
Adding packages to dev dev-dependencies: notebook
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   notebook
pdm.termui:   python>=3.11
pdm.termui:   Adding requirement notebook
Traceback (most recent call last):
  File "/opt/homebrew/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/core.py", line 289, in main
    return Core().main(args)
           ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/core.py", line 207, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/core.py", line 202, in main
    self.handle(project, options)
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/cli/commands/add.py", line 60, in handle
    actions.do_add(
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/cli/actions.py", line 280, in do_add
    resolved = do_lock(
               ^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/cli/actions.py", line 99, in do_lock
    mapping, dependencies = resolve(
                            ^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/resolver/providers.py", line 247, in matches_gen
    yield from super_find()
               ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/resolver/providers.py", line 145, in matches_gen
    candidates = self._find_candidates(reqs[0])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/resolver/providers.py", line 124, in _find_candidates
    return self.repository.find_candidates(requirement, requirement.prerelease or self.allow_prereleases)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/models/repositories.py", line 154, in find_candidates
    cans = LazySequence(self._find_candidates(requirement))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/pdm/models/repositories.py", line 354, in _find_candidates
    for c in finder.find_all_packages(requirement.project_name, allow_yanked=requirement.is_pinned)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/unearth/finder.py", line 281, in find_all_packages
    return LazySequence(self._find_packages(package_name, allow_yanked, hashes))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/unearth/finder.py", line 263, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/unearth/collector.py", line 133, in collect_links_from_location
    yield from _collect_links_from_index(session, location)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/unearth/collector.py", line 153, in _collect_links_from_index
    page = fetch_page(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/unearth/collector.py", line 140, in fetch_page
    resp = _get_html_response(session, location)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/unearth/collector.py", line 175, in _get_html_response
    resp = session.get(
           ^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/requests/sessions.py", line 747, in send
    r.content
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pdm/2.7.1/libexec/lib/python3.11/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read, 27115 more expected)', IncompleteRead(0 bytes read, 27115 more expected))

Expected behavior

Normally the packages are added successfully without any error.

Environment Information

PDM version:
  2.7.1
Python Interpreter:
  /Users/mjboothaus/code/github/username/projectname/.venv/bin/python (3.11)
Project Root:
  /Users/mjboothaus/code/github/username/projectname
Local Packages:
  
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.2",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "22.5.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000",
  "python_full_version": "3.11.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "darwin"
}

Other - initial investigation

This seems to be an issue with the update to version 2.7.x.

As an additional note, I thought that there is supposed to be a .pdm.toml file created in the project directory but I am not seeing this file.

It appears from searches I did that there may be some issue with urllib3 incompatibility with the requests package or something like this. I am using a Homebrew install of pdm - although I also tried using a pipx install. With this version I am seeing the following behaviour:

PDM version:
  2.7.0
Python Interpreter:
  /Users/mjboothaus/code/github/mjboothaus/christianity-explored/.venv/bin/python (3.11)
Project Root:
  /Users/mjboothaus/code/github/mjboothaus/christianity-explored
Local Packages:
  
/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py:723: RuntimeWarning: Failed to get latest version: Expecting value: line 1 column 1 (char 0)
  warnings.warn(f"Failed to get latest version: {e}", RuntimeWarning, stacklevel=1)
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.2",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "22.5.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000",
  "python_full_version": "3.11.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "darwin"
}
/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py:723: RuntimeWarning: Failed to get latest version: Expecting value: line 1 column 1 (char 0)
  warnings.warn(f"Failed to get latest version: {e}", RuntimeWarning, stacklevel=1)

and when I try to add a package I get a different error:

Adding group dev to lockfile
Adding packages to dev dev-dependencies: notebook
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   notebook
pdm.termui:   python>=3.11
pdm.termui:   Adding requirement notebook
Traceback (most recent call last):
  File "/Users/mjboothaus/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 289, in main
    return Core().main(args)
           ^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 207, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 202, in main
    self.handle(project, options)
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/commands/add.py", line 60, in handle
    actions.do_add(
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py", line 280, in do_add
    resolved = do_lock(
               ^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py", line 99, in do_lock
    mapping, dependencies = resolve(
                            ^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py", line 247, in matches_gen
    yield from super_find()
               ^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py", line 145, in matches_gen
    candidates = self._find_candidates(reqs[0])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/providers.py", line 124, in _find_candidates
    return self.repository.find_candidates(requirement, requirement.prerelease or self.allow_prereleases)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/models/repositories.py", line 150, in find_candidates
    cans = LazySequence(self._find_candidates(requirement))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/models/repositories.py", line 350, in _find_candidates
    for c in finder.find_all_packages(requirement.project_name, allow_yanked=requirement.is_pinned)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/unearth/finder.py", line 281, in find_all_packages
    return LazySequence(self._find_packages(package_name, allow_yanked, hashes))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/unearth/finder.py", line 263, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/unearth/collector.py", line 133, in collect_links_from_location
    yield from _collect_links_from_index(session, location)
  File "/Users/mjboothaus/.local/pipx/venvs/pdm/lib/python3.11/site-packages/unearth/collector.py", line 83, in parse_json_response
    data = json.loads(page.content)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[Mjboothaus]

I did find this issue... although not sure if it is related/similar behaviour.

pydantic/pydantic#5698

[Mjboothaus]

I also tried a plain pip install pdm into the virtual environment and got the same error as the initial one above.

p.s. thanks for all the great work on pdm, I have tried many/all(?) the Python package managers and I like this the best by far :)

[frostming]

2.7.1 has been yanked from PyPI, but apparently you can still install it with homebrew :(

I also tried a plain pip install pdm into the virtual environment and got the same error as the initial one above.

That's because you have already populated with broken caches, and the old PDM you installed from pip can't parse it.

At this point you can run pdm cache clear http to fix it

[sanmai-NL]

@frostming Please be warned, that now even with the recommended installation method, the yanked 2.7.1. is being installed:

$ curl -sSL https://raw.githubusercontent.com/pdm-project/pdm/main/install-pdm.py | python3 -

Installing PDM (2.7.1): Creating virtual environment
Installing PDM (2.7.1): Installing PDM and dependencies

[Mjboothaus]

Please note that I was still receiving a different error after re-installation (error relating to JSONDecoder from memory) although this resolved when I ran pdm cache clear.

Cheers!

[laluka]

Thanks @frostming , you saved me/us hours! <3