TypeError when updating/adding packages to a project #2027

jnboehm · 2023-06-16T09:40:59Z

I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

With version 2.7.4 I get a TypeError when updating/adding packages to a project (while fetching hashes).

Actual behavior

STATUS: Fetching hashes for resolved packages...
Traceback (most recent call last):
  File "/gpfs01/berens/user/nboehm/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 289, in main
    return Core().main(args)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 207, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 202, in main
    self.handle(project, options)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/cli/commands/update.py", line 71, in handle
    self.do_update(
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/cli/commands/update.py", line 176, in do_update
    do_sync(
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/cli/actions.py", line 190, in do_sync
    synchronizer = project.core.synchronizer_class(
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/synchronizers.py", line 131, in __init__
    self.working_set = environment.get_working_set()
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/environments/base.py", line 148, in get_working_set
    return WorkingSet([paths["platlib"], paths["purelib"]])
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/models/working_set.py", line 66, in __init__
    self._dist_map = {normalize_name(dist.metadata["Name"]): dist for dist in distributions(path=paths)}
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/models/working_set.py", line 66, in <dictcomp>
    self._dist_map = {normalize_name(dist.metadata["Name"]): dist for dist in distributions(path=paths)}
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/utils.py", line 339, in normalize_name
    name = re.sub(r"[^A-Za-z0-9]+", "-", name)
  File "/gpfs01/berens/user/nboehm/miniconda3/envs/py10/lib/python3.10/re.py", line 209, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

I'm not sure what the exact underlying issue is, but calling pdm update --update-eager or pdm add some-package triggered the above error. I'm not really sure how to further debug it. The version I updated from was 2.7.0.

Expected behavior

The program updates/adds the requested packages.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
PDM version:
  2.7.4
Python Interpreter:
  /gpfs01/berens/user/nboehm/miniconda3/envs/py311/bin/python3.11 (3.11)
Project Root:
  /gpfs01/berens/user/nboehm/dev/cells
Local Packages:

{
  "implementation_name": "cpython",
  "implementation_version": "3.11.3",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "3.10.0-1160.el7.x86_64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Mon Oct 19 16:18:59 UTC 2020",
  "python_full_version": "3.11.3",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "linux"
}

The text was updated successfully, but these errors were encountered:

pawamoy · 2023-06-16T09:44:56Z

Maybe try to clear PDM's cache 😕

pdm cache clear hashes
pdm cache clear http

jnboehm · 2023-06-16T09:58:10Z

No, that unfortunately did not resolve the problem. Thanks for the quick response!

frostming · 2023-06-16T10:01:53Z

Can you provide a stable steps to reproduce? I guess you can't if you start from a new project.

jnboehm · 2023-06-16T10:10:39Z

Yeah, trying to start a new project just works. Not really sure what problem occurred when I updated :/ I suppose I could actually just create a new repository and install everything from scratch, so maybe that would be the solution in this case.

frostming · 2023-06-16T10:26:30Z

self._dist_map = {normalize_name(dist.metadata["Name"])

The error says the Name is invalid in the installed packages set. You can check the $env/lib/python3.11/site-packages/*.dist-info/METADATA files to see why they are broken.

jnboehm · 2023-06-16T11:59:36Z

Thanks for the hint. What would I roughly be looking for? I'm not sure in what way the name would need to be mangled in order to return something that raises the TypeError

frostming · 2023-06-16T12:07:21Z

Thanks for the hint. What would I roughly be looking for? I'm not sure in what way the name would need to be mangled in order to return something that raises the TypeError

idk, possibly a None or missing. metadata["Name"] returns None when the key is missing.

jnboehm · 2023-06-16T12:24:04Z

Hmm, could it be related to a dependency that I specified with an URL? I tried to install all the same packages in a different, fresh project and got the following error:

pdm install -v
STATUS: Resolving packages from lockfile...
STATUS: Fetching hashes for resolved packages...
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

  ✖ Install jaxlib 0.4.12+cuda12.cudnn89 failed
pdm.termui: Error occurs:
Traceback (most recent call last):
  File "/gpfs01/berens/user/nboehm/miniconda3/envs/py10/lib/python3.10/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/synchronizers.py", line 275, in install_candidate
    self.manager.install(can)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 375, in build
    self.obtain(allow_all=False)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 407, in obtain
    raise CandidateNotFound(f"The URL requirement {self.req.as_line()} is a wheel but incompatible")
pdm.exceptions.CandidateNotFound: The URL requirement jaxlib @ https://storage.googleapis.com/jax-releases/cuda12/jaxlib-0.4.12+cuda12.cudnn89-cp311-cp311-manylinux2014_x86_64.whl is a wheel but incompatible
Retry failed jobs
  ✖ Install jaxlib 0.4.12+cuda12.cudnn89 failed
pdm.termui: Error occurs:
Traceback (most recent call last):
  File "/gpfs01/berens/user/nboehm/miniconda3/envs/py10/lib/python3.10/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/synchronizers.py", line 275, in install_candidate
    self.manager.install(can)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 375, in build
    self.obtain(allow_all=False)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/models/candidates.py", line 407, in obtain
    raise CandidateNotFound(f"The URL requirement {self.req.as_line()} is a wheel but incompatible")
pdm.exceptions.CandidateNotFound: The URL requirement jaxlib @ https://storage.googleapis.com/jax-releases/cuda12/jaxlib-0.4.12+cuda12.cudnn89-cp311-cp311-manylinux2014_x86_64.whl is a wheel but incompatible
Traceback (most recent call last):
  File "/gpfs01/berens/user/nboehm/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 289, in main
    return Core().main(args)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 207, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 202, in main
    self.handle(project, options)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/cli/commands/install.py", line 88, in handle
    actions.do_sync(
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/cli/actions.py", line 204, in do_sync
    synchronizer.synchronize()
  File "/gpfs01/berens/user/nboehm/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/synchronizers.py", line 443, in synchronize
    raise InstallationError("Some package operations are not complete yet")
pdm.exceptions.InstallationError: Some package operations are not complete yet

Could that be somehow related? I installed jaxlib previously (which worked), but it doesn't show up when I type pdm self list.

The METADATA for the jaxlib package is as follows (at least the name looks okay to me):

Metadata-Version: 2.1
Name: jaxlib
Version: 0.4.12+cuda12.cudnn89
Summary: XLA library for JAX
Home-page: https://github.com/google/jax
Author: JAX team
Author-email: [email protected]
License: Apache-2.0
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: scipy (>=1.7)
Requires-Dist: numpy (>=1.21)
Requires-Dist: ml-dtypes (>=0.1.0)
Provides-Extra: cuda11_pip
Requires-Dist: nvidia-cublas-cu11 (>=11.11) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cuda-cupti-cu11 (>=11.8) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cuda-nvcc-cu11 (>=11.8) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cuda-runtime-cu11 (>=11.8) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cudnn-cu11 (>=8.8) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cufft-cu11 (>=10.9) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cusolver-cu11 (>=11.4) ; extra == 'cuda11_pip'
Requires-Dist: nvidia-cusparse-cu11 (>=11.7) ; extra == 'cuda11_pip'
Provides-Extra: cuda12_pip
Requires-Dist: nvidia-cublas-cu12 ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cuda-cupti-cu12 ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cuda-nvcc-cu12 ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cuda-runtime-cu12 ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cudnn-cu12 (>=8.9) ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cufft-cu12 ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cusolver-cu12 ; extra == 'cuda12_pip'
Requires-Dist: nvidia-cusparse-cu12 ; extra == 'cuda12_pip'

# jaxlib: support library for JAX

jaxlib is the support library for JAX. While JAX itself is a pure Python package,
jaxlib contains the binary (C/C++) parts of the library, including Python bindings,
the XLA compiler, the PJRT runtime, and a handful of handwritten kernels.
For more information, including installation and build instructions, refer to main
JAX README: https://github.com/google/jax/.

jnboehm · 2023-06-19T08:26:59Z

Okay, I took another look today and found out that after (manually) updating from jaxlib-0.4.10 to 0.4.12 the old folder jaxlib-0.4.10+cuda12.cudnn88.dist-info is still present, but the METADATA file is missing (it's actually empty, except for the direct_url.json file). So it seems that the update process messed something up.

The way I "updated" was by way of updating the URL (since jax with GPU support can only be installed that way). So my hunch is that after I changed the line from

"jaxlib @ https://storage.googleapis.com/jax-releases/cuda12/jaxlib-0.4.10+cuda12.cudnn88-cp311-cp311-manylinux2014_x86_64.whl",

to

"jaxlib @ https://storage.googleapis.com/jax-releases/cuda12/jaxlib-0.4.12+cuda12.cudnn89-cp311-cp311-manylinux2014_x86_64.whl",

something broke in the update process. Not really sure where the fault lies here, but removing the folder manually fixed the error for me.

nejcskofic · 2023-06-28T09:55:27Z

I encountered the same issue. Issue is that metadata["Name"] returns None. In my case offending package wascertifi. I'm using conda environment with some preinstalled packages (internal tools needed for development). Below are steps to reproduce it:

Create new conda environment with certifi==2023.5.7. Site packages contain bellow entries for certifi:

├── certifi
│   ├── cacert.pem
│   ├── core.py
│   ├── __init__.py
│   ├── __main__.py
│   └── py.typed
├── certifi-2023.5.7-py3.11.egg-info
│   ├── dependency_links.txt
│   ├── installed-files.txt
│   ├── not-zip-safe
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt

Invoke pdm install where lock file specifies version 2022.12.7 forcing downgrade. Downgrade completes without an error.
Invoke pdm list. Command fails with [TypeError]: cannot use a string pattern on a bytes-like object. Content of site packages for certifi:

├── certifi
│   ├── cacert.pem
│   ├── core.py
│   ├── __init__.py
│   ├── __main__.py
│   └── py.typed
├── certifi-2022.12.7.dist-info
│   ├── LICENSE
│   ├── METADATA
│   ├── RECORD
│   ├── top_level.txt
│   └── WHEEL
├── certifi-2023.5.7-py3.11.egg-info
│   └── installed-files.txt

Removing folder certifi-2023.5.7-py3.11.egg-info resolved the issue.

jnboehm added the 🐛 bug Something isn't working label Jun 16, 2023

frostming closed this as completed in 8984291 Jul 3, 2023

j178 mentioned this issue Apr 3, 2024

recursive script j178/pdm#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeError when updating/adding packages to a project #2027

TypeError when updating/adding packages to a project #2027

jnboehm commented Jun 16, 2023

pawamoy commented Jun 16, 2023

jnboehm commented Jun 16, 2023

frostming commented Jun 16, 2023

jnboehm commented Jun 16, 2023

frostming commented Jun 16, 2023

jnboehm commented Jun 16, 2023

frostming commented Jun 16, 2023 •

edited

Loading

jnboehm commented Jun 16, 2023

jnboehm commented Jun 19, 2023 •

edited

Loading

nejcskofic commented Jun 28, 2023

TypeError when updating/adding packages to a project #2027

TypeError when updating/adding packages to a project #2027

Comments

jnboehm commented Jun 16, 2023

Steps to reproduce

Actual behavior

Expected behavior

Environment Information

pawamoy commented Jun 16, 2023

jnboehm commented Jun 16, 2023

frostming commented Jun 16, 2023

jnboehm commented Jun 16, 2023

frostming commented Jun 16, 2023

jnboehm commented Jun 16, 2023

frostming commented Jun 16, 2023 • edited Loading

jnboehm commented Jun 16, 2023

jnboehm commented Jun 19, 2023 • edited Loading

nejcskofic commented Jun 28, 2023

frostming commented Jun 16, 2023 •

edited

Loading

jnboehm commented Jun 19, 2023 •

edited

Loading