libcusparse.so.11 not defined in file libcusparse.so.11 with link time reference

[ZhiyuanChen]

🐛 Describe the bug

import torch

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.9/site-packages/torch/__init__.py", line 192, in <module>
    from torch._C import *  # noqa: F403
ImportError: /opt/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cu.so: symbol cusparseSpSM_analysis, version libcusparse.so.11 not defined in file libcusparse.so.11 with link time reference

Versions

PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 7.6.1810 (Core) (x86_64)
GCC version: (Anaconda gcc) 11.2.0
Clang version: 3.4.2 (tags/RELEASE_34/dot2-final)
CMake version: version 3.22.1
Libc version: glibc-2.17

Python version: 3.9.13 (main, Aug 25 2022, 23:26:10) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: N/A
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.4.0
[pip3] torch==1.13.0
[pip3] torchaudio==0.13.0
[pip3] torchvision==0.14.0
[conda] blas 1.0 mkl https://repo.anaconda.com/pkgs/main
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.4.0 h06a4308_640 https://repo.anaconda.com/pkgs/main
[conda] mkl-service 2.4.0 py39h7f8727e_0 https://repo.anaconda.com/pkgs/main
[conda] mkl_fft 1.3.1 py39hd3c417c_0 https://repo.anaconda.com/pkgs/main
[conda] mkl_random 1.2.2 py39h51133e4_0 https://repo.anaconda.com/pkgs/main
[conda] numpy 1.21.5 py39h6c91a56_3 https://repo.anaconda.com/pkgs/main
[conda] numpy-base 1.21.5 py39ha15fc14_3 https://repo.anaconda.com/pkgs/main
[conda] numpydoc 1.4.0 py39h06a4308_0 https://repo.anaconda.com/pkgs/main
[conda] pytorch 1.13.0 py3.9_cuda11.7_cudnn8.5.0_0 pytorch
[conda] pytorch-cuda 11.7 h67b0de4_0 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torchaudio 0.13.0 py39_cu117 pytorch
[conda] torchvision 0.14.0 py39_cu117 pytorch

cc @ezyang @gchanan @zou3519 @seemethere @malfet @ngimel

[ZhiyuanChen]

It was a clean conda install with no additional package, the command is
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia

[samdow]

Seems related to #89417

[bottler]

The workaround conda install libcusparse=11.7.3.50 -c nvidia given in #90673 might help.

[malfet]

Likely fixed by pytorch/builder#1227
@atalman can you please check

[atalman]

@malfet testing the fix now

[malfet]

Fixed, closing (please reopen/file new one if this is not the case)

[ZhiyuanChen]

I tried a clean install again but seems still not working

[ZhiyuanChen]

The workaround conda install libcusparse=11.7.3.50 -c nvidia given in #90673 might help.

this does help

[atalman]

@ZhiyuanChen it should be fixed I do see our packages are updated here: https://conda.anaconda.org/pytorch/noarch/

pytorch-cuda-11.7-h67b0de4_1.tar.bz2 | 3 KB | 2022-12-13 00:53:26 +0000

pytorch-cuda-11.7-h67b0de4_1.tar.bz2 3 KB 2022-12-13 00:53:26 +0000

[ZhiyuanChen]

@ZhiyuanChen it should be fixed I do see our packages are updated here: https://conda.anaconda.org/pytorch/noarch/

pytorch-cuda-11.7-h67b0de4_1.tar.bz2 | 3 KB | 2022-12-13 00:53:26 +0000

pytorch-cuda-11.7-h67b0de4_1.tar.bz2 3 KB 2022-12-13 00:53:26 +0000

I think I'll try again later.... Clean install is really painful