ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'

[LucienBrule]

Environment

Windows Version:
    Microsoft Windows [Version 10.0.20246.1]

cat /etc/lsb-release:

    DISTRIB_ID=Ubuntu
    DISTRIB_RELEASE=20.04
    DISTRIB_CODENAME=focal
    DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

Kernel:
    Linux version 4.19.128-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) 

wsl --list -v 
    * Ubuntu    Running         2

Driver Info:
    Nvidia Driver 460.20
    Card: RTX 3080

Steps to reproduce

1. Install CUDA 11.1 Follow guide https://docs.nvidia.com/cuda/wsl-user-guide/index.html
2. Install cuDNN v8.0.4 Follow guide https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html
3. Setup Conda

$ conda create -n tf
$ conda activate tf
$ pip install tf-nightly-gpu
$ pip install jupyter

4. Start jupyter server, copy paste blocks from this notebook https://www.tensorflow.org/hub/tutorials/image_feature_vector
5. Go to "Train the network"

Expected behavior

Afaik I'm running the latest if not late enough builds of everything so, no warnings, just standard info.

Actual behavior

Massive console spam of

[timestamp] : W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal   : Value 'sm_86' is not defined for option 'gpu-name'

It still worked fine but seemed worth it to report.

Question

• What in the CUDA stack makes it too old?
• Should I downgrade CUDA in WSL to match something else?

[cshang2017]

Has the same issue.

[gdh1995]

It's because tensorflow lacks supports for new GPUs, but not an error of WSL .

I find some related issues in the tensorflow repo, like tensorflow/tensorflow#43718

[Harsh188]

Hi, the latest version of TensorFlow supports CUDA 11 for the new GPUs. However, I'm getting a similar issue. Check out the issue I raised on the tensorflow repo.

[LucienBrule]

Thank you all for your input, will look into it further

[WillLiGitHub]

same issue in Geforce rtx 3080

$ failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
$ W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'

set system env can fix my problem:
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

[Harsh188]

Hello,

I've found a software that makes TF 2.30 compatible with cudav11.1. It works only for Ubuntu 20.04 at the moment. If you are interested, then check out the article Install TensorFlow & PyTorch for the RTX 3090, 3080, 3070 by lambda stack.

[usr-ein]

same issue in Geforce rtx 3080

$ failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
$ W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'

set system env can fix my problem:
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

I tried this solution, but all it does is stop using the GPU:

E tensorflow/stream_executor/cuda/cuda_driver.cc:328] failed call to cuInit: CUDA_ERROR_NO_DEVICE: 
no CUDA-capable device is detected

[westlake-moonlight]

I met this issue yesterday. It turns out that this problem is caused by the old "ptxas", which doesn't support RTX3090, 3080 GPUs.
On my computer, there were 2 cudas: cuda-11.0 and cuda-11.1 in the directory usr/local. The TensorFlow was trying to use the cuda-11.0 somehow, and the old "ptxas" in the folder cuda-11.0/bin is causing the problem.

Solution: I renamed the cuda-11.0 to something else, so that the TensorFlow can only find and use the "ptxas" in the cuda-11.1. After that, the problem is fixed: the warning was gone and everything is OK.

Environment: Ubuntu 20.04, TensorFlow 2.4, Cuda 11.1.1, CuDNN 8.05, Nvidia Driver Version: 455.32.

[johndpope]

I'm successfully running rtx 3090 with latest beta 470 driver + cuda 11.4 but need to use gcc9(stay away from 10.3 / system update)
I recommend using timeshift to save state when everything is working.
This morning I had to roll back a system update that made the graphics card driver downgrade to 460. crazy.

UPDATE - Sorry I'm not using WSL. just native dual boot.

[alon1samuel]

I'm also getting the same problem
Environment: Ubuntu 20.04, TensorFlow 2.4, Cuda 11.4.0, CuDNN 8.2.2.26, Nvidia Driver Version: 470.57.02, nvcc - 10.1

2021-07-30 18:18:03.339065: W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'

[talhaanwarch]

usr/local

renaming did not helped me

[microsoft-github-policy-service]

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!

[xman3911]

I have the same issue when use Tensorflow2.4.1 with CUDA11.0. But it is solved when I use Tesorflow2.5.0 with CUDA11.2. So the reason maybe is the CUDA's version is too old.