Cant find libcudart.so

[SirWaffle]

I am running on windows, using miniconda3 and python 3.9.

I have cudatoolkit, cudnn, pytorch, transformers, accelerate, bitsandbytes, and dependencies installed via conda.

when attempting to run a simple test script:

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import BloomModel, BloomConfig
from transformers import BloomTokenizerFast, BloomForCausalLM
from transformers import BloomForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("E:/MLModels/bloom")
model = BloomForCausalLM.from_pretrained("E:/MLModels/bloom", device_map="auto", load_in_8bit=True)

prompt = "It was a dark and stormy night"
result_length = 50
inputs = tokenizer(prompt, return_tensors="pt")

tokenout = model.generate(inputs["input_ids"], max_length=result_length)

I see this error when running from a vscode session:

Exception has occurred: RuntimeError
Failed to import transformers.models.bloom.modeling_bloom because of the following error (look up to see its traceback):
argument of type 'WindowsPath' is not iterable

and this output:

WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C'), WindowsPath('/ProgramData/Miniconda3/envs/llm/lib')}
C:\ProgramData\Miniconda3\envs\llm\lib\site-packages\bitsandbytes\cuda_setup\paths.py:97: UserWarning: C:\ProgramData\Miniconda3\envs\llm did not contain libcudart.so as expected! Searching further paths...
  warn(
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!   
CUDA SETUP: Loading binary C:\ProgramData\Miniconda3\envs\llm\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...

I see that its searching for libcudart.so, which is non-existent on my machine.

Is this file supposed to exist in windows? Or do I need to do some trickery to get this working on windows?

[lessw2020]

As a general concept, .so files (shared object) are the equivalent of .dll (dynamic linked library) on windows.
But .so is for linux and android.
Thus, I believe it would take a recompile for Windows before you would be able to run on Windows.

[lessw2020]

note - you may want to just run using Windows subsystem for Linux (WSL) and then you should be able to run as expected.
Alternatively, this is a cuda related file so you may want to check the nvidia forums as well.

[younesbelkada]

Hi @SirWaffle !
Thanks for your message, you may want to check this issue: #17 as I think you had more or less the same issue but not sure

[TimDettmers]

Currently, the library does not offer Windows support. It would be great if you can help us to make it work under Windows. For that, the first step is to compile from source on a Windows machine.

Let's move this discussion to issue #30.

[swumagic]

Bitsandbytes was not supported windows before, but my method can support windows.（yuhuang）
1 open folder J:\StableDiffusion\sdwebui，Click the address bar of the folder and enter CMD
or WIN+R, CMD 。enter，cd /d J:\StableDiffusion\sdwebui
2 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes

3 J:\StableDiffusion\sdwebui\py310\python.exe -m pip uninstall bitsandbytes-windows

4 J:\StableDiffusion\sdwebui\py310\python.exe -m pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl

Replace your SD venv directory file（python.exe Folder） here（J:\StableDiffusion\sdwebui\py310）