Starting tabbyml behind nginx reverse proxy requires websocket support

[oli-ver]

Describe the bug
Running any prompts in the chatbox on the web page of Tabby or in the VSCode plugin result in the following error message:

{
  "error": true,
  "message": "[Network] undefined"
}

Information about your version
Please provide output of tabby --version

./tabby --version
tabby 0.18.0

Information about your GPU
Please provide output of nvidia-smi

Only CPU mode (linux virtual server)

Additional context
Add any other context about the problem here.

Running tabby natively or in Docker on Debian GNU/Linux 11 (bullseye). No GPU available on the system.

Installation steps native:

useradd -m -s /usr/bin/zsh tabby
su - tabby
wget https://github.com/TabbyML/tabby/releases/download/v0.18.0/tabby_x86_64-manylinux2014.zip
unzip tabby_x86_64-manylinux2014.zip
mv dist/* ~
chmod +x tabby llama-server
TABBY_WEBSERVER_JWT_TOKEN_SECRET=mysecret ./tabby serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --port 8090 --host 127.0.0.1 --device cpu

Installation steps docker:

services:
  tabby:
    restart: always
    image: registry.tabbyml.com/tabbyml/tabby
    environment:
     - ENDPOINT=https://my-endpoint.com
     - TABBY_WEBSERVER_JWT_TOKEN_SECRET=mysecret
     - LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:$ #workaround cuda errors, see https://github.com/TabbyML/tabby/issues/2634#issuecomment-2244530283
    entrypoint: /opt/tabby/bin/tabby-cpu
    command: serve --device cpu --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct
    volumes:
      - "./data:/data"
    ports:
      - 127.0.0.1:8090:8080

The only configuration I made was to create a first admin and configure the endpoint URL of my server. I am running zabby behind a nginx reverse proxy with SSL configuration. Both setups natively or in docker result in the same error.

While writing this bug report I found the resolution as well. I tried to run tabby without reverse proxy and it worked. It seemed to be related to my nginx reverse proxy configuration. Initially it was:

        location / {
            proxy_pass       http://localhost:8090;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }

Following the instructions found here I changed it to:

        location / {
            proxy_pass       http://localhost:8090;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

With this configuration it works.

Sharing this experience now more for other users experiencing the same problem.

I will create a pull request for an documentation update both for running behind reverse proxy and how to run in docker without GPU.

[oli-ver]

#3291 resolves the nginx issues, this works fine now. @wsxiaoys can we continue the discussion what should be the preferred way to run tabby in docker without gpu support?

As I said I am using the cpu entrypoint:

entrypoint: /opt/tabby/bin/tabby-cpu

The only VGA adapter my system has is this one (probably some virtual machine graphics adapter):

00:02.0 VGA compatible controller: Device 1234:1111 (rev 02) (prog-if 00 [VGA controller])
        Subsystem: Red Hat, Inc. Device 1100
        Flags: bus master, fast devsel, latency 0
        Memory at fe000000 (32-bit, prefetchable) [size=8M]
        Memory at febd0000 (32-bit, non-prefetchable) [size=4K]
        Expansion ROM at 000c0000 [disabled] [size=128K]
        Kernel driver in use: bochs-drm
        Kernel modules: bochs_drm

Running without the mentioned LD_LIBRARY_PATH results in the following warnings and makes it not possible to run tabby:

tabby-1  | 2024-10-21T15:30:18.506246Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <embedding>: /opt/tabby/bin/llama-server: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory
⠦    90.220 s	Starting...2024-10-21T15:30:19.508165Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:98: llama-server <embedding> exited with status code 127, args: `Command { std: "/opt/tabby/bin/llama-server" "-m" "/data/models/TabbyML/Nomic-Embed-Text/ggml/model.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }`
tabby-1  | 2024-10-21T15:30:19.508195Z  WARN llama_cpp_server::supervisor: crates/llama-cpp-server/src/supervisor.rs:110: <embedding>: /opt/tabby/bin/llama-server: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

When setting the variable LD_LIBRARY_PATH to /usr/local/cuda/lib64:/usr/local/cuda/compat the warnings vanish and Tabby runs and works:

$ docker compose logs -f
⠹     1.763 s	Starting...
tabby-1  | ████████╗ █████╗ ██████╗ ██████╗ ██╗   ██╗
tabby-1  | ╚══██╔══╝██╔══██╗██╔══██╗██╔══██╗╚██╗ ██╔╝
tabby-1  |    ██║   ███████║██████╔╝██████╔╝ ╚████╔╝ 
tabby-1  |    ██║   ██╔══██║██╔══██╗██╔══██╗  ╚██╔╝  
tabby-1  |    ██║   ██║  ██║██████╔╝██████╔╝   ██║   
tabby-1  |    ╚═╝   ╚═╝  ╚═╝╚═════╝ ╚═════╝    ╚═╝   
tabby-1  | 
tabby-1  | 📄 Version 0.18.0
tabby-1  | 🚀 Listening at http://0.0.0.0:8080
tabby-1  | 

These paths exist inside the container, not sure how they are being used:

docker compose exec -it tabby /bin/bash
root@60659fbcee75:/# ls /usr/local/cuda/lib64
libOpenCL.so.1             libcudart.so.11.7.99    libcufile_rdma.so.1.3.1    libcusparse.so.11.7.4.91  libnppidei.so.11.7.4.75  libnppist.so.11.7.4.75  libnvToolsExt.so.1            libnvrtc.so.11.2
libOpenCL.so.1.0           libcufft.so.10          libcurand.so.10            libnppc.so.11             libnppif.so.11           libnppisu.so.11         libnvToolsExt.so.1.0.0        libnvrtc.so.11.7.99
libOpenCL.so.1.0.0         libcufft.so.10.7.2.91   libcurand.so.10.2.10.91    libnppc.so.11.7.4.75      libnppif.so.11.7.4.75    libnppisu.so.11.7.4.75  libnvblas.so.11
libcublas.so.11            libcufftw.so.10         libcusolver.so.11          libnppial.so.11           libnppig.so.11           libnppitc.so.11         libnvblas.so.11.10.3.66
libcublas.so.11.10.3.66    libcufftw.so.10.7.2.91  libcusolver.so.11.4.0.1    libnppial.so.11.7.4.75    libnppig.so.11.7.4.75    libnppitc.so.11.7.4.75  libnvjpeg.so.11
libcublasLt.so.11          libcufile.so.0          libcusolverMg.so.11        libnppicc.so.11           libnppim.so.11           libnpps.so.11           libnvjpeg.so.11.8.0.2
libcublasLt.so.11.10.3.66  libcufile.so.1.3.1      libcusolverMg.so.11.4.0.1  libnppicc.so.11.7.4.75    libnppim.so.11.7.4.75    libnpps.so.11.7.4.75    libnvrtc-builtins.so.11.7
libcudart.so.11.0          libcufile_rdma.so.1     libcusparse.so.11          libnppidei.so.11          libnppist.so.11          libnvToolsExt.so        libnvrtc-builtins.so.11.7.99
root@60659fbcee75:/# ls /usr/local/cuda/compat
libcuda.so  libcuda.so.1  libcuda.so.515.105.01  libnvidia-nvvm.so  libnvidia-nvvm.so.4  libnvidia-nvvm.so.515.105.01  libnvidia-ptxjitcompiler.so.1  libnvidia-ptxjitcompiler.so.515.105.01
root@60659fbcee75:/#

For me this is for now fine, simply because it works, but I don't think this is the way to run it in docker. If I understood correctly from our discussion before it should not work like that.

I assume the cuda driver can be loaded first and does not cause errors and then tabby uses the CPU mode because of the entrypoint that was set?

When running prompts my CPU cores are utilized fully and I receive answers in an acceptable time frame (single user usage):