From ef5ffec90a4a4fb35dd87d474fd5e702e2bf74cd Mon Sep 17 00:00:00 2001
From: Jack Eadie <jack@spice.ai>
Date: Tue, 5 Nov 2024 10:04:02 +1000
Subject: [PATCH] CUDA & Apple silicon support (#575)

* CUDA & Apple silicon support

* update
---
 spiceaidocs/docs/installation.md | 40 ++++++++++++++++++++++++++++++--
 1 file changed, 38 insertions(+), 2 deletions(-)

diff --git a/spiceaidocs/docs/installation.md b/spiceaidocs/docs/installation.md
index e59f9bd1..a0537a27 100644
--- a/spiceaidocs/docs/installation.md
+++ b/spiceaidocs/docs/installation.md
@@ -74,7 +74,7 @@ x86_x64 and ARM binaries for linux, Windows, and macOS are available for downloa
     sudo apt update
     sudo apt install build-essential curl openssl libssl-dev pkg-config protobuf-compiler cmake
     ```
-  
+
   2. Install Go
     ```shell
     export GO_VERSION="1.22.4"
@@ -120,4 +120,40 @@ SPICED_CUSTOM_FEATURES="postgres sqlite" make install
 # Run the following to temporarily add spice to your PATH.
 # Add it to the end of your .bashrc or .zshrc to permanently add spice to your PATH.
 export PATH="$PATH:$HOME/.spice/bin"
-```
\ No newline at end of file
+```
+
+### Build with Hardware Acceleration
+Spice OSS supports running both local language models and embedding models on dedicated hardware. This is for models from either Huggingface or models located locally.
+
+Currently, neither CUDA nor metal have dedicated Dockerfiles or release binaries.
+
+#### CUDA Support
+
+**Steps**:
+1. GPUs with Cuda compute capabilities < 7.5 are not supported (V100, Titan V, GTX 1000 series, ...).
+1. Ensure both Cuda and associated Nvidia drivers are installed. Requires CUDA version 12.2 or higher
+1. Ensure Nvidia binaries are in your path:
+```shell
+export PATH=$PATH:/usr/local/cuda/bin
+```
+1. Build Spice OSS with CUDA support:
+```shell
+make install-with-models-cuda
+```
+
+This ensures CUDA devices are selected on model load, and CUDA-specifiy kernels used when possible.
+
+#### Metal Support
+
+**Steps**:
+1. Ensure you have Xcode installed.
+```shell
+xcode-select --install
+```
+
+1. Build Spice OSS with Metal support:
+```shell
+make install-with-models-metal
+```
+
+Similarily, this ensures Metal devices are selected on model load, and Metal-specific kernels used when possible.