Skip to content

Commit

Permalink
Update tutorial-installing-llamacpp-aws-cuda.md
Browse files Browse the repository at this point in the history
  • Loading branch information
mcharytoniuk authored May 21, 2024
1 parent 2a3e1b2 commit df6b330
Showing 1 changed file with 12 additions and 8 deletions.
20 changes: 12 additions & 8 deletions tutorial-installing-llamacpp-aws-cuda.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,34 @@ system. This tutorial was specifically written for an installation on a `Ubuntu
1. Start an EC2 instance of any class with a GPU with CUDA support.

If you want to compile llama.cpp on this instance, you will need at least 4GB for CUDA drivers and enough space for your LLM of choice. I recommed at least 30GB. Perform the following steps of this tutorial on the instance you started.
2. Install NVIDIA Drivers:

2. Install build dependencies:
```shell
sudo apt update
```

```shell
sudo apt install build-essential ccache
```

3. Install NVIDIA Drivers:
```shell
sudo apt install nvidia-driver-550-server nvidia-headless-550-server nvidia-utils-550-server
```

3. Install CUDA Toolkit (only the Base Installer). Download it and follow instructions from
4. Install CUDA Toolkit (only the Base Installer). Download it and follow instructions from
https://developer.nvidia.com/cuda-downloads

At the time of writing this tutorial, the highest available Ubuntu version supported is 22.04. But do not fear! :) We'll get it to work with some small workarounds (see the [Potential Errors](#potential-errors) section)
< At the time of writing this tutorial, the highest available Ubuntu version supported is 22.04. But do not fear! :) We'll get it to work with some small workarounds (see the [Potential Errors](#potential-errors) section)
4. Compile llama.cpp:
5. Compile llama.cpp:
```shell
git clone https://github.com/ggerganov/llama.cpp.git
```
```shell
cd llama.cpp
```
```shell
LLAMA_CUDA=1 make
LLAMA_CUDA=1 make -j
```
5. Benchmark llama.cpp (optional):
Expand Down Expand Up @@ -99,7 +103,7 @@ use T4 GPU, which would be `compute_75`.
For example:

```shell
CUDA_DOCKER_ARCH=compute_75 LLAMA_CUDA=1 make
CUDA_DOCKER_ARCH=compute_75 LLAMA_CUDA=1 make -j
```

### NVCC not found
Expand Down

0 comments on commit df6b330

Please sign in to comment.