Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: add cURL support to server Dockerfiles #6474

Merged

Conversation

elepedus
Copy link
Contributor

@elepedus elepedus commented Apr 4, 2024

Adding cURL to support --hf-repo and --hf-file params when running through Docker.

@phymbert
Copy link
Collaborator

phymbert commented Apr 4, 2024

Thanks, I think you can do all images in one PR for server

@elepedus
Copy link
Contributor Author

elepedus commented Apr 4, 2024

Roger that. I'll add to this PR :)

@elepedus elepedus changed the title server: add cURL support to full.Dockerfile server: add cURL support to server Dockerfiles Apr 4, 2024
@elepedus
Copy link
Contributor Author

elepedus commented Apr 4, 2024

I'm having some issues testing some of the platform-specific images locally e.g. CUDA/Rocm/Vulkan. Will our CI pipeline catch issues if I just submit a PR?

e.g the CUDA images don't find the CUDA libraries (is it because I'm on a Mac?)

docker run -p 8000:8000 local/llama.cpp:full-cuda-curl -s -m ./gemma-2b-it-q4_k_m.gguf --hf-repo lmstudio-ai/gemma-2b-it-GGUF --hf-file gemma-2b-it-q4_k_m.gguf --port 8000 --host 0.0.0.0 -n 512 
./server: error while loading shared libraries: libcuda.so.1: cannot open shared object file: No such file or directory

e.g.2: the ROCm images error out when installing packages due to GPG signature issues

 => CACHED [5/8] WORKDIR /app                                                                                                                                                                         0.0s
 => [6/8] COPY . .                                                                                                                                                                                    0.0s
 => ERROR [7/8] RUN apt-get update &&     apt-get install -y libcurl4-openssl-dev                                                                                                                     2.3s
------
 > [7/8] RUN apt-get update &&     apt-get install -y libcurl4-openssl-dev:
0.325 Get:1 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
0.325 Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
0.424 Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
0.448 Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
0.618 Get:5 https://repo.radeon.com/amdgpu/5.6/ubuntu jammy InRelease [5427 B]
0.723 Get:6 https://repo.radeon.com/rocm/apt/5.6 jammy InRelease [2615 B]
0.736 Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
0.736   At least one invalid signature was encountered.
1.048 Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
1.048   At least one invalid signature was encountered.
1.353 Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
1.353   At least one invalid signature was encountered.
1.656 Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
1.656   At least one invalid signature was encountered.
1.961 Err:5 https://repo.radeon.com/amdgpu/5.6/ubuntu jammy InRelease
1.961   At least one invalid signature was encountered.
2.264 Err:6 https://repo.radeon.com/rocm/apt/5.6 jammy InRelease
2.264   At least one invalid signature was encountered.
2.290 Reading package lists...
2.333 W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: At least one invalid signature was encountered.
2.333 E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
2.333 W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: At least one invalid signature was encountered.
2.333 E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
2.333 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: At least one invalid signature was encountered.
2.333 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
2.333 W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: At least one invalid signature was encountered.
2.333 E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
2.333 W: GPG error: https://repo.radeon.com/amdgpu/5.6/ubuntu jammy InRelease: At least one invalid signature was encountered.
2.333 E: The repository 'https://repo.radeon.com/amdgpu/5.6/ubuntu jammy InRelease' is not signed.
2.333 W: GPG error: https://repo.radeon.com/rocm/apt/5.6 jammy InRelease: At least one invalid signature was encountered.
2.333 E: The repository 'https://repo.radeon.com/rocm/apt/5.6 jammy InRelease' is not signed.

e.g.3: Intel server image downloads the model but fails to load it (probably because I'm on a Mac)

|ID|       Device Type|                                         Name|capability|units      |group   |group  |Global mem size|
|--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------|
| 0|    [opencl:cpu:0]|                       VirtualApple @ 2.50GHz|       3.0|         12|    8192|     64|     8221675520|
| 1|    [opencl:acc:0]|               Intel(R) FPGA Emulation Device|       1.2|         12|67108864|     64|     8221675520|
ggml_backend_sycl_set_mul_device_mode: true
llama_model_load: error loading model: DeviceList is empty. -30 (PI_ERROR_INVALID_VALUE)
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model './gemma-2b-it-q4_k_m.gguf'

e.g.4: Vulcan can't install SDK due to GPG key errors

ERROR: failed to solve: process "/bin/sh -c wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | apt-key add - &&     wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list &&     apt update -y &&     apt-get install -y vulkan-sdk" did not complete successfully: exit code: 100

@phymbert
Copy link
Collaborator

phymbert commented Apr 4, 2024

You cannot test CUDA image without nvidia container runtime. It's OK.
I worry more about the EditorConfig issue:
https://github.com/ggerganov/llama.cpp/actions/runs/8553404533/job/23436492415?pr=6474

Probably linked with:

@elepedus elepedus force-pushed the update_docker_images_to_support_curl branch from be01d76 to 42f31ca Compare April 4, 2024 12:09
@phymbert phymbert merged commit 0a1d889 into ggerganov:master Apr 4, 2024
21 checks passed
tybalex pushed a commit to rubra-ai/tools.cpp that referenced this pull request Apr 17, 2024
* server: add cURL support to `full.Dockerfile`

* server: add cURL support to `full-cuda.Dockerfile` and `server-cuda.Dockerfile`

* server: add cURL support to `full-rocm.Dockerfile` and `server-rocm.Dockerfile`

* server: add cURL support to `server-intel.Dockerfile`

* server: add cURL support to `server-vulkan.Dockerfile`

* fix typo in `server-vulkan.Dockerfile`

Co-authored-by: Georgi Gerganov <[email protected]>

---------

Co-authored-by: Georgi Gerganov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants