Assertion Error with OpenAI API Server on Linux

[jasonacox]

The following tests run with no issue on my Mac (M2). However, an assertion error occurs running these same API calls on a Linux Ubuntu 22.04 box (using CPU only and with a GTX 3090 GPU).

Test - Download https://huggingface.co/jartine/mistral-7b.llamafile/blob/main/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile

# Run Server

chmod +x mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile
./mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile

# Run API Test

curl -i http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
    "role": "system",
    "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
},
{
    "role": "user",
    "content": "Write a limerick about python exceptions"
}
]
}'

Error

loading weights...
{"timestamp":1702100777,"level":"INFO","function":"main","line":3039,"message":"HTTP server listening","hostname":"127.0.0.1","port":8080}
all slots are idle and system prompt is empty, clear the KV cache
llama.cpp/server/json.h:21313: assert(it != m_value.object->end()) failed (cosmoaddr2line /data/ai/llamafile/mistral-7b 4247a4 42ba67 42ce5b 45da

Ref: #24

[jart]

This was fixed in #36. I just tested the weights that are on Hugging Face. I can't reproduce this issue. Could you please run the following command for me:

sha256sum mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile

If it prints something other than 9fc5f94f1fb497744931fd31362248d43bea89b00a41fd5d65bcdb19f5c501ef then please either redownload the weights from Hugging Face. If your Internet connection is slow, then please build llamafile from source using the instructions in the README and then adapt the following instructions to transplant your weights into the llamafile you just built. #24 (comment)

[jasonacox]

Thanks, @jart !

That was it! I had the wrong sha. I deleted and redownloaded and it worked perfectly. You are brilliant. Thanks for this great project. Closing this.

[jart]

You're very welcome. Enjoy using the llamafile api. There's also new shell scriptable tutorials in the readme you may enjoy. Please don't hesitate to reach out to us again if you have feedback, need support, or want to cast your vote on feature request issues.