Continue: Write comments for this code produces math questions

[max-wittig]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: macOS 14.4
- Continue: v0.9.112 (pre-release)
- IDE: VSCode 1.88.0

Description

When selecting code and asking continue.dev to comment on it, it will produce math equations instead

Using OpenAI configured like this:

    {
      "title": "Mistral",
      "provider": "openai",
      "model": "mistral-7b-instruct",
      "apiKey": "your-api-key",
      "apiBase": "https://your-base-url.com/llm/v1"
    }

Using Mistal with Ollama the extension produces the expected result, but it seems using the OpenAI provider only empty strings get send, which will lead Mistral to dream about math.

async_llm_engine.py:436] Received request cmpl-6def2b5f4c4d4d61952dbf6a82262f4d: prompt: '<s>',

To reproduce

1. Select this function and Press CMD + P

   def get_attribute(attributes: list[dict[str, str]], key: str) -> Optional[str]:
      for attribute in attributes:
          if attribute.get("key") == key:
              return attribute.get("value")
      return None

2. Write Continue: Write comments for this code
3. See the result below

Log output

async_llm_engine.py:436] Received request cmpl-6def2b5f4c4d4d61952dbf6a82262f4d: prompt: '<s>',

[sestinj]

@max-wittig Probably there's a very clear prompt formatting problem somewhere.

The most transparent way to debug this would be to open the Output panel next to the Terminal in VS Code and then select "Continue - LLM Prompts/Completions" in the dropdown. This will display the exact prompt sent to LiteLLM so that we can determine if the problem comes before the request.

Continue should be attempting to format the prompt on its own, but if LiteLLM's README is up to date, then they might not support the /completions endpoint, in which case they are probably double wrapping the prompt.

I'll try setting LiteLLM on my own tomorrow as well to investigate.

Separately, I don't know if you're just trying many models, but I can share a few recommendations if you are interested.

[fgreinacher]

(I'm from Max' team)

@sestinj This is the output when trying to create comments for a simple hello world function:

Settings:
contextLength: 4096
model: mistral-7b-instruct
maxTokens: 1024
raw: true

############################################

[INST] ```python
def hello_world():
  print("Hello world")



```

Please rewrite the entire code block above in order to satisfy the following request: "Write comments for this code. Do not change anything about the code itself.". [/INST]Sure! Here's the entire rewritten code block:
```python
==========================================================================
==========================================================================
Completion:

Q: Let c = 106.1 + -106. Let o = c + -0.1. Let r = 0.1 - o. What is the closest to r in 2, -1/3, -1?
A: 2

Not sure if it's correctly formatted. Looks like Continue.dev is doing some templating on its own. Usually this is done at the server-side I think.

[fgreinacher]

This seems to have broken recently.

It is working with the same configuration in version 0.8.22 where the log output looks like this:

[INST] You are a helpful code assistant. Your task is to rewrite the following code with these instructions: "Original request: Write comments for this code. Do not change anything about the code itself.
Updated request: Write comments for this code. Do not change anything about the code itself."
```python
def hello_world():
  print("Hello world")

```

Just rewrite the code without explanations: [/INST]
```python==========================================================================
==========================================================================
Completion:

 ```python
# This function prints "Hello world" when called
def hello_world():
  # The print statement outputs the given message to the console
  print("Hello world")

[bufferoverflow]

Seems to be caused by #1029

[sestinj]

@bufferoverflow That is the PR that changed the prompt, but it's not clear that the prompt alone is the problem. Both prompts take the same approach of pre-empting the model. So it seems something is amiss in the request itself.

I ran a few tests. Tried Mistral-7b-Instruct with each of the following pairs: [LiteLLM, TogetherAI], [LiteLLM, Ollama], [Direct, TogetherAI], [Direct, Ollama]. All of these worked fine, except for the first, in which LiteLLM just threw an error that seems unrelated to all of this.

This leads to me to think that the LiteLLM+vLLM combination is causing the problem. Since the log shared above by Max shows that vLLM got an empty prompt in the request, my guess would be that when LiteLLM is setup with vLLM like this it does something to re-format the prompt.

Still working on figuring out figuring out what exactly is going on, as I don't want to put blame on another tool until I've verified it's not us. I'll hopefully have more to share soon after looking through the LiteLLM code, and if it's something there I'll aim to make whatever PR is necessary.

[bufferoverflow]

@sestinj I found the problem 🎉 It works with the following hack, which forces the use of the /v1/chat/completions endpoint instead of the /v1/completions endpoint

diff --git a/core/llm/index.ts b/core/llm/index.ts
index ed920eaa..a9e03095 100644
--- a/core/llm/index.ts
+++ b/core/llm/index.ts
@@ -58,7 +58,7 @@ export abstract class BaseLLM implements ILLM {
         return false;
       }
     }
-    return true;
+    return false;
   }
 
   supportsPrefill(): boolean {

[bufferoverflow]

@sestinj not sure how you envision this. What about extending https://github.com/continuedev/continue/blob/preview/schema/json/ModelDescription.json with EndpointType and adding an enum with chat/completion, completion, so people could set the endpoint to use?

[fgreinacher]

This leads to me to think that the LiteLLM+vLLM combination is causing the problem. Since the log shared above by Max shows that vLLM got an empty prompt in the request, my guess would be that when LiteLLM is setup with vLLM like this it does something to re-format the prompt.

We do indeed have a custom chat template configured for vLLM via--chat-template /config/mistral-mixtral-chat-template.jinja:

{% if messages[0]['role'] == 'system' -%}
    {% set loop_messages = messages[1:] -%}
    {% set system_message = messages[0]['content'].strip() + ' ' -%}
{% else -%}
    {% set loop_messages = messages -%}
    {% set system_message = '' -%}
{% endif -%}
{{ bos_token -}}
{% for message in loop_messages -%}
    {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
    {% endif -%}
    {% if loop.index0 == 0 -%}
        {% set content = system_message + message['content'] -%}
    {% else -%}
        {% set content = message['content'] -%}
    {% endif -%}
    {% if message['role'] == 'user' -%}
        {{ '[INST] ' + content.strip() + ' [/INST]' -}}
    {% elif message['role'] == 'assistant' -%}
        {{ content.strip() + eos_token -}}
    {% endif -%}
{% endfor -%}

We have adapted this from https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/blob/main/tokenizer_config.json#L42 so that we can use system prompts.

[fgreinacher]

For completeness sake: The new prompt does not work with the legacy /completions endpoint:

It does work well with the /chat/completions endpoint though:

So as @bufferoverflow suggested above it would probably be best to let users select which endpoint to use.

[sestinj]

@bufferoverflow @fgreinacher this sounds like a good solution to me. I'll make an update now.

Is my understanding correct that what happens here is that vLLM applies the chat template on top of whatever is sent through the /completions endpoint?

[bufferoverflow]

Is my understanding correct that what happens here is that vLLM applies the chat template on top of whatever is sent through the /completions endpoint?

yes, it seems so

@sestinj I found a simpler solution, see #1151

[sestinj]

@bufferoverflow yes this looks perfect!

[sestinj]

PR is now merged: #1151, and release is coming out in 20min

To solve, you now can set "useLegacyCompletionsEndpoint": false in your model in config.json. I'll wait to close the issue until it's verified that this works

[max-wittig]

Works pretty well with 0.9.113 and "useLegacyCompletionsEndpoint": false.

Thanks for your efforts!