Correct format for tiny-llama

[krecicki]

I have gone through https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/llama_chat_format.py#L399

For tiny-llama-1.1b-v1.0Chat openbuddy works good. However in this video https://www.youtube.com/watch?v=UIuUj3yb640 he uses the format <|system|>\nYou are an helpful assistant.\n<|user|>\write a text about upx.\n<|assistant|> and it performs way better.

How can I make a new format to make this work for tiny-llama? I have taken openbuddy and made a version for tiny-llama. However, I updated the llama_chat_format.py file and Invalid chat handler error occurs. I cannot seem to figure out what to do here to get it to use my custom code?

@register_chat_format("openbuddy")
def format_openbuddy(
    messages: List[llama_types.ChatCompletionRequestMessage],
    **kwargs: Any,
) -> ChatFormatterResponse:
    _system_message = """You are a helpful, respectful and honest INTP-T AI Assistant named Buddy. You are talking to a human User.
Always answer as helpfully and logically as possible, while being safe. Your answers should not include any harmful, political, religious, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
You can speak fluently in many languages, for example: English, Chinese.
You cannot access the internet, but you have vast knowledge, cutoff: 2021-09.
You are trained by OpenBuddy team, (https://openbuddy.ai, https://github.com/OpenBuddy/OpenBuddy), you are based on LLaMA and Falcon transformers model, not related to GPT or OpenAI.

"""
    _roles = dict(user="User", assistant="Assistant")
    _sep = "\n"
    system_message = _system_message
    _messages = _map_roles(messages, _roles)
    _messages.append((_roles["assistant"], None))
    _prompt = _format_add_colon_single(system_message, _messages, _sep)
    return ChatFormatterResponse(prompt=_prompt)
"""

So far I have this update, am I missing anything here..

  @register_chat_format("tiny-llama")
  def format_tiny_llama_custom(
      messages: List[llama_types.ChatCompletionRequestMessage],
      **kwargs: Any,
  ) -> ChatFormatterResponse:
      _system_message = "You are a helpful assistant."
      _roles = dict(user="<|user|>", assistant="<|assistant|>")
      _sep = "\\n"

    system_message = f"{_roles['system']}\\n{_system_message}</s>\\n"
    _messages = map_roles(messages, _roles)
    _messages.append((_roles["assistant"], None))

    _prompt = system_message + _sep.join([
        f"{role}\\{message}</s>\\n" if message else ""
        for role, message in _messages
    ])

    return ChatFormatterResponse(prompt=_prompt)

Strange enough OpenBuddy works better than any of the others. I even made this custom one. It doesn't work as well. What is going on here?

[abetlen]

@krecicki can you link the huggingface for the model you're using? I thnk there may actually be a bug though in the zephyr chat format.

[krecicki]

It is a tiny-llama model I merged with a PEFT LoRA I made locally its not on HF -- OpenBuddy works great for some reason?