[Feature Request] Instruct Templating

[Juxsta]

Would it be possible to supply an option for using an instruction template using Jinja? This would significantly improve the performance of instruct models.

[Juxsta]

For an MVP a possible solution could be to allow a custom prompt parameter where you can pass in a template and variables similar to the way the current aiuser prompt setting works

[zhaobenny]

From what I understand, the Jinja template is to mostly translate to a format more suitable for instruct models?

Does existing OpenAI proxy servers not do that type of translation for instruct models already?

[Juxsta]

I don't believe this is a supported feature since you would need to be able to pull variables from the prompt to support it. This would be possible if the prompt was constructed in such a way the templating could be done through the proxy, but at that point, the templating may as well have been done on the client.

[Juxsta]

Just for clarity the feature request I'm proposing is being able to make a setting such as

template: -| 
 <s> [INST] You are {bot_name}. You are responding in a discord chat room in the {discord_server} in the {discord_channel} channel. 
 Your personality traits are:
 {Persona}
 [/INST] 
 </s>
 Message History: 
 {Message History}
 [INST] Respond as {bot_name} [/INST]
 {bot_name}:

[zhaobenny]

Ah, I understand now.

Currently, the cog is not greatly compatible with instruct models (mostly because I found chat models better to chat with and easier to prompt with). I'll see if I can add this feature whenever I get to improving compatibility with instruct models.