Support for tool/function calling

[christianliebel]

I would like to request the addition of tool/function calling functionality to the Prompt API. This feature is available in some models and allows the model to invoke specific actions using a well-defined contract, typically in JSON format. This functionality is beneficial for various use cases that require outputs of a specific structure.

Examples:

• https://docs.retellai.com/retell-llm/tool-calling
• https://docs.mistral.ai/capabilities/function_calling/
• https://platform.openai.com/docs/guides/function-calling

[KenjiBaheux]

Thank you for the suggestion to add tool/function calling functionality.

To assess the feasibility of this feature, we would appreciate it if folks could provide more details on the typical context size required for defining the functions needed in actual use cases. This information will help us understand the potential impact on performance and resource requirements, especially for an on-device context.

[CakeCrusher]

Function calling at this size of model is not practical. Although constrained generation would be a practical intermediate solution for this.
You can delegate constrained generation to the user by giving them access to output logits.

This is the most practical approach IMO

[ChristianWeyer]

Function Calling may indeed be too heavy for those models.
However, models fine-tuned for JSON data extraction would be really helpful. Then we can use approved patterns that are e.g. implemented in Instructor (https://js.useinstructor.com/)