Token usage stats from responses should be returned to the client. #618
Comments
|
@SOE-YoungS , great idea, we will take a look at getting this added. |
5 tasks
lemillermicrosoft
pushed a commit
that referenced
this issue
Jun 7, 2023
… etc) (#1181) ### Motivation and Context Common feature request and missing capability to acquire the actual details from the models we are integrating. Resolves #802 Resolves #618 Resolves Partially #351 Getting the results alternative to #693 ### Description This change introduces a new Property in `object? SKContext.LastPromptResults` which will contain a list of each result detail for a given prompt. The Connector namespace will also provide a Extension methods that converts this object in the actual class that you will be using to serialize or get the detailed information about the Model result. Normal suggested usage ``` var textResult = await excuseFunction.InvokeAsync("I missed the F1 final race"); ``` Getting model result as json: ``` var modelResultJson = JsonSerializer.Serialize(textResult.LastPromptResults); ``` Getting model result as a traversable object (using a connector extension): ``` var modelResultJson = textResult.GetOpenAILastPromptResult()?.Usage.TotalTokens; ```
shawncal
pushed a commit
to shawncal/semantic-kernel
that referenced
this issue
Jul 6, 2023
… etc) (microsoft#1181) ### Motivation and Context Common feature request and missing capability to acquire the actual details from the models we are integrating. Resolves microsoft#802 Resolves microsoft#618 Resolves Partially microsoft#351 Getting the results alternative to microsoft#693 ### Description This change introduces a new Property in `object? SKContext.LastPromptResults` which will contain a list of each result detail for a given prompt. The Connector namespace will also provide a Extension methods that converts this object in the actual class that you will be using to serialize or get the detailed information about the Model result. Normal suggested usage ``` var textResult = await excuseFunction.InvokeAsync("I missed the F1 final race"); ``` Getting model result as json: ``` var modelResultJson = JsonSerializer.Serialize(textResult.LastPromptResults); ``` Getting model result as a traversable object (using a connector extension): ``` var modelResultJson = textResult.GetOpenAILastPromptResult()?.Usage.TotalTokens; ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
OpenAI / Azure OpenAI both return token stats with responses.
These are useful for tracking usage etc... on a case by case basis for things like "inter-departmental billing" etc...
Currently SK only supports returning the completion (string), meaning token usage is not accessible to the end user & unavailable for future evaluation / tracking.
SK, should return an object containing token stats & completion string.
Example use case(s):
The text was updated successfully, but these errors were encountered: