You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bedrock now supports the use of prompt routers to choose between multiple models based on the input. This can reduce cost and latency for simple questions while still using more powerful models for more complex inputs.
When a prompt router is used, the Bedrock API will include information about the selected model alongside other response metadata, either in response['trace']['promptRouter'] for invocation with the Converse endpoint or as part of the final metadata event in event['metadata']['trace']['promptRouter'] when using the ConverseStream endpoint.
It would be helpful to return this metadata to the caller when accessing routed LLMs through landchain-aws, since the model choice determines the cost of execution of the query and this may be of importance to the caller.
The text was updated successfully, but these errors were encountered:
Bedrock now supports the use of prompt routers to choose between multiple models based on the input. This can reduce cost and latency for simple questions while still using more powerful models for more complex inputs.
When a prompt router is used, the Bedrock API will include information about the selected model alongside other response metadata, either in
response['trace']['promptRouter']
for invocation with theConverse
endpoint or as part of the final metadata event inevent['metadata']['trace']['promptRouter']
when using theConverseStream
endpoint.It would be helpful to return this metadata to the caller when accessing routed LLMs through
landchain-aws
, since the model choice determines the cost of execution of the query and this may be of importance to the caller.The text was updated successfully, but these errors were encountered: