Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for custom/provisioned models in Bedrock #922

Merged
merged 6 commits into from
Jul 31, 2024

Conversation

dlqqq
Copy link
Member

@dlqqq dlqqq commented Jul 30, 2024

Description

Adds support for custom/provisioned models on Amazon Bedrock. To do so:

  • Select Amazon Bedrock (custom/provisioned) :: * in the chat model dropdown.
  • Enter the ARN of the custom/provisioned model as the model ID.
  • Enter the provider of the custom/provisioned model. This is the provider of the foundation model that it is based on.
    • Providers are prefixed to each foundation model ID in Amazon Bedrock. So for example, if your model is based off anthropic.claude-v2, then you should enter anthropic (all lowercase) as the provider.
    • The current list of available providers for chat are: ai21, amazon, anthropic, cohere, meta, and mistral.

Demo

Screenshot 2024-07-30 at 3 15 12 PM

@dlqqq dlqqq added the enhancement New feature or request label Jul 30, 2024
Copy link
Collaborator

@srdas srdas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this addition. Will this make the drop down for all Bedrock models redundant? Do we need PR #923 ?

Tested the branch. For example:
image
... followed by issuing a query:
image

Suggested improvements:

  1. As can be seen in the example, you can enter the ARN or the model id. Say that in the string below the Model ID entry box?
  2. State the full form of ARN in parenthesis (Amazon Resource Name) and point to this page for reference: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html
  3. The model provider asked for in the UI is always lower case, whereas in the model-ids table it is upper case. This may be confusing to the user, can we make this more clear if lower case is required?
  4. Add this feature/enhancement to the documentation?

@dlqqq
Copy link
Member Author

dlqqq commented Jul 31, 2024

@srdas

Will this make the drop down for all Bedrock models redundant?

No, it does not. We should continue to allow users to select Bedrock models by picking one from a list of options rather than force them to type out the model ID manually.

  1. While you can use BedrockCustomProvider to call foundation models, this is an implementation detail that should not be relied on. It is possible that in the future, custom/provisioned models are only accessible by a ChatModel class different from ChatBedrock. Users who want to call foundation models should just select one provided by BedrockChatProvider.
  2. Sure.
  3. Sure.
  4. Sure.

@dlqqq
Copy link
Member Author

dlqqq commented Jul 31, 2024

@srdas I've added a mention of bedrock-custom in the user documentation. I noticed however that we do not have a section on Amazon Bedrock at all, so I've opened this issue for follow-up. Do you think you can help with this? #924

Copy link
Collaborator

@srdas srdas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dlqqq for making the suggested changes. I have tested the new code with Model IDs but have not been able to test it with an ARN as yet (though my env/installation is updated with langchain-aws 0.1.13), which is needed. Am approving the code and documentation changes so far modulo testing for the ARN.

I have also assigned the new issue #924 to myself to expand the documentation.

@dlqqq
Copy link
Member Author

dlqqq commented Jul 31, 2024

@srdas Thank you!

Note to other readers: this PR requires langchain-aws>=0.1.13.

@dlqqq dlqqq merged commit b953568 into jupyterlab:main Jul 31, 2024
8 checks passed
@dlqqq dlqqq deleted the bedrock-custom branch July 31, 2024 22:35
Marchlak pushed a commit to Marchlak/jupyter-ai that referenced this pull request Oct 28, 2024
* add BedrockCustomProvider

* pre-commit

* format help text

* show model ID field

* edit BedrockCustomProvider help text

* add mention of BedrockCustom in docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants