feat: runtime configuration of vision-capable models #5919
Conversation
…ies to models through env var NEXT_PUBLIC_VISION_MODELS.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
@Yiming3 is attempting to deploy a commit to the NextChat Team on Vercel. A member of the Team first needs to authorize it. |
WalkthroughThe changes in this pull request introduce two new constants in Changes
Possibly related PRs
Suggested reviewers
Tip CodeRabbit's docstrings feature is now available as part of our Early Access Program! Simply use the command Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (1)
app/utils.ts (1)
263-264: Add Unit Tests forisVisionModelFunctionTo ensure that the
isVisionModelfunction correctly identifies vision models, consider adding unit tests that cover various model names, including edge cases and potential false positives or negatives.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (2)
app/constant.ts(1 hunks)app/utils.ts(2 hunks)
🔇 Additional comments (1)
app/constant.ts (1)
307-308: Ensure EXCLUDE_VISION_MODEL_REGEXES Accurately Excludes Intended Models
Verify that /claude-3-5-haiku-20241022/ precisely matches and excludes only the specific model "claude-3-5-haiku-20241022" without affecting other models named similarly.
Apply this diff to clarify the intent:
export const EXCLUDE_VISION_MODEL_REGEXES = [
- /claude-3-5-haiku-20241022/
+ /^claude-3-5-haiku-20241022$/
];This change ensures that only the exact model name is excluded.
app/constant.ts
Outdated
| export const VISION_MODEL_REGEXES = [ | ||
| /vision/, | ||
| /gpt-4o/, | ||
| /claude-3/, | ||
| /gemini-1\.5/, | ||
| /gemini-exp/, | ||
| /learnlm/, | ||
| /qwen-vl/, | ||
| /qwen2-vl/, | ||
| /gpt-4-turbo(?!.*preview)/, // Matches "gpt-4-turbo" but not "gpt-4-turbo-preview" | ||
| /^dall-e-3$/, // Matches exactly "dall-e-3" | ||
| ]; |
There was a problem hiding this comment.
💡 Codebase verification
Let me generate the final response based on the test results.
Regex Patterns in VISION_MODEL_REGEXES Need Adjustments
The test results reveal some issues with the regex patterns:
- The
/gpt-4-turbo(?!.*preview)/pattern is not working as intended in bash regex - it's not matching "gpt-4-turbo" as expected - The
/gemini-1\.5/pattern works correctly, matching "gemini-1.5-pro" - The
/^dall-e-3$/pattern works as expected, matching exactly "dall-e-3" but not "dall-e-3-beta"
Consider:
- Using a simpler negative pattern for gpt-4-turbo, like
^gpt-4-turbo$to match exactly "gpt-4-turbo" - Adding test cases for other patterns (vision, gpt-4o, claude-3, etc.) to ensure they match their intended targets
🔗 Analysis chain
Verify Correctness of VISION_MODEL_REGEXES Patterns
Ensure that each regex in VISION_MODEL_REGEXES accurately matches the intended vision models without unintended matches. Pay special attention to:
/gemini-1\.5/: Confirm that the backslash is correctly escaping the dot to matchgemini-1.5./gpt-4-turbo(?!.*preview)/: Ensure that this pattern matches"gpt-4-turbo"but not"gpt-4-turbo-preview".
Run the following script to test the regex patterns against a list of model names:
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Test VISION_MODEL_REGEXES against existing model names
# Define model names for testing
model_names=(
"gpt-4-turbo"
"gpt-4-turbo-preview"
"gemini-1.5-pro"
"gemini-1x5-pro"
"dall-e-3"
"dall-e-3-beta"
)
# Define regex patterns
regex_patterns=(
"vision"
"gpt-4o"
"claude-3"
"gemini-1\\.5"
"gemini-exp"
"learnlm"
"qwen-vl"
"qwen2-vl"
"gpt-4-turbo(?!.*preview)"
"^dall-e-3$"
)
# Test each model name against each regex
for model in "${model_names[@]}"; do
for pattern in "${regex_patterns[@]}"; do
if [[ $model =~ $pattern ]]; then
echo "Model '$model' matches regex '$pattern'"
fi
done
done
Length of output: 5253
Yiming3
changed the title
|
please add unit test and solve the conflicts |
Thanks for the feedback. Solved merge conflicts and added unit tests. |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
test/vision-model-checker.test.ts (4)
4-13: Consider more robust environment variable handling.While the current setup is good, consider using a more thorough environment cleanup to prevent test pollution.
beforeEach(() => { jest.resetModules(); - process.env = { ...originalEnv }; + process.env = JSON.parse(JSON.stringify(originalEnv)); }); afterEach(() => { - process.env = originalEnv; + process.env = JSON.parse(JSON.stringify(originalEnv)); });
15-32: Consider adding edge cases and more specific test descriptions.While the current test covers various model patterns well, consider:
- Adding edge cases (empty string, null, undefined)
- Using more descriptive test names for each model type
- Adding version-specific test cases for models like gemini-1.5-pro vs gemini-2.0
-test("should identify vision models using regex patterns", () => { +test.each([ + ["gpt-4-vision", "GPT-4 Vision model"], + ["claude-3-opus", "Claude 3 Opus model"], + ["gemini-1.5-pro", "Gemini 1.5 Pro model"], + // ... other models +])("should identify %s as vision model (%s)", (model, description) => { + expect(isVisionModel(model)).toBe(true); +}); + +test("should handle edge cases", () => { + expect(isVisionModel("")).toBe(false); + expect(isVisionModel(null)).toBe(false); + expect(isVisionModel(undefined)).toBe(false); +});
38-49: Enhance non-vision models test coverage.The test covers basic cases but could be improved for better maintainability and coverage.
-test("should not identify non-vision models", () => { - const nonVisionModels = [ - "gpt-3.5-turbo", - "gpt-4-turbo-preview", - "claude-2", - "regular-model", - ]; - - nonVisionModels.forEach((model) => { - expect(isVisionModel(model)).toBe(false); - }); -}); +test.each([ + ["gpt-3.5-turbo", "Standard GPT-3.5"], + ["gpt-4-turbo-preview", "Preview GPT-4"], + ["claude-2", "Standard Claude 2"], + ["gpt-4-0125-preview", "Specific version preview"], + ["regular-model", "Generic model name"], +])("should not identify %s as vision model (%s)", (model, description) => { + expect(isVisionModel(model)).toBe(false); +});
51-66: Add more comprehensive environment variable tests.Consider adding tests for:
- Malformed environment variable values
- Mixed valid and invalid model names
- Whitespace handling
+ test("should handle malformed NEXT_PUBLIC_VISION_MODELS", () => { + // Test malformed values + process.env.NEXT_PUBLIC_VISION_MODELS = "model1,,model2, model3 "; + expect(isVisionModel("model1")).toBe(true); + expect(isVisionModel("model2")).toBe(true); + expect(isVisionModel("model3")).toBe(true); + + // Test invalid format + process.env.NEXT_PUBLIC_VISION_MODELS = "{}"; + expect(isVisionModel("model1")).toBe(false); + });🧰 Tools
🪛 Biome (1.9.4)
[error] 63-63: Avoid the delete operator which can impact performance.
Unsafe fix: Use an undefined assignment instead.
(lint/performance/noDelete)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
test/vision-model-checker.test.ts(1 hunks)
🧰 Additional context used
🪛 Biome (1.9.4)
test/vision-model-checker.test.ts
[error] 63-63: Avoid the delete operator which can impact performance.
Unsafe fix: Use an undefined assignment instead.
(lint/performance/noDelete)
test/vision-model-checker.test.ts
Outdated
| delete process.env.NEXT_PUBLIC_VISION_MODELS; | ||
| expect(isVisionModel("unrelated-model")).toBe(false); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Replace delete operator with undefined assignment.
The delete operator can impact performance as flagged by static analysis.
- delete process.env.NEXT_PUBLIC_VISION_MODELS;
- expect(isVisionModel("unrelated-model")).toBe(false);
+ process.env.NEXT_PUBLIC_VISION_MODELS = undefined;
+ expect(isVisionModel("unrelated-model")).toBe(false);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| delete process.env.NEXT_PUBLIC_VISION_MODELS; | |
| expect(isVisionModel("unrelated-model")).toBe(false); | |
| process.env.NEXT_PUBLIC_VISION_MODELS = undefined; | |
| expect(isVisionModel("unrelated-model")).toBe(false); |
🧰 Tools
🪛 Biome (1.9.4)
[error] 63-63: Avoid the delete operator which can impact performance.
Unsafe fix: Use an undefined assignment instead.
(lint/performance/noDelete)
| test("should exclude specific models", () => { | ||
| expect(isVisionModel("claude-3-5-haiku-20241022")).toBe(false); | ||
| }); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Enhance exclusion testing with more patterns and documentation.
The current test is too specific. Consider:
- Adding more exclusion patterns
- Documenting why certain models are excluded
- Using test.each for better readability
-test("should exclude specific models", () => {
- expect(isVisionModel("claude-3-5-haiku-20241022")).toBe(false);
-});
+test.each([
+ // Models excluded due to specific version constraints
+ ["claude-3-5-haiku-20241022", "Haiku variant without vision capabilities"],
+ // Add more excluded models with reasons
+])("should exclude %s (%s)", (model, reason) => {
+ expect(isVisionModel(model)).toBe(false);
+});Committable suggestion skipped: line range outside the PR's diff.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
app/config/build.ts (1)
43-43: Add JSDoc comment to document the visionModels property.Consider adding documentation to explain the purpose and expected format of this configuration property.
+ /** Comma-separated list of additional vision model identifiers */ visionModels: process.env.VISION_MODELS || "",test/vision-model-checker.test.ts (1)
15-32: Enhance test organization using test.each.Consider using test.each for better readability and maintenance:
- test("should identify vision models using regex patterns", () => { - const visionModels = [ - "gpt-4-vision", - "claude-3-opus", - "gemini-1.5-pro", - "gemini-2.0", - "gemini-exp-vision", - "learnlm-vision", - "qwen-vl-max", - "qwen2-vl-max", - "gpt-4-turbo", - "dall-e-3", - ]; - - visionModels.forEach((model) => { - expect(isVisionModel(model)).toBe(true); - }); + test.each([ + ["gpt-4-vision", "OpenAI vision model"], + ["claude-3-opus", "Anthropic vision model"], + ["gemini-1.5-pro", "Google vision model"], + ["gemini-2.0", "Google vision model"], + ["gemini-exp-vision", "Google experimental vision model"], + ["learnlm-vision", "Custom vision model"], + ["qwen-vl-max", "Qwen vision model"], + ["qwen2-vl-max", "Qwen2 vision model"], + ["gpt-4-turbo", "OpenAI turbo model"], + ["dall-e-3", "OpenAI image generation model"], + ])("should identify %s as vision model (%s)", (model, description) => { + expect(isVisionModel(model)).toBe(true); + });
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
app/config/build.ts(1 hunks)app/constant.ts(1 hunks)app/utils.ts(2 hunks)test/vision-model-checker.test.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- app/utils.ts
- app/constant.ts
🧰 Additional context used
🪛 Biome (1.9.4)
test/vision-model-checker.test.ts
[error] 63-63: Avoid the delete operator which can impact performance.
Unsafe fix: Use an undefined assignment instead.
(lint/performance/noDelete)
🔇 Additional comments (1)
test/vision-model-checker.test.ts (1)
63-63: Replace delete operator with undefined assignment.
The delete operator can impact performance as flagged by static analysis.
- delete process.env.VISION_MODELS;
+ process.env.VISION_MODELS = undefined;🧰 Tools
🪛 Biome (1.9.4)
[error] 63-63: Avoid the delete operator which can impact performance.
Unsafe fix: Use an undefined assignment instead.
(lint/performance/noDelete)
| @@ -40,6 +40,7 @@ export const getBuildConfig = () => { | |||
| buildMode, | |||
| isApp, | |||
| template: process.env.DEFAULT_INPUT_TEMPLATE ?? DEFAULT_INPUT_TEMPLATE, | |||
| visionModels: process.env.VISION_MODELS || "", | |||
There was a problem hiding this comment.
Environment variable name mismatch with PR objectives.
The PR objectives mention NEXT_PUBLIC_VISION_MODELS, but the code uses VISION_MODELS. This inconsistency should be resolved.
- visionModels: process.env.VISION_MODELS || "",
+ visionModels: process.env.NEXT_PUBLIC_VISION_MODELS || "",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| visionModels: process.env.VISION_MODELS || "", | |
| visionModels: process.env.NEXT_PUBLIC_VISION_MODELS || "", |
| import { isVisionModel } from "../app/utils"; | ||
|
|
||
| describe("isVisionModel", () => { | ||
| const originalEnv = process.env; | ||
|
|
||
| beforeEach(() => { | ||
| jest.resetModules(); | ||
| process.env = { ...originalEnv }; | ||
| }); | ||
|
|
||
| afterEach(() => { | ||
| process.env = originalEnv; | ||
| }); | ||
|
|
||
| test("should identify vision models using regex patterns", () => { | ||
| const visionModels = [ | ||
| "gpt-4-vision", | ||
| "claude-3-opus", | ||
| "gemini-1.5-pro", | ||
| "gemini-2.0", | ||
| "gemini-exp-vision", | ||
| "learnlm-vision", | ||
| "qwen-vl-max", | ||
| "qwen2-vl-max", | ||
| "gpt-4-turbo", | ||
| "dall-e-3", | ||
| ]; | ||
|
|
||
| visionModels.forEach((model) => { | ||
| expect(isVisionModel(model)).toBe(true); | ||
| }); | ||
| }); | ||
|
|
||
| test("should exclude specific models", () => { | ||
| expect(isVisionModel("claude-3-5-haiku-20241022")).toBe(false); | ||
| }); | ||
|
|
||
| test("should not identify non-vision models", () => { | ||
| const nonVisionModels = [ | ||
| "gpt-3.5-turbo", | ||
| "gpt-4-turbo-preview", | ||
| "claude-2", | ||
| "regular-model", | ||
| ]; | ||
|
|
||
| nonVisionModels.forEach((model) => { | ||
| expect(isVisionModel(model)).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| test("should identify models from VISION_MODELS env var", () => { | ||
| process.env.VISION_MODELS = "custom-vision-model,another-vision-model"; | ||
|
|
||
| expect(isVisionModel("custom-vision-model")).toBe(true); | ||
| expect(isVisionModel("another-vision-model")).toBe(true); | ||
| expect(isVisionModel("unrelated-model")).toBe(false); | ||
| }); | ||
|
|
||
| test("should handle empty or missing VISION_MODELS", () => { | ||
| process.env.VISION_MODELS = ""; | ||
| expect(isVisionModel("unrelated-model")).toBe(false); | ||
|
|
||
| delete process.env.VISION_MODELS; | ||
| expect(isVisionModel("unrelated-model")).toBe(false); | ||
| expect(isVisionModel("gpt-4-vision")).toBe(true); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add missing edge case tests.
Consider adding tests for the following edge cases:
- Model names with special characters
- Case sensitivity handling
- Empty string input
- Null/undefined input
+ test.each([
+ ["", "empty string"],
+ [null, "null input"],
+ [undefined, "undefined input"],
+ ["GPT-4-VISION", "uppercase input"],
+ ["gpt-4-vision!", "special characters"],
+ ])("should handle edge case: %s (%s)", (input, description) => {
+ expect(() => isVisionModel(input)).not.toThrow();
+ });Committable suggestion skipped: line range outside the PR's diff.
🧰 Tools
🪛 Biome (1.9.4)
[error] 63-63: Avoid the delete operator which can impact performance.
Unsafe fix: Use an undefined assignment instead.
(lint/performance/noDelete)
|
更新一下所有README文档 给一个使用示例 |
Update all README documents and give a usage example |
|
Your build has completed! |
好的，已经更新 |
OK, updated |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
README.md (1)
358-364: Enhance documentation clarity for VISION_MODELS.The documentation would benefit from these improvements:
- Add a period after "etc" in the example description
- Clarify the interaction between explicit listings and pattern matching
### `VISION_MODELS` (optional) > Default: Empty > Example: `gpt-4-vision,claude-3-opus,my-custom-model` means add vision capabilities to these models in addition to the default pattern matches (which detect models containing keywords like "vision", "claude-3", "gemini-1.5", etc.). -Add additional models to have vision capabilities, beyond the default pattern matching. Multiple models should be separated by commas. +Specify additional models to enable vision capabilities beyond the default pattern matching. The default patterns automatically detect models containing keywords like "vision", "claude-3", and "gemini-1.5". Use this variable to explicitly add more models, separating multiple entries with commas.🧰 Tools
🪛 LanguageTool
[style] ~361-~361: In American English, abbreviations like “etc.” require a period.
Context: ...ike "vision", "claude-3", "gemini-1.5", etc). Add additional models to have vision...(ETC_PERIOD)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
README.md(1 hunks)README_CN.md(1 hunks)README_JA.md(1 hunks)app/utils.ts(2 hunks)
🧰 Additional context used
🪛 LanguageTool
README.md
[style] ~361-~361: In American English, abbreviations like “etc.” require a period.
Context: ...ike "vision", "claude-3", "gemini-1.5", etc). Add additional models to have vision...
(ETC_PERIOD)
🔇 Additional comments (4)
app/utils.ts (2)
8-8: LGTM: Clean import of vision model regex constants
The import statement is well-structured and follows the project's import style.
257-266: Verify the model detection logic
The implementation combines environment-based configuration with regex pattern matching, providing flexibility in model detection. However, let's verify the behavior when clientConfig is undefined or when visionModels is empty.
Run the following script to check the usage:
✅ Verification successful
Let me generate one more verification script to check the implementation of getClientConfig and its usage:
This will help us understand how getClientConfig handles undefined cases and how the visionModels property is defined in the type system.
Vision model detection logic is properly implemented and handles edge cases
The implementation is robust and handles undefined/empty cases correctly:
- When
clientConfigis undefined (empty JSON from queryMeta), the code gracefully falls back to regex-based detection - When
visionModelsis empty, the logic correctly falls back to regex patterns - Test cases explicitly verify empty/missing scenarios
- The regex patterns in
VISION_MODEL_REGEXESandEXCLUDE_VISION_MODEL_REGEXESprovide a reliable fallback mechanism - The implementation is consistently used across different platform integrations (OpenAI, Anthropic, Google, Tencent)
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify the vision model detection implementation
# Test: Search for all usages of isVisionModel and the patterns defined in constants
# Expect: Consistent usage of the function and well-defined patterns
# Check the constant definitions
echo "Checking vision model regex patterns..."
rg "VISION_MODEL_REGEXES.*=.*\\[" -A 5
# Check all usages of isVisionModel
echo "Checking isVisionModel usage..."
rg "isVisionModel\\(" -A 2
# Check the client config implementation
echo "Checking client config implementation..."
rg "visionModels.*=" -A 2
Length of output: 3968
Script:
#!/bin/bash
# Check getClientConfig implementation and usage
echo "Checking getClientConfig implementation..."
ast-grep --pattern 'function getClientConfig() {
$$$
}'
echo "Checking ClientConfig type definition..."
rg "type ClientConfig.*{" -A 10
echo "Checking environment variable usage..."
rg "VISION_MODELS.*=" -A 2
Length of output: 1661
README_CN.md (1)
238-244: LGTM: Well-documented environment variable
The documentation for VISION_MODELS is clear, comprehensive, and includes:
- Default value
- Usage examples
- Clear explanation of how it complements the default pattern matching
README_JA.md (1)
220-226: LGTM: Consistent documentation in Japanese
The Japanese documentation for VISION_MODELS maintains consistency with other language versions while providing clear, localized explanations.
|
Yo! good job 👍 I like for You and alias to your provide and support project @pleasestartconnectGeminiGeneralbot♊️update🩷 |
|
💻 变更类型 | Change Type
🔀 变更说明 | Description of Change
This PR provides runtime configuration of vision-capable models suggested from #5843
VISION_MODELSto allow runtime configuration of vision-capable modelsKey Changes:
VISION_MODEL_REGEXESandEXCLUDE_VISION_MODEL_REGEXESinconstant.tsisVisionModel()to use regex patterns instead of string includesVISION_MODELSenvironment variable📝 补充信息 | Additional Information
Summary by CodeRabbit
Summary by CodeRabbit
New Features
VISION_MODELSfor user-defined vision capabilities.Bug Fixes
Documentation
README.md,README_CN.md, andREADME_JA.mdto include the newVISION_MODELSenvironment variable with examples and clarifications.