Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Openai embedding fix to support jina-embeddings-v2 #4642

Merged
merged 11 commits into from
Nov 18, 2023
Merged

Openai embedding fix to support jina-embeddings-v2 #4642

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
df8288c
update default max_new_tokens to 4096
wizd Nov 11, 2023
454fcf3
Merge pull request #4579 from oobabooga/dev
oobabooga Nov 13, 2023
2337aeb
Merge pull request #4606 from oobabooga/dev
oobabooga Nov 15, 2023
8a2af87
Merge pull request #4608 from oobabooga/dev
oobabooga Nov 15, 2023
0ee8d2b
Merge pull request #4627 from oobabooga/dev
oobabooga Nov 17, 2023
f889302
Merge pull request #4628 from oobabooga/dev
oobabooga Nov 17, 2023
3146124
Merge pull request #4632 from oobabooga/dev
oobabooga Nov 17, 2023
204d967
Merge branch 'oobabooga:main' into openai_embedding_fix
wizd Nov 17, 2023
828f6a7
add support to load embeding model from local path to device (like cuda)
wizd Nov 18, 2023
d3fb920
check model name before loading
wizd Nov 18, 2023
b3b5202
Small changes
oobabooga Nov 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion extensions/openai/completions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ def chat_completions_common(body: dict, is_legacy: bool = False, stream=False) -

max_tokens = generate_params['max_new_tokens']
if max_tokens in [None, 0]:
generate_params['max_new_tokens'] = 4096
generate_params['max_new_tokens'] = 200
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting this high isn't necessary, as the auto_max_new_tokens fills the context. The 200 reference value is used when the context is fully used to remove old messages.

generate_params['auto_max_new_tokens'] = True

requested_model = generate_params.pop('model')
Expand Down
6 changes: 3 additions & 3 deletions extensions/openai/embeddings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import os

import numpy as np
from transformers import AutoModel

from extensions.openai.errors import ServiceUnavailableError
from extensions.openai.utils import debug_msg, float_list_to_base64
from modules.logging_colors import logger
from transformers import AutoModel

embeddings_params_initialized = False

Expand Down Expand Up @@ -43,11 +43,11 @@ def load_embedding_model(model: str):
try:
print(f"Try embedding model: {model} on {embeddings_device}")
if 'jina-embeddings' in model:
embeddings_model = AutoModel.from_pretrained(model, trust_remote_code=True) # trust_remote_code is needed to use the encode method
# Move the model to the device
embeddings_model = AutoModel.from_pretrained(model, trust_remote_code=True) # trust_remote_code is needed to use the encode method
embeddings_model = embeddings_model.to(embeddings_device)
else:
embeddings_model = SentenceTransformer(model, device=embeddings_device)

print(f"Loaded embedding model: {model}")
except Exception as e:
embeddings_model = None
Expand Down