feat: support self-hosted embedding service via BentoML #324

parano · 2023-08-10T09:53:53Z

This PR adds an option for RealChar to use a self-hosted embedding service powered by SentenceBert and BentoML.

By default, this integration uses the docker image published here

The default model that comes with the docker image is all-MiniLM-L6-v2. RealChar users may customize it to use a different text embedding model based on their needs. Check out the source code for the embedding service here: https://github.com/bentoml/sentence-embedding-bento,

TODO:

Test the integration end-to-end

.env.example

cli.py

Shaunwei · 2023-08-11T07:52:17Z

BentoML does not support loading pydantic models from URLs; output will be a normal dictionary.

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/main.py", line 410, in main
    run(
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/main.py", line 578, in run
    server.run()
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/config.py", line 473, in load
    self.loaded_app = import_from_string(self.app)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/main.py", line 37, in <module>
    CatalogManager.initialize(overwrite=overwrite_chroma)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/utils.py", line 65, in initialize
    cls._instances[cls] = cls(*args, **kwargs)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 26, in __init__
    self.db = get_chroma()
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/database/chroma.py", line 31, in get_chroma
    chroma = Chroma(
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 128, in __init__
    embedding_function=self._embedding_function.embed_documents
AttributeError: 'function' object has no attribute 'embed_documents'

Looks like the embed function is not compatible

aarnphm · 2023-08-11T11:35:32Z

BentoML does not support loading pydantic models from URLs; output will be a normal dictionary.

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/main.py", line 410, in main
    run(
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/main.py", line 578, in run
    server.run()
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/config.py", line 473, in load
    self.loaded_app = import_from_string(self.app)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/main.py", line 37, in <module>
    CatalogManager.initialize(overwrite=overwrite_chroma)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/utils.py", line 65, in initialize
    cls._instances[cls] = cls(*args, **kwargs)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 26, in __init__
    self.db = get_chroma()
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/database/chroma.py", line 31, in get_chroma
    chroma = Chroma(
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 128, in __init__
    embedding_function=self._embedding_function.embed_documents
AttributeError: 'function' object has no attribute 'embed_documents'

Looks like the embed function is not compatible

The client return back a dictionary, but we can load that dict into a pydantic model if needed. It seems like this _embedding_function here is a callable instead of pydantic model.

@parano

realtime_ai_character/database/chroma.py

Co-authored-by: shaun <[email protected]>

Co-authored-by: Aaron Pham <[email protected]>

aarnphm

LGTM

Shaunwei · 2023-08-12T03:54:56Z

    CatalogManager.initialize(overwrite=overwrite_chroma)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/utils.py", line 65, in initialize
    cls._instances[cls] = cls(*args, **kwargs)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 37, in __init__
    self.load_characters_from_community(overwrite)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 134, in load_characters_from_community
    self.load_data(character_name, directory / 'data')
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 150, in load_data
    self.db.add_documents(docs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 104, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 187, in add_texts
    embeddings = self._embedding_function.embed_documents(texts)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/database/chroma.py", line 18, in embed_documents
    return self.client.encode(texts).tolist()
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/bentoml/_internal/client/__init__.py", line 133, in _sync_call
    return asyncio.run(self._call(inp, _bentoml_api=_bentoml_api, **kwargs))
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
sys:1: RuntimeWarning: coroutine 'HTTPClient._call' was never awaited

I still run into errors

README.md

cli.py

aarnphm

cc @parano. The port inside container has to be 3000

parano · 2023-08-13T02:02:24Z

    CatalogManager.initialize(overwrite=overwrite_chroma)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/utils.py", line 65, in initialize
    cls._instances[cls] = cls(*args, **kwargs)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 37, in __init__
    self.load_characters_from_community(overwrite)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 134, in load_characters_from_community
    self.load_data(character_name, directory / 'data')
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/character_catalog/catalog_manager.py", line 150, in load_data
    self.db.add_documents(docs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 104, in add_documents
    return self.add_texts(texts, metadatas, **kwargs)
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/langchain/vectorstores/chroma.py", line 187, in add_texts
    embeddings = self._embedding_function.embed_documents(texts)
  File "/Users/shaun/Documents/GitHub/Realtime-AI-Character/realtime_ai_character/database/chroma.py", line 18, in embed_documents
    return self.client.encode(texts).tolist()
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/site-packages/bentoml/_internal/client/__init__.py", line 133, in _sync_call
    return asyncio.run(self._call(inp, _bentoml_api=_bentoml_api, **kwargs))
  File "/opt/homebrew/anaconda3/envs/realtime-ai-companion/lib/python3.10/asyncio/runners.py", line 33, in run
    raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop
sys:1: RuntimeWarning: coroutine 'HTTPClient._call' was never awaited

I still run into errors

Thanks @Shaunwei, will look into this

pycui · 2023-08-16T18:33:12Z

Hi - any updates to this PR?

aarnphm · 2023-08-17T20:20:36Z

Hi - any updates to this PR?

Hey you can try out this patch

openllm.patch

parano added 2 commits August 10, 2023 02:49

feat: support self-hosted embedding service via BentoML

406ae6a

Merge branch 'main' into bentoml-text-embedding

0bbca1c

Shaunwei self-requested a review August 10, 2023 18:25

Shaunwei reviewed Aug 10, 2023

View reviewed changes

.env.example Outdated Show resolved Hide resolved

Shaunwei reviewed Aug 10, 2023

View reviewed changes

.env.example Outdated Show resolved Hide resolved

parano added 3 commits August 10, 2023 15:18

fix: chroma embedding func expects list[float]

4ca94cf

chore: avoid using 3000 port

cb637d4

chore: fix ruff warning

23fcb35

Shaunwei reviewed Aug 11, 2023

View reviewed changes

cli.py Outdated Show resolved Hide resolved

aarnphm reviewed Aug 11, 2023

View reviewed changes

realtime_ai_character/database/chroma.py Outdated Show resolved Hide resolved

parano and others added 3 commits August 11, 2023 14:47

Update embedding service port in CLI

af9b556

Co-authored-by: shaun <[email protected]>

Update embed_func protocol to work with Chroma interface

e164bc2

Co-authored-by: Aaron Pham <[email protected]>

Fix embeddings protocol

20285e7

aarnphm approved these changes Aug 11, 2023

View reviewed changes

aarnphm reviewed Aug 12, 2023

View reviewed changes

README.md Show resolved Hide resolved

aarnphm reviewed Aug 12, 2023

View reviewed changes

cli.py Show resolved Hide resolved

aarnphm suggested changes Aug 12, 2023

View reviewed changes

Merge branch 'main' into bentoml-text-embedding

0aa6caf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support self-hosted embedding service via BentoML #324

feat: support self-hosted embedding service via BentoML #324

parano commented Aug 10, 2023

Shaunwei commented Aug 11, 2023 •

edited

Loading

aarnphm commented Aug 11, 2023

aarnphm left a comment

Shaunwei commented Aug 12, 2023

aarnphm left a comment

parano commented Aug 13, 2023

pycui commented Aug 16, 2023

aarnphm commented Aug 17, 2023 •

edited

Loading

feat: support self-hosted embedding service via BentoML #324

Are you sure you want to change the base?

feat: support self-hosted embedding service via BentoML #324

Conversation

parano commented Aug 10, 2023

Shaunwei commented Aug 11, 2023 • edited Loading

aarnphm commented Aug 11, 2023

aarnphm left a comment

Choose a reason for hiding this comment

Shaunwei commented Aug 12, 2023

aarnphm left a comment

Choose a reason for hiding this comment

parano commented Aug 13, 2023

pycui commented Aug 16, 2023

aarnphm commented Aug 17, 2023 • edited Loading

Shaunwei commented Aug 11, 2023 •

edited

Loading

aarnphm commented Aug 17, 2023 •

edited

Loading