Implement embeddings for use with LLM agents

[ahuang11]

Uses duckdb instead of chromadb. Currently subsets embeddings by table name. Not sure how to handle ephemeral tables yet.

import io
import asyncio

import lumen.ai as lmai
import panel as pn
import pandas as pd

from lumen.sources.duckdb import DuckDBSource
from lumen.ai.analysis import Join

pn.extension(
    "tabulator",
    "codeeditor",
    "vega",
    inline=False,
    template="fast",
    notifications=True,
)

llm = lmai.llm.OpenAI(
    # interceptor_path="interceptor.db"
)

embeddings = lmai.embeddings.OpenAIEmbeddings.from_dict(
    {
        "windturbines.parquet": [
            "The creator of this dataset is named Andrew HH",
            "To run the imaginary, test analysis, simply divide turbine count by 3",
        ]
    }
)

cs = lmai.memory["current_source"] = DuckDBSource(
    tables=["windturbines.parquet"],
    uri=":memory:",
    initializers=["INSTALL httpfs;", "LOAD httpfs;"],
)

custom_agent = lmai.agents.AnalysisAgent(analyses=[Join])

assistant = lmai.Assistant(
    llm=llm,
    agents=[
        lmai.agents.SourceAgent,
        lmai.agents.TableAgent,
        lmai.agents.TableListAgent,
        lmai.agents.SQLAgent,
        lmai.agents.PipelineAgent,
        lmai.agents.hvPlotAgent,
        lmai.agents.ChatAgent,
        custom_agent,
    ],
    embeddings=embeddings,
    suggestions=["Who is the creator of this dataset?"],
)
assistant.servable("Lumen.ai")
assistant.controls().servable(area="sidebar")

[codecov]

Codecov Report

Attention: Patch coverage is 27.14286% with 51 lines in your changes missing coverage. Please review.

Project coverage is 60.70%. Comparing base (73fda79) to head (18dfbf5).

Files with missing lines	Patch %	Lines
lumen/ai/embeddings.py	29.82%	40 Missing ⚠️
lumen/ai/agents.py	0.00%	7 Missing ⚠️
lumen/ai/assistant.py	33.33%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #680      +/-   ##
==========================================
- Coverage   60.82%   60.70%   -0.12%     
==========================================
  Files         100      100              
  Lines       12402    12445      +43     
==========================================
+ Hits         7543     7555      +12     
- Misses       4859     4890      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ahuang11]

Another TODO: handle ephemeral tables