feat: OpenAIGeneration model for embedder

[jordanh]

Description

Implements #9436

Demo

...
embedder:     ...summarizing retrospectiveDiscussionTopic:nzyvX6Nbzy:nzyw41YGfa for Embeddings_ember_1
embedder: completed retrospectiveDiscussionTopic:nzyvX6Nbzy:nzyw41YGfa -> Embeddings_ember_1
...

Original fullText (names redacted):

A topic "Culture & Departures" was discussed during the meeting "Retro
#108" that followed the "Hero’s Journey 👑 + Kudos" template.

Participants were prompted with, "Cavern: What challenges lie ahead?".

- Anonymous wrote, "(long, sorry!) I hope the majority of the company
  does not think that X was treated unfairly. I also hope they not
  think that the Sales team is a bunch of "sharks" who aren't willing to
  accommodate people with different personality types. From my
  perspective X was given nothing but support and help. A LOT of
  help. Despite that, X was consistently the lowest performer on the
  team by a large margin.

I was very frustrated by X's going away message, which to me came across
as selfish, immature, and unnecessarily inflammatory. No company is
perfect, we can all improve. I like X a lot as a person. But I don't
think X was treated poorly during his time here and it bothers me
that X left how he did, making all those accusations. - Y "

- Anonymous wrote, "I feel that the recent departures somehow impact the
  trust of many members towards how things are handled, and we need now
  more than ever to regroup, be very transparent and reflect on how we
  can do better as a team "
- Anonymous wrote, "I didn't like X's insinuations that we are kinda
  sorta racist. I think he would have been the first person to tell you
  that Z (former sales member) was bad at their job.

We can be better at DE&I, but I also see that our company has a wide
variety of people from all kinds of backgrounds working together and
thriving. Let's not lose sight of that. "

- Anonymous wrote, "I feel like we have had some weird days, maybe even
  with a bit of tension. Watching P's new TT video made me reflect
  on some silent meetings we have sat on during the past weeks. I don't
  think this is a very big issue, but I wonder if we are keeping some
  tensions unsaid."
- Anonymous wrote, "X's message hit me hard, it made me wonder how
  much his perspective or general ... concern? is shared"

OpenAI GPT 3.5 Turbo:

During the meeting "Retro #108", the topic of "Culture & Departures" was discussed following the "Hero’s Journey 👑 + Kudos" template. Participants were prompted with the question, "Cavern: What challenges lie ahead?". Various anonymous comments were made regarding recent departures impacting trust within the team, reflections on diversity, tensions within the company, and differing perspectives on the departure of a former team member. The discussion highlighted the importance of transparency, reflection, and unity in moving forward as a team.

Compare with Zephyr:

 During the "Retro #108" meeting, using the "Hero’s Journey 👑 + Kudos" template, the topic of culture and departures was discussed. Participants were prompted to contemplate the challenges that lie ahead in the cavernous space. An anonymous individual expressed their hope that the majority of the company did not view X's departure as unfair or interpret the Sales team's actions as unaccommodating towards those with different personality types. The writer described X as receiving a significant amount of assistance and support, yet continuously being the lowest-performing member of the team by a substantial margin. They expressed frustration with X's dismissal message, which they felt came across as selfish, immature, and inflammatory. Another anonymous individual voiced concerns regarding the impact of recent departures on members' trust in the handling of issues. They suggested that the company should be transparent and reflect on how to improve as a team. A third anonymous writer disapproved of X insinuations about potential racism within the company, highlighting Z's poor job performance as evidence. The individual expressed concern regarding the company's efforts towards DE&I but acknowledged that it employs a diverse group of individuals who thrive together. Another anonymous writer reflected on some tense meetings and wondered if there were unsaid tensions present. A final anonymous writer was moved by X's farewell message and questioned how widely held perspectives or concerns might be.

Testing scenarios

If you have a set of embeddings generated using a local LLM for summarization, I recommend running a query such as...

UPDATE "EmbeddingsMetadata" SET "models" = '{}'
WHERE id IN (
    SELECT em.id
    FROM "EmbeddingsMetadata" em
    JOIN "Embeddings_ember_1" e ON em.id = e."embeddingsMetadataId"
    WHERE NOT "embedText" IS NULL
)

...in order to cause the embedder to re-embed rows that required summarization.

Make sure to enable this summarizer in .env by setting:

AI_EMBEDDING_MODELS='[{"model": "text-embeddings-inference:llmrails/ember-v1", "url": "http://localhost:3040/"}]'
AI_GENERATION_MODELS='[{"model": "openai:gpt-3.5-turbo-0125"}]'
AI_EMBEDDER_ENABLED='true'
OPEN_AI_API_KEY='sk-xxxxx.....'
OPEN_AI_ORG_ID='org-yyyyy....'

• Run embedder
  • Observe summaries being generated, embeddings added

Final checklist

• I checked the code review guidelines
• I have performed a self-review of my code, the same way I'd do it for any other team member
• I have tested all cases I listed in the testing scenarios and I haven't found any issues or regressions
• Whenever I took a non-obvious choice I added a comment explaining why I did it this way
• I added the label Skip Maintainer Review Indicating the PR only requires reviewer review and can be merged right after it's approved if the PR introduces only minor changes, does not contain any architectural changes or does not introduce any new patterns and I think one review is sufficient'
• PR title is human readable and could be used in changelog

[jordanh]

This was unused. Good cruft to remove

[jordanh]

This is not supported by OpenAI, I believe it truncates by default. I changed the TextGenerationInference implementation to always truncate and removed this option.

[jordanh]

Observing the differences between Zephyr and gpt-3.5-turbo, I have a feeling we may need to play with this prompt a bit to maximize which features are included in the summary. There are choices we may want to consider:

• Preserving names vs. using pronouns if they are known
• Attempting to summarize each thread into a chunk of text, rather than summarizing the entire block

I'm not sure yet, though, so I want to forge ahead and try and do some reporting before iterating on this pipeline.

[jordanh]

There was a bug here! maxInputTokens should have been maxNewTokens

[jordanh]

There was a bug here! We want to check if em.models is missing any configured models. Before this change, we were checking if any configured models were missing any elements in em.models. The only reason this produced queued jobs before was because em.models defaults to NULL.

The <@ is the "is contained by" operator. It checks if the left-hand operand is contained within the right-hand operand. It returns true if the set on the right includes all elements or the specific structure on the left.

ARRAY[2,3] <@ ARRAY[1,2,3] returns true
ARRAY[2] <@ ARRAY[1,2,3] returns true (not what we want!)

The @> operator is the "contains" operator. It checks if the left-hand operand contains the right-hand operand. It returns true if the set on the left includes all elements or the specific structure on the right.

ARRAY[1,2,3] @> ARRAY[2,3] returns true
ARRAY[1,2] @> ARRAY[2,3] returns false (what we want)

The better fix is to just stop using the models array...but we've already got that on our list. This just makes the current state work as designed.

[jordanh]

@mattkrick this is ready for review. Wanted to knock this one out. Note: there are a few bug fixes in here, too.

[mattkrick]

didn't run a test but code looks good! i figure we'll do plenty of testing before we turn the thing on