feat: add categoriser to fixtures in tests, fix streaming bugs (WIP) #323

stefl · 2024-10-30T16:27:43Z

Description

Fixes a bug where we were not initialising the Aila instance correctly
Adds the Categoriser results to the test fixtures in E2E tests
Fixes some streaming logic bugs where it would take 30 seconds to stream the response from fixtures in E2E tests
Makes it so that Aila must be initialised just once
Fixes the issue where we are seeing repeated categorisations for each step of the lesson plan process (this should speed things up!)
Adds a new generateObject method on the LLMService which allows us to do non-streaming generations

Waiting to merge the streaming UX PR before continuing on this

vercel · 2024-10-30T16:27:47Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
oak-ai-lesson-assistant	❌ Failed (Inspect)			Dec 20, 2024 3:37pm

stefl · 2024-10-30T16:28:22Z

apps/nextjs/src/app/api/chat/chatHandler.ts

              moderationAiClient,
              ragService: (aila: AilaServices) => new AilaRag({ aila }),
+              chatCategoriser: (aila: AilaServices) =>


Pass in the categoriser at initialisation, in the same way we do with the other features

stefl · 2024-10-30T16:28:35Z

apps/nextjs/src/app/api/chat/config.ts

@@ -26,5 +26,7 @@ export const defaultConfig: Config = {
        userId: undefined,
      },
    });
+    await createdAila.initialise();


Ensure that the Aila instance is initialised

stefl · 2024-10-30T16:29:57Z

packages/aila/src/core/Aila.liveWithOpenAI.test.ts

-              topic: "Basics and Advanced Techniques",
-            },
-          }),
+          chatCategoriser: () =>


A slight change to how we initialise the categoriser

stefl · 2024-10-30T16:30:26Z

packages/aila/src/core/Aila.ts

@@ -123,15 +122,38 @@ export class Aila implements AilaServices {
    this._plugins = options.plugins;
  }

+  private checkInitialised() {
+    if (!this._initialised) {


For our own sanity, throw an error if we try to use methods that rely on initialisation

stefl · 2024-10-30T16:30:40Z

packages/aila/src/core/Aila.ts

+    if (
+      !sections.includes("title") ||
+      !sections.includes("subject") ||
+      !sections.includes("keyStage")


We don't care about topic because it's optional

stefl · 2024-10-30T16:30:55Z

packages/aila/src/core/Aila.ts

+      !sections.includes("subject") ||
+      !sections.includes("keyStage")
+    ) {
+      await this._lesson.setUpInitialLessonPlan(this._chat.messages);


This was the bug that was causing repeated categorisations

stefl · 2024-10-30T16:31:21Z

packages/aila/src/core/chat/AilaChat.ts

-    await this.moderate();
-    await this.persistChat();
-    await this.persistGeneration("SUCCESS");
+    await Promise.all([


We can parallelise these

stefl · 2024-10-30T16:32:06Z

packages/aila/src/core/chat/AilaStreamHandler.ts

    try {
      await this._chat.setupGeneration();
+      this.logStreamingStep("Setup generation complete");


In the absence of tracing, these help understand what's happening during streaming and how long each step is taking

stefl · 2024-10-30T16:32:53Z

packages/aila/src/core/llm/OpenAIService.ts

@@ -87,4 +94,31 @@ export class OpenAIService implements LLMService {

    return newStream.getReader();
  }
+
+  async generateObject<T>({


A new method that we can use for categorisation etc. that uses structured outputs

stefl · 2024-10-30T16:33:52Z

packages/aila/src/utils/lessonPlan/completedSections.ts

@@ -0,0 +1,10 @@
+import type { LessonPlanKeys, LooseLessonPlan } from "../../protocol/schema";
+
+export const completedSections = (


Helper to list the keys that have values on the lesson plan

stefl · 2024-10-30T16:34:16Z

packages/core/src/prompts/lesson-assistant/parts/interactingWithTheUser.ts

+    content: "Respond to the user and help them edit the lesson plan",
+  };
+  const nextStep = allSteps[0] ?? finalStep;
+  const step = `${nextStep.title}\n\n${nextStep.content}`;


Reduces noisy logging during streaming

stefl · 2024-10-30T16:34:33Z

packages/core/src/rag/categorisation.ts

+    .nullable()
+    .describe("The reasoning behind the categorisation"),
+  keyStage: z
+    .enum([...keyStages] as [string, ...string[]])


More tightly specify the valid options for categorisation

sonarqubecloud · 2024-10-30T16:38:02Z

Quality Gate passed

Issues
11 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

build: release candidate

…ate - AI-626 AI-627 (#359)

build: release candidate

build: release posthog hotfix

build: release candidate

feat: add categoriser to fixtures in tests, fix streaming bugs

8a49562

stefl commented Oct 30, 2024

View reviewed changes

vercel bot had a problem deploying to Preview October 30, 2024 16:29 Failure

stefl commented Oct 30, 2024

View reviewed changes

Typescript import issue using @utils

09b96c9

vercel bot had a problem deploying to Preview October 30, 2024 16:40 Failure

stefl changed the title ~~feat: add categoriser to fixtures in tests, fix streaming bugs~~ feat: add categoriser to fixtures in tests, fix streaming bugs (WIP) Oct 30, 2024

codeincontext marked this pull request as draft November 5, 2024 11:06

stefl changed the base branch from main to fix/dropdown_sections November 7, 2024 14:11

JBR90 and others added 9 commits November 11, 2024 14:12

chore: add expired exports cron job to public routes (#358)

4c346d0

Merge pull request #360 from oaknational/rc-20241111-1

fba6c1c

build: release candidate

build(release v1.14.2): See CHANGE_LOG.md

4a9d4fb

feat: prisma health check - AI-625 (#356)

854950d

feat: add additional materials button - AI-539 [migration] (#255)

d0fe2d0

Merge pull request #363 from oaknational/main

646b419

build: release candidate

build(release v1.15.0): See CHANGE_LOG.md

713f18c

feat: add FeatureFlagProvider (#353)

1d4995e

feat: link to hubspot form from requests for full access and higher r…

05ccce6

…ate - AI-626 AI-627 (#359)

codeincontext and others added 29 commits December 10, 2024 14:50

test: remove legacy chromatic mode and use desktop as default (#439)

071d2d1

Merge pull request #428 from oaknational/rc-20241206-1

56f6d23

build: release candidate

build: fix release config cjs

a0d7c11

Merge pull request #442 from oaknational/rc-20241126-1

8c0fa5e

build: release candidate

build(release v1.18.0): See CHANGE_LOG.md

09646c7

Merge remote-tracking branch 'origin/production'

8b4e32a

chore: specify types for useProgressForDownloads (#433)

c91c5fb

fix: remove key from chat provider (#431)

35141b3

chore: add dependency cruiser (#438)

392b422

chore: reinstate react hooks linting, fix linting issues (#441)

0e43509

fix: improve chances of showing 5 relevant plans (#444)

4955a1e

fix: posthog consistent event properties (#440)

7c24485

test: storybook decorators (#443)

35e25fc

Merge pull request #453 from oaknational/rc-20241216-2

125a274

build: release candidate

build(release v1.18.1): See CHANGE_LOG.md

bf3c502

chore: update posthog (#454)

b114694

chore: update posthog (#454)

61ccbb6

fix: sonar duplication (#397)

dea3106

feat: soft delete (#383)

9dee4ad

Merge pull request #455 from oaknational/rc-20241216-3

0e8e48a

build: release posthog hotfix

build(release v1.18.2): See CHANGE_LOG.md

82d4c37

Merge remote-tracking branch 'origin/production'

a842173

test: use tighter "satisfies" typing in storybook (#456)

36a533d

Merge pull request #458 from oaknational/rc-20241218-1

4f2211e

build: release candidate

build(release v1.19.0): See CHANGE_LOG.md

a0999c1

Merge branch 'main' into fix/no_live_openai

8c07138

feat: react scan (#445)

2f255b3

fix: missing object fixture in replay

d4b3d82

Merge branch 'main' into fix/no_live_openai

392317f

vercel bot had a problem deploying to Preview December 20, 2024 15:37 Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add categoriser to fixtures in tests, fix streaming bugs (WIP) #323

feat: add categoriser to fixtures in tests, fix streaming bugs (WIP) #323

stefl commented Oct 30, 2024 •

edited

Loading

vercel bot commented Oct 30, 2024 •

edited

Loading

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

stefl Oct 30, 2024

sonarqubecloud bot commented Oct 30, 2024

		@@ -0,0 +1,10 @@
		import type { LessonPlanKeys, LooseLessonPlan } from "../../protocol/schema";

		export const completedSections = (

feat: add categoriser to fixtures in tests, fix streaming bugs (WIP) #323

Are you sure you want to change the base?

feat: add categoriser to fixtures in tests, fix streaming bugs (WIP) #323

Conversation

stefl commented Oct 30, 2024 • edited Loading

Description

vercel bot commented Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Oct 30, 2024

Quality Gate passed

stefl commented Oct 30, 2024 •

edited

Loading

vercel bot commented Oct 30, 2024 •

edited

Loading