Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outbox prisma adapter #231

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
0353847
AP-5046 WIP prisma adapter.
kamilwylegala Sep 9, 2024
14fbfba
WIP prisma adapter.
kamilwylegala Sep 9, 2024
d55405e
Working test.
kamilwylegala Sep 11, 2024
e93bdad
WIP
kamilwylegala Sep 11, 2024
963014a
Working test for saving outbox entries.
kamilwylegala Nov 26, 2024
7444fab
failing test for updating.
kamilwylegala Nov 26, 2024
9c971fc
Bulk update + insert.
kamilwylegala Nov 27, 2024
5ad155f
Failed entries handling.
kamilwylegala Nov 27, 2024
d9e3d41
fetching entries up to the retry count limit.
kamilwylegala Nov 27, 2024
c61533d
Narrowed down types.
kamilwylegala Nov 27, 2024
552e0b6
lint fix
kamilwylegala Nov 27, 2024
f143384
Use generated db client from test dir.
kamilwylegala Nov 27, 2024
432875b
Build includes building test prisma client.
kamilwylegala Nov 27, 2024
ccf29a6
Fixed import.
kamilwylegala Nov 27, 2024
3f39f92
prisma main dependency.
kamilwylegala Nov 27, 2024
0b7d718
prisma client dev dependency.
kamilwylegala Nov 27, 2024
c2108e3
Build before lint.
kamilwylegala Nov 28, 2024
a7a9cb0
Ignore db client in biome.
kamilwylegala Nov 28, 2024
1b3c72c
Peer prisma.
kamilwylegala Nov 28, 2024
77b1d26
inferred type
kamilwylegala Nov 28, 2024
856ee17
debugging ci
kamilwylegala Nov 28, 2024
45d7310
keep prisma outside test folder in root
kamilwylegala Nov 28, 2024
00ca6d0
Fixed import in spec.
kamilwylegala Nov 28, 2024
33b954b
temp ts ignore.
kamilwylegala Nov 28, 2024
829c9ac
wait for db.
kamilwylegala Nov 28, 2024
1f39dbf
wait for db.
kamilwylegala Nov 28, 2024
8b818c1
wait for db.
kamilwylegala Nov 28, 2024
ab58167
Redundant docker start.
kamilwylegala Nov 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions packages/outbox-prisma-adapter/lib/outbox-prisma-adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,37 @@ export class OutboxPrismaAdapter<SupportedEvents extends CommonEventDefinition[]
id: outboxEntry.id,
type: messageType,
created: outboxEntry.created,
updated: outboxEntry.updated,
data: outboxEntry.data,
status: outboxEntry.status,
},
})
}

flush(outboxAccumulator: OutboxAccumulator<SupportedEvents>): Promise<void> {
return Promise.resolve(undefined)
async flush(outboxAccumulator: OutboxAccumulator<SupportedEvents>): Promise<void> {
const entries = await outboxAccumulator.getEntries()

const prismaModel: PrismaClient[typeof this.modelName] = this.prisma[this.modelName]

for (const entry of entries) {
await prismaModel.upsert({
Copy link
Owner

@kibertoad kibertoad Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

upserts often have suboptimal performance with plenty of locking, can we somehow simplify this to be bulk inserts and bulk updates?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely. I'm working on making tests green. Once I cover all cases, Bulk update/insert would do the job.

Actually upsert has some weird behavior, maybe let's switch to bulk inserts/updates right away.

where: {
id: entry.id,
},
update: {
status: 'SUCCESS',
updated: new Date(),
},
create: {
id: entry.id,
type: getMessageType(entry.event),
created: entry.created,
updated: new Date(),
data: entry.data,
status: 'SUCCESS',
},
})
}
}

getEntries(maxRetryCount: number): Promise<OutboxEntry<SupportedEvents[number]>[]> {
Expand Down
124 changes: 123 additions & 1 deletion packages/outbox-prisma-adapter/test/outbox-prisma-adapter.spec.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { OutboxEntry } from '@message-queue-toolkit/outbox-core'
import { InMemoryOutboxAccumulator, type OutboxEntry } from '@message-queue-toolkit/outbox-core'
import {
type CommonEventDefinition,
enrichMessageSchemaWithBase,
Expand Down Expand Up @@ -37,6 +37,7 @@ describe('outbox-prisma-adapter', () => {
id UUID PRIMARY KEY,
type TEXT NOT NULL,
created TIMESTAMP NOT NULL,
updated TIMESTAMP,
retry_count INT NOT NULL DEFAULT 0,
data JSONB NOT NULL,
status TEXT NOT NULL
Expand Down Expand Up @@ -74,6 +75,7 @@ describe('outbox-prisma-adapter', () => {
id: expect.any(String),
type: 'entity.created',
created: expect.any(Date),
updated: expect.any(Date),
retryCount: 0,
data: {
id: expect.any(String),
Expand All @@ -87,4 +89,124 @@ describe('outbox-prisma-adapter', () => {
},
])
})

it('should insert successful entries from accumulator', async () => {
const accumulator = new InMemoryOutboxAccumulator<SupportedEvents>()

const entry1 = {
id: uuidv7(),
event: events.created,
status: 'CREATED',
data: {
id: uuidv7(),
payload: {
message: 'TEST EVENT',
},
metadata: {},
timestamp: new Date().toISOString(),
},
retryCount: 0,
created: new Date(),
} satisfies OutboxEntry<SupportedEvents[number]>
accumulator.add(entry1)

const entry2 = {
id: uuidv7(),
event: events.created,
status: 'CREATED',
data: {
id: uuidv7(),
payload: {
message: 'TEST EVENT 2',
},
metadata: {},
timestamp: new Date().toISOString(),
},
retryCount: 0,
created: new Date(),
} satisfies OutboxEntry<SupportedEvents[number]>
accumulator.add(entry2)

await outboxPrismaAdapter.flush(accumulator)

const entriesAfterFlush = await outboxPrismaAdapter.getEntries(10)

expect(entriesAfterFlush).toMatchObject([
{
id: entry1.id,
status: 'SUCCESS',
},
{
id: entry2.id,
status: 'SUCCESS',
},
])
})

it("should update successful entries' status to 'SUCCESS'", async () => {
const accumulator = new InMemoryOutboxAccumulator<SupportedEvents>()

const entry1 = {
id: uuidv7(),
event: events.created,
status: 'CREATED',
data: {
id: uuidv7(),
payload: {
message: 'TEST EVENT',
},
metadata: {},
timestamp: new Date().toISOString(),
},
retryCount: 0,
created: new Date(),
} satisfies OutboxEntry<SupportedEvents[number]>
accumulator.add(entry1)

const entry2 = {
id: uuidv7(),
event: events.created,
status: 'CREATED',
data: {
id: uuidv7(),
payload: {
message: 'TEST EVENT 2',
},
metadata: {},
timestamp: new Date().toISOString(),
},
retryCount: 0,
created: new Date(),
} satisfies OutboxEntry<SupportedEvents[number]>
accumulator.add(entry2)

await outboxPrismaAdapter.createEntry(entry1)
await outboxPrismaAdapter.createEntry(entry2)

const beforeFlush = await outboxPrismaAdapter.getEntries(10)
expect(beforeFlush).toMatchObject([
{
id: entry1.id,
status: 'CREATED',
},
{
id: entry2.id,
status: 'CREATED',
},
])

outboxPrismaAdapter.flush(accumulator)

const afterFlush = await outboxPrismaAdapter.getEntries(10)
expect(afterFlush).toMatchObject([
{
id: entry1.id,
status: 'SUCCESS',
},
{
id: entry2.id,
status: 'SUCCESS',
},
])
})
})
1 change: 1 addition & 0 deletions packages/outbox-prisma-adapter/test/schema.prisma
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ datasource db {
model OutboxEntry {
id String @id @default(uuid()) @db.Uuid
created DateTime @default(now())
updated DateTime @default(now()) @updatedAt
type String
retryCount Int @default(0) @map("retry_count")
data Json
Expand Down
Loading