feat(migrate): implement mutation batcher #5541

bjoerge · 2024-01-22T14:12:46Z

Description

Implements mutation batcher. Given a set of mutations, it will try to batch them into arrays not exceeding the given byte size limit

What to review

Does the logic seem right?

Testing

Take a look at the tests in packages/@sanity/migrate/src/runner/utils/tests/batchMutations.test.ts

Notes for release

n/a

vercel · 2024-01-22T14:12:53Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
performance-studio	✅ Ready (Inspect)	Visit Preview		Jan 24, 2024 1:52pm
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 24, 2024 1:52pm

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Jan 24, 2024 1:52pm

github-actions · 2024-01-22T14:16:47Z

Package	Documentation Change
sanity	-3%

Full Report

@sanity/migrate

This branch	Next branch
1 documented	0 documented
35 not documented	0 not documented

@sanity/diff

This branch	Next branch
13 documented	13 documented
16 not documented	16 not documented

@sanity/block-tools

This branch	Next branch
4 documented	4 documented
9 not documented	9 not documented

@sanity/types

This branch	Next branch
55 documented	55 documented
239 not documented	239 not documented

sanity/desk

This branch	Next branch
84 documented	84 documented
64 not documented	64 not documented

@sanity/portable-text-editor

This branch	Next branch
21 documented	21 documented
44 not documented	44 not documented

@sanity/mutator

This branch	Next branch
7 documented	7 documented
4 not documented	4 not documented

@sanity/cli

This branch	Next branch
1 documented	1 documented
31 not documented	31 not documented

@sanity/schema/_internal

This branch	Next branch
0 documented	0 documented
12 not documented	12 not documented

@sanity/util/paths

This branch	Next branch
1 documented	1 documented
15 not documented	15 not documented

sanity/router

This branch	Next branch
17 documented	17 documented
26 not documented	26 not documented

@sanity/util/legacyDateFormat

This branch	Next branch
0 documented	0 documented
3 not documented	3 not documented

@sanity/schema

This branch	Next branch
0 documented	0 documented
2 not documented	2 not documented

sanity/structure

This branch	Next branch
2 documented	2 documented
6 not documented	6 not documented

sanity/cli

This branch	Next branch
2 documented	2 documented
0 not documented	0 not documented

@sanity/vision

This branch	Next branch
0 documented	0 documented
3 not documented	3 not documented

@sanity/util/fs

This branch	Next branch
0 documented	0 documented
3 not documented	3 not documented

sanity/_internal

This branch	Next branch
0 documented	0 documented
1 not documented	1 not documented

@sanity/util/createSafeJsonParser

This branch	Next branch
1 documented	0 documented
0 not documented	0 not documented

sanity/_internalBrowser

This branch	Next branch
0 documented	0 documented
3 not documented	3 not documented

@sanity/util/content

This branch	Next branch
1 documented	1 documented
5 not documented	5 not documented

sanity

This branch	Next branch
177 documented	183 documented
843 not documented	846 not documented

github-actions · 2024-01-22T14:20:28Z

Component Testing Report Updated Jan 24, 2024 1:53 PM (UTC)

File	Status	Duration	Passed	Skipped	Failed
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	34s	15	0	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	7s	3	0	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	✅ Passed (Inspect)	12s	3	0	0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	13s	4	2	0
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	13s	6	0	0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	34s	15	0	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	19s	9	0	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	1m 3s	18	0	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	13s	6	0	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	❌ Failed (Inspect)	12s	2	0	1

bjoerge · 2024-01-22T19:07:55Z

Current dependencies on/for this PR:

next
- PR feat: migration runner #5540

This stack of pull requests is managed by Graphite.

socket-security · 2024-01-22T19:10:40Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@types/[email protected]	None	`+1`	4.91 kB	types

View full report↗︎

juice49 · 2024-01-23T11:29:13Z

packages/@sanity/migrate/src/runner/utils/__tests__/batchMutations.test.ts

+    expect(await it.next()).toEqual({value: [second], done: false})
+    expect(await it.next()).toEqual({value: undefined, done: true})
+  })
+  test('when each mutation is smaller then max batch size', async () => {


Should this read "bigger", instead of "smaller"?

juice49 · 2024-01-23T11:50:29Z

packages/@sanity/migrate/src/runner/utils/batchMutations.ts

+      continue
+    }
+
+    // the mutation itself may exceed the payload size, need to handle that


Will we throw an error in this scenario?

I think it's better to try posting the mutation. The limits we currently have in the migraiton runner is quite conservative compared to the payload size accepted by the mutate endpoint (which is 4MB)

…inst mutate endpoint

…ing against mutate endpoint

bjoerge requested a review from a team as a code owner January 22, 2024 14:12

bjoerge requested review from jtpetty and removed request for a team January 22, 2024 14:12

bjoerge requested a review from juice49 January 22, 2024 14:13

vercel bot deployed to Preview – performance-studio January 22, 2024 14:18 View deployment

vercel bot deployed to Preview – test-studio January 22, 2024 14:20 View deployment

bjoerge mentioned this pull request Jan 22, 2024

feat(migrate): limit mutate request concurrency #5544

Merged

vercel bot deployed to Preview – performance-studio January 22, 2024 19:12 View deployment

vercel bot deployed to Preview – test-studio January 22, 2024 19:13 View deployment

bjoerge force-pushed the migration-runner-batch-mutations branch from a81864b to 7ca5626 Compare January 23, 2024 10:24

bjoerge force-pushed the feat/migration-runner branch from c08e78f to f49ec7f Compare January 23, 2024 10:25

vercel bot deployed to Preview – performance-studio January 23, 2024 10:28 View deployment

vercel bot deployed to Preview – test-studio January 23, 2024 10:29 View deployment

bjoerge force-pushed the feat/migration-runner branch from f49ec7f to 707d6b7 Compare January 23, 2024 10:35

bjoerge force-pushed the migration-runner-batch-mutations branch from 7ca5626 to 173baba Compare January 23, 2024 10:36

vercel bot deployed to Preview – performance-studio January 23, 2024 10:45 View deployment

vercel bot deployed to Preview – test-studio January 23, 2024 10:45 View deployment

juice49 reviewed Jan 23, 2024

View reviewed changes

bjoerge force-pushed the feat/migration-runner branch from 707d6b7 to cd86097 Compare January 23, 2024 11:49

juice49 reviewed Jan 23, 2024

View reviewed changes

bjoerge force-pushed the migration-runner-batch-mutations branch from 173baba to e8aba07 Compare January 23, 2024 11:53

bjoerge mentioned this pull request Jan 23, 2024

feat(migration): improve progress as migration is running #5550

Merged

vercel bot deployed to Preview – performance-studio January 23, 2024 12:05 View deployment

vercel bot deployed to Preview – test-studio January 23, 2024 12:06 View deployment

bjoerge force-pushed the feat/migration-runner branch from cd86097 to 2a5597f Compare January 23, 2024 12:39

bjoerge force-pushed the feat/migration-runner branch from 0301e25 to 0bc11c0 Compare January 24, 2024 12:56

bjoerge force-pushed the migration-runner-batch-mutations branch from 334a881 to 5af81f4 Compare January 24, 2024 12:56

This was referenced Jan 24, 2024

feat: migration runner #5540

Merged

feat(migrate): use safe JSON parser when streaming from Export HTTP API #5542

Merged

vercel bot deployed to Preview – performance-studio January 24, 2024 13:01 View deployment

binoy14 mentioned this pull request Jan 24, 2024

feat(migration): add a prompt before runing a real migration #5552

Merged

vercel bot deployed to Preview – test-studio January 24, 2024 13:01 View deployment

bjoerge force-pushed the migration-runner-batch-mutations branch from 5af81f4 to 0474ecf Compare January 24, 2024 13:19

vercel bot deployed to Preview – performance-studio January 24, 2024 13:24 View deployment

vercel bot deployed to Preview – test-studio January 24, 2024 13:25 View deployment

bjoerge added 2 commits January 24, 2024 14:28

feat(migrate): implement mutation batcher and use when submitting aga…

a873556

…inst mutate endpoint

fixup! feat(migrate): implement mutation batcher and use when submitt…

0cb219a

…ing against mutate endpoint

vercel bot deployed to Preview – performance-studio January 24, 2024 13:28 View deployment

bjoerge force-pushed the migration-runner-batch-mutations branch from bbac6c9 to 0cb219a Compare January 24, 2024 13:29

vercel bot deployed to Preview – performance-studio January 24, 2024 13:33 View deployment

vercel bot deployed to Preview – test-studio January 24, 2024 13:34 View deployment

bjoerge added 3 commits January 24, 2024 14:46

feat(migrate): limit request concurrency

182ff03

fix(migrate): workaround issue with p-map and ESM

6a8519f

feat(cli): allow user provided concurrency

140e222

bjoerge merged commit 5bd2ab2 into feat/migration-runner Jan 24, 2024
19 of 22 checks passed

bjoerge deleted the migration-runner-batch-mutations branch January 24, 2024 13:49

vercel bot deployed to Preview – performance-studio January 24, 2024 13:51 View deployment

vercel bot deployed to Preview – test-studio January 24, 2024 13:52 View deployment

bjoerge mentioned this pull request Jan 26, 2024

feat(migrate): add bufferThroughFile utility #5572

Merged

This was referenced Jan 26, 2024

feat(cli): add pretty mutation formatting to migration runner #5573

Merged

feat(cli): improve reporting when dry running from export #5609

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(migrate): implement mutation batcher #5541

feat(migrate): implement mutation batcher #5541

bjoerge commented Jan 22, 2024

vercel bot commented Jan 22, 2024 •

edited

Loading

github-actions bot commented Jan 22, 2024 •

edited

Loading

github-actions bot commented Jan 22, 2024 •

edited

Loading

bjoerge commented Jan 22, 2024 •

edited by juice49

Loading

socket-security bot commented Jan 22, 2024 •

edited

Loading

juice49 Jan 23, 2024

bjoerge Jan 24, 2024

juice49 Jan 23, 2024

bjoerge Jan 24, 2024

feat(migrate): implement mutation batcher #5541

feat(migrate): implement mutation batcher #5541

Conversation

bjoerge commented Jan 22, 2024

Description

What to review

Testing

Notes for release

vercel bot commented Jan 22, 2024 • edited Loading

github-actions bot commented Jan 22, 2024 • edited Loading

github-actions bot commented Jan 22, 2024 • edited Loading

bjoerge commented Jan 22, 2024 • edited by juice49 Loading

socket-security bot commented Jan 22, 2024 • edited Loading

juice49 Jan 23, 2024

Choose a reason for hiding this comment

bjoerge Jan 24, 2024

Choose a reason for hiding this comment

juice49 Jan 23, 2024

Choose a reason for hiding this comment

bjoerge Jan 24, 2024

Choose a reason for hiding this comment

vercel bot commented Jan 22, 2024 •

edited

Loading

github-actions bot commented Jan 22, 2024 •

edited

Loading

github-actions bot commented Jan 22, 2024 •

edited

Loading

bjoerge commented Jan 22, 2024 •

edited by juice49

Loading

socket-security bot commented Jan 22, 2024 •

edited

Loading