feat(migrate): use safe JSON parser when streaming from Export HTTP API #5542
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Ignored Deployment
|
Full Report@sanity/migrate
@sanity/diff
@sanity/block-tools
@sanity/types
sanity/desk
@sanity/portable-text-editor
@sanity/mutator
@sanity/cli
@sanity/schema/_internal
@sanity/util/paths
sanity/router
@sanity/util/legacyDateFormat
@sanity/schema
sanity/structure
sanity/cli
@sanity/vision
@sanity/util/fs
sanity/_internal
@sanity/util/createSafeJsonParser
sanity/_internalBrowser
@sanity/util/content
sanity
|
|
Component Testing Report Updated Jan 24, 2024 1:21 PM (UTC)
|
bjoerge
force-pushed
the
feat/migration-runner
branch
from
c08e78f to
f49ec7f
Compare
juice49
force-pushed
the
feat/migration-runner-json-parser
branch
from
be60eca to
547a495
Compare
juice49
requested review from
ricokahler,
bjoerge and
a team
and removed request for
a team and
ricokahler
juice49
changed the title
@sanity/export and @sanity/migrate
juice49
changed the title
bjoerge
force-pushed
the
feat/migration-runner
branch
from
f49ec7f to
707d6b7
Compare
juice49
force-pushed
the
feat/migration-runner-json-parser
branch
from
547a495 to
1bfcda7
Compare
bjoerge
force-pushed
the
feat/migration-runner
branch
from
707d6b7 to
cd86097
Compare
juice49
force-pushed
the
feat/migration-runner-json-parser
branch
from
1bfcda7 to
e1b7e7e
Compare
bjoerge
force-pushed
the
feat/migration-runner-json-parser
branch
from
d39ee19 to
0517de5
Compare
This was referenced Jan 25, 2024
bjoerge
pushed a commit
that referenced
this pull request
Jan 26, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
binoy14
pushed a commit
that referenced
this pull request
Jan 26, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
bjoerge
pushed a commit
that referenced
this pull request
Jan 26, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
bjoerge
pushed a commit
that referenced
this pull request
Jan 26, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
bjoerge
pushed a commit
that referenced
this pull request
Jan 29, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
bjoerge
pushed a commit
that referenced
this pull request
Jan 29, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
bjoerge
pushed a commit
that referenced
this pull request
Jan 29, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
bjoerge
pushed a commit
that referenced
this pull request
Jan 30, 2024
…PI (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jan 30, 2024
* feat(migrate): move into monorepo from poc * test(migrate): make sure uint8array-extras package gets transpiled from esm * feat(cli): scaffold create/run migration cli commands * fix(migrate): inline functions from uint8array-extras for now Can't be imported from package because of ESM * feat: basic migration cli support * feat: basic `migrations list` command * feat: migration create command w/templates * feat(sanity): add exports for sanity/migrate + sanity/migrate/mutations * chore(migrate): add test script * test(migrate): add tests for `parseJSON` * feat(migrate): add customisable parser and iterator type to `parseJSON` * feat(migrate): add JSON parser that can handle chunks interrupted by an error object * refactor(migrate): rename targets => destinations * feat(migrate): use safe JSON parser when streaming from Export HTTP API (#5542) * refactor(util): create shared JSON parser for `@sanity/export` and `@sanity/migrate` * feat(migrate): add safe JSON parser * chore(util): update comment * chore(export): update comment * feat(migrate): use safe JSON parser when streaming from Export HTTP API * feat(migrate): add JSON options and type parameter to NDJSON utility * refactor(migrate): use type parameter * fix(migrate): add missing export * fix(util): fix package.json formatting * feat(migrate): implement mutation batcher and use when submitting against mutate endpoint * feat(migrate): limit request concurrency * fix(migrate): workaround issue with p-map and ESM * feat(cli): allow user provided concurrency * feat(migration): improve progress as migration is running (#5550) * feat(migration): add a prompt before runing a real migration (#5552) ### Description <!-- What changes are introduced? Why are these changes introduced? What issue(s) does this solve? (with link, if possible) --> Adds a prompt before running a real migration with the projectId and dataset so the user is aware and intentionally selects running a real migration ### What to review <!-- What steps should the reviewer take in order to review? What parts/flows of the application/packages/tooling is affected? --> `sanity migration run <name-of-migration> --dry=false` should show a prompt like "This migration will run on the “test” dataset in “yajkdsl” project. Are you sure?" ### Testing <!-- Did you add sufficient testing for this change? If not, please explain how you tested this change and why it was not possible/practical for writing an automated test --> ### Notes for release <!-- A description of the change(s) that should be used in the release notes. --> - N/A * feat(migration): support passing an array of documentTypes to migration (#5566) * chore(migration): add better error message when request fails (#5567) * chore(migration): add better error message when request fails * fix(migration): add error message if json parsing fails * feat(migrate): add asyncIterableToStream util * feat(migrate): add bufferThroughFile utility * fix(migrate): buffer exports through file * fix(cli): use dryRun from @sanity/migrate * refactor(migrate): rename function * refactor(migrate): add ndjson support for both parsing and stringifying in ndjson util * refactor(sanity): remove mutiny dependency in favor of inlined creators * fix(sanity): add --no-progress flag * fix(migrate): simplify bufferThroughFile * feat(migrate): unify contexts, add support for document lookup and query during a migration * fix(dev): fix example migration script imports * feat(migration): support async hooks in migration nodes (#5564) * fix(sanity): use --no-dry instead of --dry=false * feat(migrate): add support for yielding transactions * fix(migrate): minor typing issue * fix(sanity): polish cli and migration templates * chore(deps): remove unused deps * fix(sanity): fix quoting of documentTypes in migration templates * fix(sanity): rename migration name => title * fix(migrate): run migration against all documents if documentTypes is omitted * fix(migration): fixes dry run not showing any information * fix(migration): show proper error message when the file has code issues * fix(migrate): various naming consistency fixes * fix(sanity): add a few examples to help text for 'sanity migration create' * fix(sanity): improve error handling on 'sanity migration run' without id * fix(migrate): fix error asserting transaction * fix(migrate): use transactionId instead of id on mutation payload * feat(cli): add pretty mutation formatting to migration runner (#5573) * feat(cli): add pretty mutation formatting to migration runner * feat(cli): improve formatting * feat(cli): add indent option to `prettyFormatMutation` * feat(cli): support indentation when printing JSON * feat(cli): support migrations for multiple document types * feat(cli): improve dry run migration formatting * refactor(cli): abstract tree reporter * chore(cli): remove unused import * fix(cli): add missing closing parenthesis * feat(cli): add migration dry-run output * feat(cli): underline document id when pretty formatting mutation * feat(cli): add transaction support to pretty formatter * feat(cli): add transaction support to pretty mutation formatter * refactor(cli): reduce complexity * feat(cli): pretty format mutations when committing migration * chore(migrate): instrument with some debug logging * refactor(migration): move filtered document methods to a separate `filtered`-context key, add a client with limited concurrency to migration context * fix(migrate): add request tags and user agent * fix(sanity): improve minimal example --------- Co-authored-by: Ash <[email protected]> Co-authored-by: Binoy Patel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When streaming from the Export HTTP API, NDJSON lines may be interrupted by an error object. Handling for this scenario was already implemented in #1787, exclusively for the exporter CLI. I've moved the safe JSON parser to
@sanity/utilso that it can be used by both@sanity/exportand@sanity/migrate.What to review
Testing
Build the project, run the
dataset exportandmigration runCLI commands.I've added tests for the JSON parser: