Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node.js production runtime POC #4998

Merged
merged 19 commits into from
May 26, 2023
Merged

Conversation

alexkirsz
Copy link
Contributor

@alexkirsz alexkirsz commented May 17, 2023

Description

This implements a Node.js production runtime for Turbopack, to first be used for next build --turbo, but that can easily be extended to support more production use cases (e.g. ncc).

The runtime works differently from the dev runtime we're currently using for next dev: instead of having each chunk register itself against a global registry when evaluated, chunks export their module factories as a CommonJS default export. The runtime itself lives in a separate chunk, and then an "exported chunk" is generated that instantiates runtime entries and exports a given entry module.

Corresponding Next.js PR: vercel/next.js#49942 vercel/next.js#50375

@vercel
Copy link

vercel bot commented May 17, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
examples-cra-web 🔄 Building (Inspect) Visit Preview 💬 Add feedback May 26, 2023 7:59am
examples-designsystem-docs 🔄 Building (Inspect) Visit Preview 💬 Add feedback May 26, 2023 7:59am
9 Ignored Deployments
Name Status Preview Comments Updated (UTC)
examples-basic-web ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-gatsby-web ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-kitchensink-blog ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-native-web ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-nonmonorepo ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-svelte-web ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-tailwind-web ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
examples-vite-web ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am
turbo-site ⬜️ Ignored (Inspect) Visit Preview May 26, 2023 7:59am

@github-actions
Copy link
Contributor

✅ This change can build next-swc

@@ -0,0 +1,359 @@
/* eslint-disable @next/next/no-assign-module-variable */
Copy link
Contributor Author

@alexkirsz alexkirsz May 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a slimmed down version of our dev runtime that doesn't include HMR.

It's currently missing some more recent updates to our dev runtime.

Eventually, this runtime should be optimized for conciseness and performance. But for now, readability is privileged.

crates/turbopack-build/src/ecmascript/node/content.rs Outdated Show resolved Hide resolved
crates/turbopack-dev/src/chunking_context.rs Outdated Show resolved Hide resolved
@github-actions
Copy link
Contributor

github-actions bot commented May 17, 2023

⚠️ CI failed ⚠️

The following steps have failed in CI:

  • Turbopack Rust tests (mac/win, non-blocking)

See workflow summary for details

@socket-security

This comment was marked as resolved.

@alexkirsz
Copy link
Contributor Author

The Socket comment has nothing to do with this PR, and relates to test fixtures. cc @mehulkar might need to update these fixtures?

@mehulkar
Copy link
Contributor

mehulkar commented May 22, 2023

@alexkirsz no idea. It looks like turbopack jobs are failing and I can't tell by scrolling the logs for it which fixtures they're using. Ping me on slack if there's something I'm missing?

oh i see you're talking about socket, my bad. Yeah you can ignore this.

Copy link
Member

@sokra sokra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I quite dislike the duplication of the runtime code. Could we move the shared part to turbopack-ecmascript? (I want to avoid dev to prod differences as this could lead to production only problems)

We originally had the the logic for import() in turbopack-dev as it uses an extra indirection (manifest chunk) to allow lazy compiling the graph behind the import(). This won't be needed in prod. But I think it's ok to have it in turbopack-ecmascript and maybe use some kind of ChunkingContext trait method to control if the manifest module should be in a separate chunk or not.

crates/turbopack-build/js/src/runtime.js Outdated Show resolved Hide resolved
crates/turbopack-core/src/chunk/chunking_context.rs Outdated Show resolved Hide resolved
crates/turbopack-core/src/ident.rs Show resolved Hide resolved
crates/turbopack-ecmascript/js/package.json Outdated Show resolved Hide resolved
@alexkirsz alexkirsz force-pushed the alexkirsz/web-397-next-build-poc branch from 0189585 to 8e19721 Compare May 24, 2023 16:32
@alexkirsz alexkirsz marked this pull request as draft May 24, 2023 16:32
@alexkirsz
Copy link
Contributor Author

Turning this back into a draft while I address #4998 (comment)

Copy link
Member

@sokra sokra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No way I can read all that code... It's probably fine...

.github/workflows/test.yml Show resolved Hide resolved
crates/turbopack-core/src/chunk/chunking_context.rs Outdated Show resolved Hide resolved
crates/turbopack-core/src/ident.rs Show resolved Hide resolved
Copy link
Member

@sokra sokra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't able to fully review that, but I trust the test suite 🤞

@alexkirsz alexkirsz merged commit a066e12 into main May 26, 2023
@alexkirsz alexkirsz deleted the alexkirsz/web-397-next-build-poc branch May 26, 2023 10:12
sokra pushed a commit to vercel/next.js that referenced this pull request May 26, 2023
This PR is extracted from #49942
and mostly contains changes necessary after the Turbopack PR adding the
Node.js production runtime vercel/turborepo#4998,
without any of the actual Next Build stuff, in order to be able to merge
both quickly.

* ChunkData moved from tp-dev to tp-core, the ES-serializable part moved
to tp-ecmascript;
* all runtime types moved to tp-ecmascript-runtime

This also upgrades Turbopack to turbopack-230526.2:

* vercel/turborepo#5102 <!-- Donny/강동윤 - refactor:
Fix binary bloat caused by `ValueDebugFormat` impl -->
* vercel/turborepo#4998 <!-- Alex Kirszenberg -
Node.js production runtime POC -->
hydRAnger pushed a commit to hydRAnger/next.js that referenced this pull request Jun 12, 2023
…el#50375)

This PR is extracted from vercel#49942
and mostly contains changes necessary after the Turbopack PR adding the
Node.js production runtime vercel/turborepo#4998,
without any of the actual Next Build stuff, in order to be able to merge
both quickly.

* ChunkData moved from tp-dev to tp-core, the ES-serializable part moved
to tp-ecmascript;
* all runtime types moved to tp-ecmascript-runtime

This also upgrades Turbopack to turbopack-230526.2:

* vercel/turborepo#5102 <!-- Donny/강동윤 - refactor:
Fix binary bloat caused by `ValueDebugFormat` impl -->
* vercel/turborepo#4998 <!-- Alex Kirszenberg -
Node.js production runtime POC -->
kodiakhq bot pushed a commit to vercel/next.js that referenced this pull request Jun 19, 2023
This contains the original POC for `next build --turbo`. The implementation is _just enough_ to get pages building, and doesn't support the app router yet.

I'll write more details here on the implementation and what the next steps are next week.

Necessary changes on the Turbo side: vercel/turborepo#4998
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 25, 2024
### Description

This implements a Node.js production runtime for Turbopack, to first be
used for `next build --turbo`, but that can easily be extended to
support more production use cases (e.g. ncc).

The runtime works differently from the dev runtime we're currently using
for `next dev`: instead of having each chunk register itself against a
global registry when evaluated, chunks export their module factories as
a CommonJS default export. The runtime itself lives in a separate chunk,
and then an "exported chunk" is generated that instantiates runtime
entries and exports a given entry module.

Corresponding Next.js PR:
~~#49942
#50375

---------

Co-authored-by: Tobias Koppers <[email protected]>
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Jul 29, 2024
### Description

This implements a Node.js production runtime for Turbopack, to first be
used for `next build --turbo`, but that can easily be extended to
support more production use cases (e.g. ncc).

The runtime works differently from the dev runtime we're currently using
for `next dev`: instead of having each chunk register itself against a
global registry when evaluated, chunks export their module factories as
a CommonJS default export. The runtime itself lives in a separate chunk,
and then an "exported chunk" is generated that instantiates runtime
entries and exports a given entry module.

Corresponding Next.js PR:
~~#49942
#50375

---------

Co-authored-by: Tobias Koppers <[email protected]>
ForsakenHarmony pushed a commit to vercel/next.js that referenced this pull request Aug 1, 2024
### Description

This implements a Node.js production runtime for Turbopack, to first be
used for `next build --turbo`, but that can easily be extended to
support more production use cases (e.g. ncc).

The runtime works differently from the dev runtime we're currently using
for `next dev`: instead of having each chunk register itself against a
global registry when evaluated, chunks export their module factories as
a CommonJS default export. The runtime itself lives in a separate chunk,
and then an "exported chunk" is generated that instantiates runtime
entries and exports a given entry module.

Corresponding Next.js PR:
~~#49942
#50375

---------

Co-authored-by: Tobias Koppers <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants