sea: support embedding assets #50960

joyeecheung · 2023-11-29T00:07:08Z

src: print string content better in BlobDeserializer

When it's a short string, print it inline, otherwise print it
from a separate line. Also add the missing line breaks finally.

sea: support embedding assets

With this patch:

Users can now include assets by adding a key-path dictionary
to the configuration as the assets field. At build time, Node.js
would read the assets from the specified paths and bundle them into
the preparation blob. In the generated executable, users can retrieve
the assets using the sea.getAsset() and sea.getAssetAsBlob() API.

{
  "main": "/path/to/bundled/script.js",
  "output": "/path/to/write/the/generated/blob.blob",
  "assets": {
    "a.jpg": "/path/to/a.jpg",
    "b.txt": "/path/to/b.txt"
  }
}

The single-executable application can access the assets as follows:

const { getAsset } = require('node:sea');
// Returns a copy of the data in an ArrayBuffer
const image = getAsset('a.jpg');
// Returns a string decoded from the asset as UTF8.
const text = getAsset('b.txt', 'utf8');
// Returns a Blob containing the asset without copying.
const blob = getAssetAsBlob('a.jpg');

Drive-by: update the documentation to include a section dedicated
to the injected main script and refer to it as "injected main
script" instead of "injected module" because it's a script, not
a module.

Refs: nodejs/single-executable#68

nodejs-github-bot · 2023-11-29T00:07:13Z

Review requested:

@nodejs/loaders
@nodejs/single-executable
@nodejs/startup

nodejs-github-bot · 2023-11-29T00:20:58Z

CI: https://ci.nodejs.org/job/node-test-pull-request/55991/

nodejs-github-bot · 2023-11-29T02:32:32Z

CI: https://ci.nodejs.org/job/node-test-pull-request/55994/

doc/api/errors.md

src/json_parser.cc

targos · 2023-11-29T07:25:44Z

Should we have a getAssetNames() API or something else that allows the user to list the bundled assets?

doc/api/single-executable-applications.md

joyeecheung · 2023-11-29T14:48:02Z

From #50941 (comment)

What is wrong with saying in docs here is an ArrayBuffer, but if you mess with it you’ll crash? Then as a developer I could just adhere to that and live happily ever after.

One thing we need to decide: should we expose getRawAsset() to allow users reading the asset directly without copying. With a Blob users still have to do some form of copying at the end of the day unless their use case is piping the whole thing. Currently if you get the raw buffer and then wrap it with an ArrayBufferView and then mutate any of the elements, the process would crash due to access protection (probably because the segments postject uses are protected, or I'm not sure if there are unprotected segments we can do this injection with)

pipobscure · 2023-11-29T19:50:28Z

Just a note: I really appreciate this happening. I have previously been using nexe and pkg and therefore know a lot of the pain points of trying to keep current with node. So I just want to give @joyeecheung huge thanks and kudos for moving this forward! THANKS

doc/api/errors.md

aduh95

A few nits

doc/api/single-executable-applications.md

lib/sea.js

test/fixtures/sea/get-asset.js

doc/api/errors.md

anonrig · 2023-12-02T21:32:18Z

doc/api/single-executable-applications.md

@@ -178,14 +178,52 @@ The configuration currently reads the following top-level fields:
  "output": "/path/to/write/the/generated/blob.blob",
  "disableExperimentalSEAWarning": true, // Default: false
  "useSnapshot": false,  // Default: false
-  "useCodeCache": true // Default: false
+  "useCodeCache": true, // Default: false


@nodejs/single-executable I recommend creating a JSON schema and distributing it through nodejs.org, to have better intellisense when this json includes "$schema" parameter. This could help with maintenance and usability.

Do you want to open an issue in https://github.com/nodejs/single-executable/issues instead?

doc/api/single-executable-applications.md

lib/sea.js

src/json_parser.cc

targos · 2023-12-04T18:17:42Z

Just repeating #50960 (comment) in case you missed it:

Should we have a getAssetNames() API or something else that allows the user to list the bundled assets?

doc/api/single-executable-applications.md

src/json_parser.h

src/json_parser.cc

src/node_sea.cc

With this patch: Users can now include assets by adding a key-path dictionary to the configuration as the `assets` field. At build time, Node.js would read the assets from the specified paths and bundle them into the preparation blob. In the generated executable, users can retrieve the assets using the `sea.getAsset()` and `sea.getAssetAsBlob()` API. ```json { "main": "/path/to/bundled/script.js", "output": "/path/to/write/the/generated/blob.blob", "assets": { "a.jpg": "/path/to/a.jpg", "b.txt": "/path/to/b.txt" } } ``` The single-executable application can access the assets as follows: ```cjs const { getAsset } = require('node:sea'); // Returns a copy of the data in an ArrayBuffer const image = getAsset('a.jpg'); // Returns a string decoded from the asset as UTF8. const text = getAsset('b.txt', 'utf8'); // Returns a Blob containing the asset. const blob = getAssetAsBlob('a.jpg'); ``` Drive-by: update the documentation to include a section dedicated to the injected main script and refer to it as "injected main script" instead of "injected module" because it's a script, not a module. PR-URL: nodejs#50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

This patch adds support for `sea.getRawAsset()` which is similar to `sea.getAsset()` but returns the raw asset in an array buffer without copying. Users should avoid writing to the returned array buffer. If the injected section is not marked as writable or not aligned, writing to the raw asset is likely to result in a crash. PR-URL: nodejs#50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Notable changes: build: * (SEMVER-MINOR) build opt to set local location of headers (Michael Dawson) #51525 crypto: * (SEMVER-MINOR) implement crypto.hash() (Joyee Cheung) #51044 * update root certificates to NSS 3.98 (Node.js GitHub Bot) #51794 doc: * add zcbenz to collaborators (Cheng Zhao) #51812 * add lemire to collaborators (Daniel Lemire) #51572 http2: * (SEMVER-MINOR) add h2 compat support for appendHeader (Tim Perry) #51412 * (SEMVER-MINOR) add server handshake utility (snek) #51172 * (SEMVER-MINOR) receive customsettings (Marten Richter) #51323 lib: * (SEMVER-MINOR) move encodingsMap to internal/util (Joyee Cheung) #51044 sea: * (SEMVER-MINOR) support sea.getRawAsset() (Joyee Cheung) #50960 * (SEMVER-MINOR) support embedding assets (Joyee Cheung) #50960 src: * (SEMVER-MINOR) print string content better in BlobDeserializer (Joyee Cheung) #50960 * (SEMVER-MINOR) support multi-line values for .env file (IlyasShabi) #51289 * (SEMVER-MINOR) add `process.loadEnvFile` and `util.parseEnv` (Yagiz Nizipli) #51476 * (SEMVER-MINOR) do not coerce dotenv paths (Tobias Nießen) #51425 stream: * (SEMVER-MINOR) implement `min` option for `ReadableStreamBYOBReader.read` (Mattias Buelens) #50888 util: * (SEMVER-MINOR) add styleText API to text formatting (Rafael Gonzaga) #51850 vm: * (SEMVER-MINOR) support using the default loader to handle dynamic import() (Joyee Cheung) #51244 PR-URL: TODO

Notable changes: build: * (SEMVER-MINOR) build opt to set local location of headers (Michael Dawson) #51525 crypto: * (SEMVER-MINOR) implement crypto.hash() (Joyee Cheung) #51044 * update root certificates to NSS 3.98 (Node.js GitHub Bot) #51794 doc: * add zcbenz to collaborators (Cheng Zhao) #51812 * add lemire to collaborators (Daniel Lemire) #51572 http2: * (SEMVER-MINOR) add h2 compat support for appendHeader (Tim Perry) #51412 * (SEMVER-MINOR) add server handshake utility (snek) #51172 * (SEMVER-MINOR) receive customsettings (Marten Richter) #51323 lib: * (SEMVER-MINOR) move encodingsMap to internal/util (Joyee Cheung) #51044 sea: * (SEMVER-MINOR) support sea.getRawAsset() (Joyee Cheung) #50960 * (SEMVER-MINOR) support embedding assets (Joyee Cheung) #50960 src: * (SEMVER-MINOR) print string content better in BlobDeserializer (Joyee Cheung) #50960 * (SEMVER-MINOR) support multi-line values for .env file (IlyasShabi) #51289 * (SEMVER-MINOR) add `process.loadEnvFile` and `util.parseEnv` (Yagiz Nizipli) #51476 * (SEMVER-MINOR) do not coerce dotenv paths (Tobias Nießen) #51425 stream: * (SEMVER-MINOR) implement `min` option for `ReadableStreamBYOBReader.read` (Mattias Buelens) #50888 util: * (SEMVER-MINOR) add styleText API to text formatting (Rafael Gonzaga) #51850 vm: * (SEMVER-MINOR) support using the default loader to handle dynamic import() (Joyee Cheung) #51244 PR-URL: #51932

When it's a short string, print it inline, otherwise print it from a separate line. Also add the missing line breaks finally. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

With this patch: Users can now include assets by adding a key-path dictionary to the configuration as the `assets` field. At build time, Node.js would read the assets from the specified paths and bundle them into the preparation blob. In the generated executable, users can retrieve the assets using the `sea.getAsset()` and `sea.getAssetAsBlob()` API. ```json { "main": "/path/to/bundled/script.js", "output": "/path/to/write/the/generated/blob.blob", "assets": { "a.jpg": "/path/to/a.jpg", "b.txt": "/path/to/b.txt" } } ``` The single-executable application can access the assets as follows: ```cjs const { getAsset } = require('node:sea'); // Returns a copy of the data in an ArrayBuffer const image = getAsset('a.jpg'); // Returns a string decoded from the asset as UTF8. const text = getAsset('b.txt', 'utf8'); // Returns a Blob containing the asset. const blob = getAssetAsBlob('a.jpg'); ``` Drive-by: update the documentation to include a section dedicated to the injected main script and refer to it as "injected main script" instead of "injected module" because it's a script, not a module. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

This patch adds support for `sea.getRawAsset()` which is similar to `sea.getAsset()` but returns the raw asset in an array buffer without copying. Users should avoid writing to the returned array buffer. If the injected section is not marked as writable or not aligned, writing to the raw asset is likely to result in a crash. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

When it's a short string, print it inline, otherwise print it from a separate line. Also add the missing line breaks finally. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

With this patch: Users can now include assets by adding a key-path dictionary to the configuration as the `assets` field. At build time, Node.js would read the assets from the specified paths and bundle them into the preparation blob. In the generated executable, users can retrieve the assets using the `sea.getAsset()` and `sea.getAssetAsBlob()` API. ```json { "main": "/path/to/bundled/script.js", "output": "/path/to/write/the/generated/blob.blob", "assets": { "a.jpg": "/path/to/a.jpg", "b.txt": "/path/to/b.txt" } } ``` The single-executable application can access the assets as follows: ```cjs const { getAsset } = require('node:sea'); // Returns a copy of the data in an ArrayBuffer const image = getAsset('a.jpg'); // Returns a string decoded from the asset as UTF8. const text = getAsset('b.txt', 'utf8'); // Returns a Blob containing the asset. const blob = getAssetAsBlob('a.jpg'); ``` Drive-by: update the documentation to include a section dedicated to the injected main script and refer to it as "injected main script" instead of "injected module" because it's a script, not a module. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

This patch adds support for `sea.getRawAsset()` which is similar to `sea.getAsset()` but returns the raw asset in an array buffer without copying. Users should avoid writing to the returned array buffer. If the injected section is not marked as writable or not aligned, writing to the raw asset is likely to result in a crash. PR-URL: #50960 Refs: nodejs/single-executable#68 Reviewed-By: Antoine du Hamel <[email protected]> Reviewed-By: Stephen Belanger <[email protected]>

Notable changes: build: * (SEMVER-MINOR) build opt to set local location of headers (Michael Dawson) #51525 crypto: * (SEMVER-MINOR) implement crypto.hash() (Joyee Cheung) #51044 * update root certificates to NSS 3.98 (Node.js GitHub Bot) #51794 doc: * add lemire to collaborators (Daniel Lemire) #51572 * add zcbenz to collaborators (Cheng Zhao) #51812 lib: * (SEMVER-MINOR) move encodingsMap to internal/util (Joyee Cheung) #51044 sea: * (SEMVER-MINOR) support sea.getRawAsset() (Joyee Cheung) #50960 * (SEMVER-MINOR) support embedding assets (Joyee Cheung) #50960 src: * (SEMVER-MINOR) print string content better in BlobDeserializer (Joyee Cheung) #50960 util: * (SEMVER-MINOR) add styleText API to text formatting (Rafael Gonzaga) #51850 vm: * (SEMVER-MINOR) support using the default loader to handle dynamic import() (Joyee Cheung) #51244 PR-URL: #52212

Notable changes: build: * (SEMVER-MINOR) build opt to set local location of headers (Michael Dawson) nodejs#51525 crypto: * (SEMVER-MINOR) implement crypto.hash() (Joyee Cheung) nodejs#51044 * update root certificates to NSS 3.98 (Node.js GitHub Bot) nodejs#51794 doc: * add zcbenz to collaborators (Cheng Zhao) nodejs#51812 * add lemire to collaborators (Daniel Lemire) nodejs#51572 http2: * (SEMVER-MINOR) add h2 compat support for appendHeader (Tim Perry) nodejs#51412 * (SEMVER-MINOR) add server handshake utility (snek) nodejs#51172 * (SEMVER-MINOR) receive customsettings (Marten Richter) nodejs#51323 lib: * (SEMVER-MINOR) move encodingsMap to internal/util (Joyee Cheung) nodejs#51044 sea: * (SEMVER-MINOR) support sea.getRawAsset() (Joyee Cheung) nodejs#50960 * (SEMVER-MINOR) support embedding assets (Joyee Cheung) nodejs#50960 src: * (SEMVER-MINOR) print string content better in BlobDeserializer (Joyee Cheung) nodejs#50960 * (SEMVER-MINOR) support multi-line values for .env file (IlyasShabi) nodejs#51289 * (SEMVER-MINOR) add `process.loadEnvFile` and `util.parseEnv` (Yagiz Nizipli) nodejs#51476 * (SEMVER-MINOR) do not coerce dotenv paths (Tobias Nießen) nodejs#51425 stream: * (SEMVER-MINOR) implement `min` option for `ReadableStreamBYOBReader.read` (Mattias Buelens) nodejs#50888 util: * (SEMVER-MINOR) add styleText API to text formatting (Rafael Gonzaga) nodejs#51850 vm: * (SEMVER-MINOR) support using the default loader to handle dynamic import() (Joyee Cheung) nodejs#51244 PR-URL: nodejs#51932

Notable changes: build: * (SEMVER-MINOR) build opt to set local location of headers (Michael Dawson) #51525 crypto: * (SEMVER-MINOR) implement crypto.hash() (Joyee Cheung) #51044 * update root certificates to NSS 3.98 (Node.js GitHub Bot) #51794 doc: * add lemire to collaborators (Daniel Lemire) #51572 * add zcbenz to collaborators (Cheng Zhao) #51812 lib: * (SEMVER-MINOR) move encodingsMap to internal/util (Joyee Cheung) #51044 sea: * (SEMVER-MINOR) support sea.getRawAsset() (Joyee Cheung) #50960 * (SEMVER-MINOR) support embedding assets (Joyee Cheung) #50960 src: * (SEMVER-MINOR) print string content better in BlobDeserializer (Joyee Cheung) #50960 util: * (SEMVER-MINOR) add styleText API to text formatting (Rafael Gonzaga) #51850 vm: * (SEMVER-MINOR) support using the default loader to handle dynamic import() (Joyee Cheung) #51244 PR-URL: #52212

Notable changes: build: * (SEMVER-MINOR) build opt to set local location of headers (Michael Dawson) nodejs#51525 crypto: * (SEMVER-MINOR) implement crypto.hash() (Joyee Cheung) nodejs#51044 * update root certificates to NSS 3.98 (Node.js GitHub Bot) nodejs#51794 doc: * add zcbenz to collaborators (Cheng Zhao) nodejs#51812 * add lemire to collaborators (Daniel Lemire) nodejs#51572 http2: * (SEMVER-MINOR) add h2 compat support for appendHeader (Tim Perry) nodejs#51412 * (SEMVER-MINOR) add server handshake utility (snek) nodejs#51172 * (SEMVER-MINOR) receive customsettings (Marten Richter) nodejs#51323 lib: * (SEMVER-MINOR) move encodingsMap to internal/util (Joyee Cheung) nodejs#51044 sea: * (SEMVER-MINOR) support sea.getRawAsset() (Joyee Cheung) nodejs#50960 * (SEMVER-MINOR) support embedding assets (Joyee Cheung) nodejs#50960 src: * (SEMVER-MINOR) print string content better in BlobDeserializer (Joyee Cheung) nodejs#50960 * (SEMVER-MINOR) support multi-line values for .env file (IlyasShabi) nodejs#51289 * (SEMVER-MINOR) add `process.loadEnvFile` and `util.parseEnv` (Yagiz Nizipli) nodejs#51476 * (SEMVER-MINOR) do not coerce dotenv paths (Tobias Nießen) nodejs#51425 stream: * (SEMVER-MINOR) implement `min` option for `ReadableStreamBYOBReader.read` (Mattias Buelens) nodejs#50888 util: * (SEMVER-MINOR) add styleText API to text formatting (Rafael Gonzaga) nodejs#51850 vm: * (SEMVER-MINOR) support using the default loader to handle dynamic import() (Joyee Cheung) nodejs#51244 PR-URL: nodejs#51932

CMCDragonkai · 2024-05-18T03:26:16Z

Can this be used for native so binaries and then pass them somehow into process.dlopen?

pipobscure · 2024-05-18T09:17:32Z

Yes that's how. Except you'll need to extract onto disk first because dlopen requires a file.

…

On Sat, 18 May 2024, 04:26 Roger Qiu, ***@***.***> wrote: Can this be used for native so binaries and then pass them somehow into process.dlopen? — Reply to this email directly, view it on GitHub <#50960 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADM5LZKZPT3S46CQAQHFOTZC3C7BAVCNFSM6AAAAAA76SV46KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJYGYZDAMBXGY> . You are receiving this because you commented.Message ID: ***@***.***>

CMCDragonkai · 2024-05-18T10:27:17Z

What about the VFS idea?

pipobscure · 2024-05-18T11:16:10Z

That will make it easy for most files, but native addons will still need to be put on disk, because the dlopen os call simply requires it. VFS would work if it's on yhe OS level (think kernel vfs). But that's outside the scope of what node/sea could do.

…

On Sat, 18 May 2024, 11:27 Roger Qiu, ***@***.***> wrote: What about the VFS idea? — Reply to this email directly, view it on GitHub <#50960 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADM5L6ZM2F7LABRUKRC65DZC4UJ5AVCNFSM6AAAAAA76SV46KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJYG43DMMBRG4> . You are receiving this because you commented.Message ID: ***@***.***>

joyeecheung · 2024-05-21T22:00:05Z

VFS would need to built on top of proper fs hooks which is related to SEA, but also there are many other use cases for such hooks, one previous proposal is nodejs/single-executable#43 (I think there are others, but can't find the links).

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Nov 29, 2023

joyeecheung force-pushed the sea-assets branch from 9ebcf1c to b35c591 Compare November 29, 2023 00:08

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 29, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 29, 2023

joyeecheung force-pushed the sea-assets branch from b35c591 to e3c9f15 Compare November 29, 2023 02:17

joyeecheung mentioned this pull request Nov 29, 2023

sea: add ability to embed auxiliary data asset for use with sea #50941

Closed

joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 29, 2023

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 29, 2023

GeoffreyBooth reviewed Nov 29, 2023

View reviewed changes

doc/api/errors.md Outdated Show resolved Hide resolved

doc/api/errors.md Outdated Show resolved Hide resolved

src/json_parser.cc Outdated Show resolved Hide resolved

targos reviewed Nov 29, 2023

View reviewed changes

doc/api/single-executable-applications.md Outdated Show resolved Hide resolved

joyeecheung force-pushed the sea-assets branch from e3c9f15 to cf7d0a1 Compare November 29, 2023 14:56

pluris reviewed Nov 30, 2023

View reviewed changes

doc/api/errors.md Outdated Show resolved Hide resolved

aduh95 reviewed Dec 1, 2023

View reviewed changes

anonrig reviewed Dec 2, 2023

View reviewed changes