Serverless Externals Plugin

Only include listed node_modules and their dependencies in Serverless.

This plugin helps Serverless package only external modules, and Rollup bundle all the other modules.

Installation

npm install serverless-externals-plugin

or

yarn add serverless-externals-plugin

serverless.yml:

plugins:
  - serverless-externals-plugin

package:
  individually: true

functions:
  handler:
    handler: dist/bundle.handler
    externals:
      report: dist/node-externals-report.json
    package:
      patterns:
        - "!./**"
        - ./dist/bundle.js

rollup.config.js:

import { rollupPlugin as externals } from "serverless-externals-plugin";

export default {
  ...
  output: { file: "dist/bundle.js", format: "cjs" },
  treeshake: {
    moduleSideEffects: "no-external",
  },
  plugins: [
    externals(__dirname, { modules: ["pkg3"] }),
    commonjs(),
    nodeResolve({ preferBuiltins: true, exportConditions: ["node"] }),
    ...
  ],
}

Example

Externals Plugin interacts with both Serverless and with your bundler (Rollup).

Let's say you have two modules in your package.json, pkg2 and pkg3. pkg3 is a module with native binaries, so it can't be bundled.

root
+-- [email protected]
+-- [email protected]
    +-- [email protected]

Because pkg3 can't be bundled, both ./node_modules/pkg3 and ./node_modules/pkg2/node_modules/pkg3 should be included in the bundle. pkg2 can just be bundled, but should import pkg3 as follows: require('pkg2/node_modules/pkg3'). It cannot just do require('pkg3') because pkg3 has a different version than pkg2/node_modules/pkg3.

In the Serverless package, only ./node_modules/pkg3/** and ./node_modules/pkg2/node_modules/pkg3/** should be included, all the other contents of node_modules are already bundled.

Externals Plugin provides a Serverless plugin and a Rollup plugin to support this.

There are other reasons modules can't be bundled. For example, readable-stream and sshpk cannot be bundled due to circular dependency errors.

Configuration

In rollup.config.js:

output: { file: "dist/bundle.js", format: "cjs" },
plugins: [
  externals(__dirname, { modules: ["aws-sdk"], packaging: { exclude: ["aws-sdk"] } }),
  ...
]

This will generate a file called node-externals-report.json next to bundle.js, with the module paths that should be packaged.

It can then be included in serverless.yml:

custom:
  externals:
    report: dist/node-externals-report.json

Configuration object

The configuration object has these options:

• modules: string[]: a list of module names that should be kept external (default [])
• report?: boolean | string: whether to generate a report or report path (default ${distPath}/node-externals-report.json)
• packaging.exclude?: string[]: modules which shouldn't be packaged by Serverless (e.g. ['aws-sdk'], default [])
• packaging.forceIncludeModuleRoots?: string[]: module roots that should always be packaged (e.g. ['node_modules/pg'], default [])
• file?: string: path to a different configuration object

It's also possible to filter on module versions. (e.g. uuid@<8). This uses a semver range.

How it works

Externals Plugin uses Arborist by NPM to analyze the node_modules tree (using loadActual()).

Using the Externals configuration (a list of modules you want to keep external), the Plugin will then build a list of all dependencies that should be kept external. This list will contain the modules in the configuration and all the (non-dev) dependencies, recursively.

In the example, the list will contain both pkg2/node_modules/pkg3 and pkg3.

The Rollup Plugin will then generate a report (e.g. node-externals-report.json) which contains the modules that are actually imported. This file is then used by the Serverless Plugin to generate a list of include patterns.

Report

If you mark both aws-sdk and sshpk as external, but you don't use sshpk, the generated report will look as follows:

node-externals-report.json:

{
  "isReport": true,
  "importedModuleRoots": [
    "node_modules/aws-sdk"
  ],
  "config": {
    "modules": [
      "aws-sdk",
      "sshpk"
    ]
  }
}

Serverless can therefore ignore sshpk and all its dependencies, making the bundle even smaller.

The report is generated by analyzing the files Rollup emits (including dynamic imports).

Rollup Plugin

A fully configured rollup.config.js could look as follows:

import { rollupPlugin as externals } from "serverless-externals-plugin";
import commonjs from "@rollup/plugin-commonjs";
import nodeResolve from "@rollup/plugin-node-resolve";

/** @type {import('rollup').RollupOptions} */
const config = {
  input: "index.js",
  output: {
    file: "bundle.js",
    format: "cjs",
    exports: "named",
    dynamicImportInCjs: false,
  },
  treeshake: {
    moduleSideEffects: "no-external",
  },
  plugins: [
    externals(__dirname, { modules: ["aws-sdk"], packaging: { exclude: ["aws-sdk"] } }),
    commonjs({ strictMode: true, ignoreDynamicRequires: true }),
    nodeResolve({ preferBuiltins: true, exportConditions: ["node"] }),
  ],
};

export default config;

Make sure externals comes before @rollup/plugin-commonjs and @rollup/plugin-node-resolve.

Make sure moduleSideEffects: "no-external" is set. By default, Rollup includes all external modules that appear in the code because they might contain side effects, even if they can be treeshaken. By setting this option Rollup will assume external modules have no side effects.

("no-external" is equivalent to (id, external) => !external)

Preferably, also set exportConditions: ["node"] as an option in the node-resolve plugin. This ensures that Rollup uses Node's resolution algorithm, so that packages like uuid can be bundled.

Next to that, in rare cases setting strictMode: true in the commonjs plugin can help bundling some modules that depend on the order of requires, like aws-sdk when only importing aws-sdk/clients/dynamodb.

Implementation

The Rollup plugin provides a resolveId function to Rollup. For every import (e.g. require('pkg3')) in your source code, Rollup will ask the Externals Plugin whether the import is external, and where to find it.

The Plugin will look for the import in the Arborist graph, and if it's declared as being external it will return the full path to the module that's being imported (e.g. pkg2/node_modules/pkg3).

node-externals.json

It's also possible to add the externals config to a file, called node-externals.json.

In node-externals.json:

{
  "modules": [
    "pkg3"
  ]
}

In rollup.config.js:

plugins: [
  externals(__dirname, { file: 'node-externals.json' }),
  ...
]

Dynamic requires

If your code or one of your Node modules does the following:

require("p" + "g");
// or
import("p" + "g");

Then this plugin will not be able to detect which modules should be packaged. You can force include them by using packaging.forceIncludeModuleRoots:

...
packaging: {
  forceIncludeModuleRoots: ["node_modules/pg"]
}

The plugin will then treat node_modules/pg as if it was imported directly in the bundle. It will also include all the dependencies.

For example: in knex, an SQL builder, pg is a peer dependency. Inside knex it is imported as follows:

// knex/lib/index.js
const resolvedClientName = resolveClientNameWithAliases(clientName);
Dialect = require(`./dialects/${resolvedClientName}/index.js`);

// knex/lib/dialects/postgres/index.js
require('pg');

If you're using knex, you'd have to force include node_modules/pg. If you want to bundle knex, you would also have to enable ignoreDynamicRequires in your rollup.config.js:

commonjs({ ignoreDynamicRequires: true }),

This will make sure the require call is not changed.

Another solution is to add knex to the list of externals. In that case the whole node_modules/knex folder will be uploaded, and none of its code will be transformed.

Rollup sometimes keeps await import expressions in the bundle, which might cause import issues if the module is not commonjs. To fix this, you can add the following to your rollup.config.js:

output: {
  dynamicImportInCjs: false
}

Usage in monorepos/workspaces

If your serverless project is a workspace within a larger monorepo, this is also supported, although not yet fully tested.

For example, in apps/lambdas/rollup.config.js:

const root = path.resolve(__dirname, "../..")
const workspaceName = "main-app"

...

plugins: [
  externals([root, workspaceName], { modules: ["undici"] }),
  ...
]

If undici is installed the monorepo root, the serverless plugin will generate include patterns as follows:

!./node_modules/**
!../../node_modules/**
../../node_modules/undici/**
!../../node_modules/undici/node_modules
../../node_modules/busboy/**
!../../node_modules/busboy/node_modules
../../node_modules/streamsearch/**
!../../node_modules/streamsearch/node_modules

Due to the way packaging in serverless works, in the final package, the included modules from the root node_modules will be merged with the included workspace node_modules, which is exactly what we want.

Caveats

Externals with side effects

It's unlikely, but if you have external modules with side effects (like polyfills), make sure to configure Rollup properly.

NOTE: This only applies to external modules. You should probably bundle your polyfills.

import "some-external-module"; // this doesn't work, Rollup will treeshake it away

As Rollup will remove external modules with side effects, make sure to add something like this to the Rollup config:

treeshake: {
  moduleSideEffects: (id, external) => !id.test(/some-external-module/) || !external
}

Only one node_modules supported

This plugin doesn't have support for analyzing multiple node_modules folders. If you have more node_modules folders on your NODE_PATH (e.g. from a Lambda layer), you can still use the external field of Rollup.

Keeping aws-sdk excluded

As the aws-sdk node module is included by default in Lambdas, you can add packaging.exclude: ["aws-sdk"] to exclude it from the Serverless package. Note that this is not recommended because of possible version differences. (Check the aws-sdk version included in runtimes here)

All subdependencies marked as external

When listing modules as external, all their subdependencies will also be marked as external.

For example:

// rollup.config.js
plugins: [
  externals(__dirname, { modules: ["botkit"] }),
  ...
]

// lambda.js
const express = require('express');
const serverlessHttp = require('serverless-http');
const app = express();

module.exports.handler = serverlessHttp(app);

In the resulting bundle, express will not be bundled. This is because botkit also depends on express, and is therefore marked as external. botkit itself and the rest of its subdependencies will be filtered out of the modules to be uploaded.

It is therefore recommended to limit the length of the modules array to only the necessary. In that way you can achieve the smallest bundles.

Todo

• Ensure compatibility with Serverless Jetpack or speedup packaging somehow
• Webpack plugin
• Esbuild plugin
• Layer support
• Look into externalizing single files in a module (e.g. .node files), and bundling the rest
• Yarn PnP support
• Pre-calculate actually used top-level externals to solve last caveat

Motivation

I wanted to include Cheerio/JSDom and AWS SDK in a Typescript project, but neither could be bundled because of obscure errors, so they needed to be external. To reduce package size, I didn't want to make every module external. Manually looking up a module and adding its dependencies to rollup.config.js and serverless.yml is simply too much work. This plugin makes this much easier.

Credits

Some Serverless-handling code was taken from Serverless Jetpack. Also inspired by Serverless Plugin Include Dependencies and Webpack Node Externals