@osd/pm is a project management tool inspired by Lerna, which enables sharing
code between OpenSearch Dashboards and OpenSearch Dashboards plugins.
To run @osd/pm, go to OpenSearch Dashboards root and run yarn osd.
Long-term we want to get rid of Webpack from production (basically, it's causing
a lot of problems, using a lot of memory and adding a lot of complexity).
Ideally we want each plugin to build its own separate production bundles for
both server and UI. To get there all OpenSearch Dashboards plugins need to
be able to build their production bundles separately from OpenSearch Dashboards , which means
they need to be able to depend on code from OpenSearch Dashboards without import-ing random
files directly from the OpenSearch Dashboards source code.
From a plugin perspective there are two different types of OpenSearch Dashboards dependencies:
runtime and static dependencies. Runtime dependencies are things that are
instantiated at runtime and that are injected into the plugin, for example
config and opensearch clients. Static dependencies are those dependencies
that we want to import. opensearch-eslint-config-opensearch-dashboards is one example of this, and
it's actually needed because eslint requires it to be a separate package. But we
also have dependencies like datemath, flot, eui and others that we
control, but where we want to import them in plugins instead of injecting them
(because injecting them would be painful to work with). (Btw, these examples
aren't necessarily a part of the OpenSearch Dashboards repo today, they are just meant as
examples of code that we might at some point want to include in the repo while
having them be importable in OpenSearch Dashboards plugins like any other npm package)
Another reason we need static dependencies is that we're starting to introduce TypeScript into OpenSearch Dashboards , and to work nicely with TypeScript across plugins we need to be able to statically import dependencies. We have for example built an observable library for OpenSearch Dashboards in TypeScript and we need to expose both the functionality and the TypeScript types to plugins (so other plugins built with TypeScript can depend on the types for the lib).
However, even though we have multiple packages we don't necessarily want to
npm publish them. The ideal solution for us is being able to work on code
locally in the OpenSearch Dashboards repo and have a nice workflow that doesn't require
publishing, but where we still get the value of having "packages" that are
available to plugins, without these plugins having to import files directly from
the OpenSearch Dashboards folder.
Basically, we just want to be able to share "static code" (aka being able to
import) between OpenSearch Dashboards and OpenSearch Dashboards plugins. To get there we need tooling.
@osd/pm is a tool that helps us manage these static dependencies, and it
enables us to share these packages between OpenSearch Dashboards and OpenSearch Dashboards plugins. It also
enables these packages to have their own dependencies and their own build
scripts, while still having a nice developer experience.
For packages that are referenced within the OpenSearch Dashboards repo itself, we are leveraging Yarn's workspaces feature. This allows yarn to optimize node_modules within the entire repo to avoid duplicate modules by hoisting common packages as high in the dependency tree as possible.
To reference a package from within the OpenSearch Dashboards repo, simply use the current
version number from that package's package.json file. Then, running yarn osd bootstrap will symlink that package into your dependency tree. That means
you can make changes to @osd/i18n and immediately have them available
in OpenSearch Dashboards itself. No npm publish needed anymore — OpenSearch Dashboards will always rely
directly on the code that's in the local packages.
For external plugins, referencing packages in OpenSearch Dashboards relies on
link: style dependencies in Yarn. With link: dependencies you specify the
relative location to a package instead of a version when adding it to
package.json. For example:
"@osd/i18n": "link:packages/osd-i18n"
Now when you run yarn it will set up a symlink to this folder instead of
downloading code from the npm registry. This allows external plugins to always
use the versions of the package that is bundled with the OpenSearch Dashboards version they
are running inside of.
"@osd/i18n": "link:../../opensearch-dashboards/packages/osd-date-math"
This works because we moved to a strict location of OpenSearch Dashboards plugins,
./plugins/{pluginName} inside of OpenSearch Dashboards , or ../opensearch-dashboards-extra/{pluginName}
relative to OpenSearch Dashboards . This is one of the reasons we wanted to move towards a setup
that looks like this:
opensearch
└── opensearch-dashboards
└── plugins
├── opensearch-dashboards-example
Relying on link: style dependencies means we no longer need to npm publish
our OpenSearch Dashboards specific packages. It also means that plugin authors no longer need
to worry about the versions of the OpenSearch Dashboards packages, as they will always use the
packages from their local OpenSearch Dashboards .
Now, instead of installing all the dependencies with just running yarn you use
the @osd/pm tool, which can install dependencies (and set up symlinks) in
all the packages using one command (aka "bootstrap" the setup).
To bootstrap OpenSearch Dashboards :
yarn osd bootstrap
By default, @osd/pm will bootstrap all packages within OpenSearch Dashboards , plus all
OpenSearch Dashboards plugins located in ./plugins or ../opensearch-dashboards-extra. There are several
options for skipping parts of this, e.g. to skip bootstrapping of OpenSearch Dashboards
plugins:
yarn osd bootstrap --skip-opensearch-dashboards-plugins
Or just skip few selected packages:
yarn osd bootstrap --exclude @osd/pm --exclude @osd/i18n
For more details, run:
yarn osd
Bootstrapping also calls the osd:bootstrap script for every included project.
This is intended for packages that need to be built/transpiled to be usable.
Bootstrapping, by default, applies a strict single-version validation where it requires all the dependencies defined
more than once to have the same version-range in the package.json files of Dashboards, packages, and plugins. If a
violation is identified, bootstrapping terminates with an error. This behavior can be controlled using the
--single-version switch. Using any switch other than the default can result in the installation of versions of the
dependencies that were never tested and this could lead to unexpected results.
yarn osd bootstrap --single-version=loose
In loose mode, bootstrapping reconciles the various versions installed as a result of having multiple ranges for a
dependency, by choosing one that satisfies all said ranges. Even though installing the chosen version updates the
yarn.lock files, no package.json changes would be needed.
yarn osd bootstrap --single-version=force
In force mode, bootstrapping acts like loose for each dependency. If despite that, a suitable version was not found,
it switches to behave like the brute-force mode (see below).
yarn osd bootstrap --single-version=brute-force
In brute-force mode, bootstrapping chooses the newest of the various versions installed, irrespective of whether it
satisfies any of the ranges. Installing the chosen version, bootstrapping updates the yarn.lock files and applies a
range, in the form of ^<version>, to all package.json files that declared the dependency.
yarn osd bootstrap --single-version=ignore
In ignore mode, bootstrapping behaves very similar to the strict mode by showing errors when different ranges of a
package are marked as dependencies, but without terminating.
Some times you want to run the same script across multiple packages and plugins,
e.g. build or test. Instead of jumping into each package and running
yarn build you can run:
yarn osd run build
And if needed, you can skip packages in the same way as for bootstrapping, e.g.
with --exclude and --skip-opensearch-dashboards-plugins:
yarn osd run build --exclude opensearch-dashboards
During development you can also use osd to watch for changes. For this to work
package should define osd:watch script in the package.json:
yarn osd watch
By default osd watch will sort all packages within OpenSearch Dashboards into batches based on
their mutual dependencies and run watch script for all packages in the correct order.
As with any other osd command, you can use --include and --exclude filters to watch
only for a selected packages:
yarn osd watch --include @osd/pm --include opensearch-dashboards
The production build process relies on both the Grunt setup at the root of the
OpenSearch Dashboards project and code in @osd/pm. The full process is described in
tasks/build/packages.js.
Packages that are only compiled to CommonJS for consumption by server code, simply use a tsconfig.json
that extends the tsconfig.base.json found at the root of the project. Similarly, packages that need to be
compiled into ES modules, to be consumed in the browser and benefit from tree-shaking, use atsconfig.json
that extends the tsconfig.browser.json available at the root of the project. However, some packages need
to be compiled for consumption by both server code and the browser. This can be achieved by retaining the
tsconfig.json and adding the following block to the package.json of the package:
"@osd/pm": {
"node": true,
"web": true
}With that, the relevant presets from @osd/babel-preset are applied to the code to produce bootstrap and
production build artifacts for the selected targets. It is acceptable to have only one target and that is
the same as setting the appropriate extends in the tsconfig.json.
Targeted builds call babel on the src folder and place artifacts into target/node and target/web
folders after calling tsc in the package root to generate type definitions.
This package is run from OpenSearch Dashboards root, using yarn osd. This will run the
"pre-built" (aka built and committed to git) version of this tool, which is
located in the dist/ folder. This will also use the included version of Yarn
instead of using your local install of Yarn.
If you need to build a new version of this package, run yarn build in this
folder.
Even though this file is generated we commit it to OpenSearch Dashboards , because it's used before dependencies are fetched (as this is the tool actually responsible for fetching dependencies).
While exploring the approach to static dependencies we built PoCs using npm 5
(which symlinks packages using file: dependencies), Yarn
workspaces, Yarn (using link: dependencies), and
Lerna.
In the end we decided to build our own tool, based on Yarn, and link:
dependencies, and workspaces. This gave us the control we wanted, and it fits
nicely into our context (e.g. where publishing to npm isn't necessarily
something we want to do).
When you add a dependency like "foo": "file:../../opensearch-dashboards/packages/foo", both
npm<5 and yarn copies the files into the node_modules folder. This means you
can't easily make changes to the plugin while developing. Therefore this is a
no-go.
In npm5 file: dependencies changed to symlink instead of copy the files. This
means you can have a nicer workflow while developing packages locally. However,
we hit several bugs when using this feature, and we often had to re-run
npm install in packages. This is likely because we used an early version of
the new file: dependencies in npm5.
This is the same feature as file: dependencies in npm5. However, we did not
hit any problems with them during our exploration.
Enables specifying multiple "workspaces" (aka packages/projects) in
package.json. When running yarn from the root, Yarn will install all the
dependencies for these workspaces and hoist the dependencies to the root (to
"deduplicate" packages). However:
Workspaces must be children of the workspace root in term of folder hierarchy. You cannot and must not reference a workspace that is located outside of this filesystem hierarchy.
So Yarn workspaces requires a shared root, which (at least currently) doesn't fit OpenSearch Dashboards , and it's therefore a no-go for now.
Lerna is based on symlinking packages (similarly to the link
feature which exists in both npm and Yarn, but it's not directly using that
feature). It's a tool built specifically for managing JavaScript projects with
multiple packages. However, it's primarily built (i.e. optimized) for monorepo
libraries, so it's focused on publishing packages and other use-cases that are
not necessarily optimized for our use-cases. It's also not ideal for the setup
we currently have, with one app that "owns everything" and the rest being
packages for that app.
See the vendor readme.