Skip to content

Commit

Permalink
docs: rfc for db-console bazel build
Browse files Browse the repository at this point in the history
Described proposal and changes required for
`pkg/ui` UI projects to establish Bazel build
workflow and integration with other bazel
rules.

Release note: None
  • Loading branch information
koorosh committed Oct 20, 2021
1 parent 9013165 commit 0bcba20
Show file tree
Hide file tree
Showing 3 changed files with 184 additions and 0 deletions.
92 changes: 92 additions & 0 deletions docs/RFCS/20210423_db-cosole-bazel-build.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
- Feature Name:
- Status: draft
- Start Date: 2021-04-23
- Authors: Andrii Vorobiov
- RFC PR: [#65924](https://github.com/cockroachdb/cockroach/pull/65924)
- Cockroach Issue:

# Summary

Migration from Make builds to Bazel build tool for the entire project is one of OKRAs which is aimed to
deliver the product in a timely and sustainable manner. This proposal outlines a subset of changes required to perform
migration to Bazel build for DB Console (and UI related packages).

# Motivation

Migration should not affect the current build process with `Make` so it can be gradual (without
breaking changes).
The migration process can be broken down into the following parts:
- establish Bazel build for Db Console (with support for both OSS and CCL licensing);
- establish Bazel build and publishing artifacts to NPM registry for `cluster-ui` package;
- build `crdb-protobufjs-client` package for both CCL and OSS licences.

Bazel builds (comparing to Make build) allow defining granular steps which can be run independently for particular
package instead of the entire project, which in turn reduces the build time and a chain of unnecessary rebuilds for
unchanged parts of code.

Github `Bazel` project already includes following tasks related to migration to Bazel for Db Console
- [build: Bazelize UI](https://github.com/cockroachdb/cockroach/issues/56068)
- [build: Get built UI to bindata (bazel)](https://github.com/cockroachdb/cockroach/issues/56069)
- [build,bazel: Implement JS protobuf support for Bazel](https://github.com/cockroachdb/cockroach/issues/59329)
- [build: Get bazel building both ccl and oss builds](https://github.com/cockroachdb/cockroach/issues/56071)

# Technical design

Current UI project has following dependencies

![DbConsole project dependencies](images/db-console-bazel-build.png?raw=true)

where:
- `*.proto` files are set of protobuf files that are required to build JS protobuf client code
- `ccl/oss` specifies build condition, whether the project is built for CCL or OSS licence
- `ccl protos/oss protos` - autogenerated JS protobuf client code
- `db-console` is main `ui` project
- `cluster-ui` is a nested `ui/cluster-ui` JS project
- `ui/(ccl|oss)/bidndata.go` - autogenerated Go sources that embed JS bundled files

#### Dependency graph of bazel rules:

![Bazel rules dependencies](images/db-console-bazel-build-dependency-graph.svg)

Bazel build pipeline for UI can be broken down in following steps:
- Build protobuf js client code [db-console: bazel build for protobufjs clients #64065](https://github.com/cockroachdb/cockroach/pull/64065)
- `cluster-ui` standalone build (react + webpack project)
- `db-console` standalone build (react + webpack project)
- embed JS assets into go libraries (`pkg/ui/distoss/distoss.go` and `pkg/ui/distccl/distccl.go`)

### Generate protobuf JS client code
Protobuf CCL and OSS packages are considered as an independent bazel builds. The installation of `protobufjs` package
requires additional steps to install required dependencies for protobufjs-cli tool. This problem persists in Makefile
as well and will be fixed with major release of protobufjs ver 7. Installation of dependencies and binding protobuf cli
is done according to example provided here: https://github.com/bazelbuild/rules_nodejs/tree/stable/examples/protobufjs.

### Build `db-console` and `cluster-ui` packages
`db-console` and `cluster-ui` are typescript packages which have their own webpack configuration to build production
bundle, run dev server and watch for changes.
To build these projects, additional project changes required:
- change relative paths for input and output files to be absolute paths in webpack configs because bazel resolves
relative paths from workspace root dir (which is root of entire `cockroach` project)
- optional dependencies that are required only for web development and was defined as a separate NPM package (with its
own package.json file in `pkg/ui/opts`) is removed and dependencies are moved in `pkg/ui/workspaces/db-console/package.json`
under `optionalDependencies` section. Along with this, installation of NPM packages is performed with `--ignore-optional`
flag to prevent installation of optional during production builds (ie `yarn install --ignore-optional`).

### Refactor project structure to use Yarn workspaces
Existing project structure had multiple `node_modules` directories for different purposes. With Bazel, every
`node_modules` is treated as separate external workspaces and with such configuration it restricts to use dependencies
from single external workspace.

To overcome this limitation, project structure is refactored to use Yarn workspaces, it allows to hoist all common
dependencies to the top level in a single `node_modules` which is considered as a single external workspace for bazel.
- `rules_nodejs` doesn't support Yarn workspaces (https://github.com/bazelbuild/rules_nodejs/issues/266) but for now the
only limitation is that project specific dependencies are included in main workspaces rather than in external workspace.
- all hoisted dependencies can be accessed from single external workspace (`@npm`) and project specific dependencies are
bounded with `filegroup`'s.

### Embeding JS assets into go libraries
`go:embed` directive (available starting from ver 1.6) is used instead of `go-bindata` for embedding JS assets into
go libraries. It simplifies build process for both Make and Bazel builds because it eliminates running external
commands to generate go sources with embedded data.

# Unresolved questions
- TBD
92 changes: 92 additions & 0 deletions docs/RFCS/images/db-console-bazel-build-dependency-graph.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/RFCS/images/db-console-bazel-build.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 0bcba20

Please sign in to comment.