Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamline build processes with esbuild #10399

Merged
merged 43 commits into from
Jul 19, 2022
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
ff559c7
build workers, third party, and combine with eslint
ggetz May 11, 2022
4bb8024
combine output
ggetz May 11, 2022
355dd42
Updated build process
ggetz May 24, 2022
54ef709
Spec and worker fixes
ggetz May 25, 2022
eb60fbf
cleanup build tasks
ggetz May 25, 2022
f6cf110
clean up gulp tasks and naming
ggetz May 25, 2022
bddf66e
Remove requireJS from LICENSE and ThirdParty.json
ggetz May 25, 2022
e71c886
Update build guide
ggetz May 25, 2022
eb87e96
Tweak travis file
ggetz May 26, 2022
44b9610
karma specs and coverage
ggetz May 27, 2022
0871e6a
sourcemap by default for dev only
ggetz May 27, 2022
27bbdbf
Tweaks for CI
ggetz May 27, 2022
ec317da
Sandcastle
ggetz May 27, 2022
16a9c80
Fix up build output names and apps
ggetz May 27, 2022
7ac3fd0
coverage
ggetz Jun 2, 2022
d867651
Tweak specs
ggetz Jun 2, 2022
290bed0
Tweak build guide
ggetz Jun 2, 2022
3938feb
Tweak tests
ggetz Jun 2, 2022
4ecefb5
Merge branch 'main' into build
ggetz Jun 6, 2022
867ab1a
Cleanup documentation
ggetz Jun 6, 2022
6404f52
Test cleanup
ggetz Jun 6, 2022
1f99b62
Cleanup
ggetz Jun 6, 2022
018aca6
Tweak watched files
ggetz Jun 6, 2022
3293a5b
Cleanup coverage
ggetz Jun 8, 2022
4cf8ba1
Legal comments
ggetz Jun 9, 2022
318c276
Cleanup rollup dependencies
ggetz Jun 14, 2022
6a31cf1
Clarify documentation
ggetz Jun 15, 2022
8d7faae
Merge branch 'main' into build
ggetz Jun 15, 2022
312cb98
Fix build error and warning
ggetz Jun 22, 2022
9905c12
Merge branch 'main' into build
ggetz Jun 22, 2022
1fbc968
2020
ggetz Jun 22, 2022
213cdfc
Rollback script naming changes
ggetz Jun 22, 2022
6ae6cda
prettier
ggetz Jun 30, 2022
8bc8355
Remove worker changes
ggetz Jul 6, 2022
3a8e5c8
Remove extraneous "the"
ggetz Jul 12, 2022
0a19ae3
Merge branch 'main' into build
ggetz Jul 13, 2022
095033d
Update CHANGES.md
ggetz Jul 13, 2022
17babe9
Tweak workers, remove pervasive spread operations
ggetz Jul 13, 2022
ad0dc45
prettier
ggetz Jul 13, 2022
24c4590
Merge branch 'main' into build
ggetz Jul 15, 2022
d6b9a96
Fix typo
ggetz Jul 15, 2022
c575a3f
Feedback update
ggetz Jul 19, 2022
7fd59fd
Tweak node target
ggetz Jul 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 0 additions & 19 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,27 +25,8 @@ Thumbs.db
!/Source/Workers/transferTypedArrayTest.js

Source/ThirdParty/_commonjsHelpers*
Source/ThirdParty/Autolinker.js
Source/ThirdParty/bitmap-sdf.js
Source/ThirdParty/dompurify.js
Source/ThirdParty/earcut.js
Source/ThirdParty/draco_decoder.wasm
Source/ThirdParty/Workers/draco_decoder_nodejs.js
Source/ThirdParty/grapheme-splitter.js
Source/ThirdParty/jsep.js
Source/ThirdParty/kdbush.js
Source/ThirdParty/ktx-parse.js
Source/ThirdParty/lerc.js
Source/ThirdParty/mersenne-twister.js
Source/ThirdParty/meshoptimizer.js
Source/ThirdParty/nosleep.js
Source/ThirdParty/pako.js
Source/ThirdParty/protobufjs.js
Source/ThirdParty/rbush.js
Source/ThirdParty/topojson.js
Source/ThirdParty/Tween.js
Source/ThirdParty/Uri.js
Source/ThirdParty/zip.js
Source/ThirdParty/Workers/pako_inflate.min.js
Source/ThirdParty/Workers/pako_deflate.min.js
Source/ThirdParty/Workers/z-worker-pako.js
Expand Down
8 changes: 4 additions & 4 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ script:
- npm --silent run eslint
- npm --silent run prettier-check

- npm --silent run build
- npm --silent run build -- --node
- npm --silent run build-ts

- npm --silent run coverage -- --browsers FirefoxHeadless --webgl-stub --failTaskOnError --suppressPassed

- travis_wait 20 npm --silent run makeZipFile -- --concurrency 1
- travis_wait 20 npm --silent run make-zip -- --concurrency 1
- npm pack &> /dev/null

- npm --silent run buildApps
- npm --silent run build-specs
- npm --silent run build-apps

- npm --silent run deploy-s3 -- -b cesium-dev -d cesium/$TRAVIS_BRANCH --confirm -c 'no-cache'
- npm --silent run deploy-status -- --status success --message Deployed
Expand Down
6 changes: 4 additions & 2 deletions Apps/CesiumViewer/CesiumViewer.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
window.CESIUM_BASE_URL = "../../Source/";
if (window.CESIUM_BASE_URL === undefined) {
window.CESIUM_BASE_URL = "../../Build/CesiumUnminified/";
}

import {
Cartesian3,
Expand All @@ -16,7 +18,7 @@ import {
Viewer,
viewerCesiumInspectorMixin,
viewerDragDropMixin,
} from "../../Source/Cesium.js";
} from "../../Build/CesiumUnminified/index.js";

function main() {
/*
Expand Down
1 change: 1 addition & 0 deletions Apps/CesiumViewer/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
window.CESIUM_BASE_URL = ".";
4 changes: 2 additions & 2 deletions Apps/Sandcastle/load-cesium-es6.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// This file loads the unbuilt ES6 version of Cesium
// into the global scope during local developmnet
window.CESIUM_BASE_URL = "../../../Source/";
import * as Cesium from "../../Source/Cesium.js";
window.CESIUM_BASE_URL = "../../../Build/CesiumUnminified/";
import * as Cesium from "../../Build/CesiumUnminified/index.js";
window.Cesium = Cesium;

// Since ES6 modules have no guaranteed load order,
Expand Down
5 changes: 0 additions & 5 deletions Apps/Sandcastle/standalone.html
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,6 @@
<script type="text/javascript" src="Sandcastle-header.js"></script>
<script type="text/javascript" src="Sandcastle-client.js"></script>
<script type="text/javascript" src="ThirdParty/pako.min.js"></script>
<script
type="text/javascript"
src="../../Build/CesiumUnminified/Cesium.js"
nomodule
></script>
<script type="module" src="load-cesium-es6.js"></script>
<script type="text/javascript" src="Sandcastle-helpers.js"></script>
<style>
Expand Down
5 changes: 0 additions & 5 deletions Apps/Sandcastle/templates/bucket-requirejs.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,6 @@
/>
<title>Cesium Demo</title>
<script type="text/javascript" src="../Sandcastle-header.js"></script>
<script
type="text/javascript"
src="../../../Build/CesiumUnminified/Cesium.js"
nomodule
></script>
<script type="module" src="../load-cesium-es6.js"></script>
</head>
<body
Expand Down
81 changes: 47 additions & 34 deletions Documentation/Contributors/BuildGuide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
- [Quickstart](#quickstart)
- [Get the Code](#get-the-code)
- [Build the Code](#build-the-code)
- [Development Server](#development-server)
- [Build Output](#build-output)
- [Build Scripts](#build-scripts)
- [Travis and Continuous Integration](#travis-and-continuous-integration)
- [Configure a Different S3 Bucket](#configure-a-different-s3-bucket)
Expand All @@ -29,8 +31,6 @@

4. Navigate to : [` http://localhost:8080/`](http://localhost:8080)

_NOTE: If you change branches, you might have to rebuild._

---

## Get the Code
Expand All @@ -56,7 +56,7 @@ _NOTE: If you change branches, you might have to rebuild._

Prerequisites:

- Install [Node.js](http://nodejs.org/) on your system. Building Cesium requires Node 6.x or newer.
- Install [Node.js](http://nodejs.org/) on your system. Building Cesium requires Node 14.x or newer.
ggetz marked this conversation as resolved.
Show resolved Hide resolved

Cesium uses [npm modules](https://docs.npmjs.com/getting-started/what-is-npm) for development, so after syncing, you need to run `npm install` from the Cesium root directory:

Expand All @@ -70,6 +70,12 @@ Once all modules have been installed, run `npm run build` to actually build the
npm run build
```

Alternatively, if you want to edit source files and see the changes reflected for testing, use `npm run build-watch`.

```
npm run build-watch
```

Cesium ships with a simple HTTP server for testing, run `npm start` after building to use it:

```
Expand All @@ -83,12 +89,7 @@ Then browse to [http://localhost:8080/](http://localhost:8080/). The landing pag
- **Test Suites** : tests using [Jasmine](https://jasmine.github.io/). [Testing guide here.](https://github.com/CesiumGS/cesium/blob/main/Documentation/Contributors/TestingGuide/README.md#testing-guide)
- **Documentation** : reference documentation built from source. [Documentation guide here.](https://github.com/CesiumGS/cesium/blob/main/Documentation/Contributors/DocumentationGuide/README.md#documentation-guide)

Cesium can be used in two different ways. Cesium can be either a set of modules using [Asynchronous Module Definition (AMD)](https://github.com/amdjs/amdjs-api/wiki/AMD), or it can be built as one combined file containing all modules. The basics:

- `npm run build` will build AMD Cesium. This also builds Cesium Viewer and Sandcastle.
- `npm run minifyRelease` creates the built version of Cesium. This also builds Hello World.

Read the complete list of build scripts below for more details.
### Development Server

By default, the server only allows connections from your local machine. To allow connections from other machines, pass
the `--public` option to npm. Note the extra `--` is intentional and required by npm.
Expand All @@ -97,12 +98,22 @@ the `--public` option to npm. Note the extra `--` is intentional and required by
npm start -- --public
```

The development server has a few other options as well, which you can see by pasing the `--help` parameter:
The development server has a few other options as well, which you can see by passing the `--help` parameter:

```
npm start -- --help
```

### Build Output

Cesium can be used a few different ways. Cesium can be either a set of platform-generic ESM modules generic to browser, or bundled targeting the browser or NodeJS environment.

- [ESM (ECMAScript modules)](https://nodejs.org/api/esm.html) - Standard for packaging JS code which are supported by most browsers and NodeJS. Modules use `import` and `export` statements. Unprocessed, individual modules are available in the `Source` directory, accessible by importing `Source/Cesium.js`; A single pre-processed bundle by importing `Build/Cesium/index.js`.
- [IIFE (immediately-invoked function expression)](https://developer.mozilla.org/en-US/docs/Glossary/IIFE) - A pre-processed bundle and optimized for the browser, which defines a `Cesium` global variable upon loading `Build/Cesium/Cesium.js`.
- [CJS (CommonJS)](https://nodejs.org/api/modules.html) - A pre-processed, bundled module packaged for running in NodeJS accessible by requiring `index.cjs`.

Read the complete list of build scripts and options below for more details.

While you can use the editor of your choice to develop Cesium, certain files, such as `glsl` and new tests, require that
the `build` task be executed in order for the changes to take effect. You can use the `build-watch` script to have this
happen automatically.
Expand All @@ -120,39 +131,41 @@ npm run [target-name]
Here's the full set of scripts and what they do.

- **Build scripts** -- build and package the source code and documentation
- `build` - A fast, developer-oriented build that prepares the source tree for use as standard [Asynchronous Module Definition (AMD)](https://github.com/amdjs/amdjs-api/wiki/AMD) modules, suitable for running tests and most examples (some Sandcastle examples require running `combine`). Run this when a GLSL shader is changed since the .glsl file is converted to a .js file with a string for the GLSL source. This runs automatically when saving files in Eclipse.
- `build-watch` - A never-ending task that watches your file system for changes to Cesium and runs `build` on the source code as needed.
- `combine` - Runs `build`, plus the [the RequireJS optimizer](http://requirejs.org/docs/optimization.html) to combine Cesium and [the Almond AMD loader](http://requirejs.org/docs/faq-optimization.html#wrap) to produce all-in-one files in the `Build/Cesium` directory that exposes the entire Cesium API attached to a single global `Cesium` object. This version is useful if you don't want to use the modules directly with a standard AMD loader.
- `minify` - Runs `combine`, plus [minifies](<http://en.wikipedia.org/wiki/Minification_(programming)>) Cesium.js.
- `combineRelease` - Runs `combine`, plus uses the optimizer to remove debugging code that validates function input and throws DeveloperErrors. The removed sections are marked with `//>>includeStart('debug', pragmas.debug);` blocks in the code.
- `minifyRelease` - Runs `minify`, and removes debugging code.
- `requirejs` - Used internally by the build system and can not be called directly.
- `buildApps` - Builds the example applications (such as Cesium Viewer) to produce self-contained, minified, deployable versions in the `Build` directory. This script requires a release build of Cesium, run the `release` script to build one if needed.
- `generateDocumentation` - Generates HTML documentation in `Build/Documentation` using [JSDoc 3](https://github.com/jsdoc3/jsdoc). More [details here](https://github.com/rahwang/cesium/tree/main/Documentation/Contributors/DocumentationGuide).
- `build` - A fast, developer-oriented build that pre-processes ESM modules, suitable for running tests and most examples. Run this when a GLSL shader is changed since the .glsl file is converted to a .js file with a string for the GLSL source. The build output, including an external sourcemap, will default to `Build/CesiumUnminified`.
- `--minify` - [Minifies](<http://en.wikipedia.org/wiki/Minification_(programming)>) the output for optimized loading. Specifying this option will output to `Build/Cesium`.
- `--removePragmas` - Optimizes the output by removing debugging code that validates function input and throws `DeveloperError`s. The removed sections are marked with `//>>includeStart('debug', pragmas.debug);` blocks in the code.
- `--node` - Bundles an `index.cjs` module targeted for use in NodeJS
- `build-watch` - A never-ending task that watches your file system for changes to Cesium and builds the source code as needed. All `build` options are also available for this task.
- `build-apps` - Builds the example applications (such as Cesium Viewer) to produce self-contained, minified, deployable versions in the `Build` directory.
- `build-doc` - Generates HTML documentation in `Build/Documentation` using [JSDoc 3](https://github.com/jsdoc3/jsdoc). More [details here](https://github.com/rahwang/cesium/tree/main/Documentation/Contributors/DocumentationGuide).
ggetz marked this conversation as resolved.
Show resolved Hide resolved
- `build-ts` - Generates a TypeScript definitions file for the Cesium library
- `build-third-party` - Generates `ThirdParty.json`, a file which lists the latest licensing information of installed third party modules
- `release` - A full release build that creates a shippable product, including generating documentation.
- `makeZipFile` - Builds a zip file containing all release files. This includes the source tree (suitable for use from an AMD-aware application), plus the combined and minified Cesium.js files, the generated documentation, the test suite, and the example applications (in both built and source form).
- `make-zip` - Builds a zip file containing all release files. This includes the source ESM modules, bundled ESM and IIFE format `Cesium.js`, plus the bundled minified versions of ESM and IIFE, the generated documentation, the test suite, and the example applications (in both built and source form).
- **Utility scripts** -- code coverage, static code analysis, and other utilities
- `coverage` - Runs coverage and opens the default browser with the results.
- `eslint` - Runs [ESLint](http://eslint.org/), a static code analysis tool, on the entire source tree.
- `eslint-watch` - A never-ending task that watches your file system for changes to Cesium and runs ESLint on any changed source files.
- `clean` - Removes all generated build artifacts.
- `clean` - Removes all generated build artifacts
- `cloc` - Runs [CLOC](https://github.com/AlDanial/cloc) to count the lines of code on the Source and Specs directories. This requires [Perl](http://www.perl.org/) to execute.
- `coverage` - Runs coverage and opens the default browser with the results
- `eslint` - Runs [ESLint](http://eslint.org/), a static code analysis tool, on the entire source tree
- `eslint-watch` - A never-ending task that watches your file system for changes to Cesium and runs ESLint on any changed source files
ggetz marked this conversation as resolved.
Show resolved Hide resolved
- `prettier` - Formats the code base using [Prettier](https://prettier.io/)
- `prettier-check` - Verifies prettier formatting, but does not write the output
- **Testing scripts** -- build and run the unit tests
- `test` - Runs all tests with [Karma](http://karma-runner.github.io/0.13/index.html) using the default browser specified in the Karma config file.
- `test-all` - Runs all tests with Karma using all browsers installed on the current system.
- `test-non-webgl` - Runs only non-WebGL tests.
- `test-webgl` - Runs only WebGL tests.
- `test-webgl-stub` - Runs all tests using the WebGL stub, which WebGL calls a noop and ignores related test expectations.
- `test-webgl-validation` - Runs all tests with Karma and enables low-level WebGL validation.
- `test-release` - Runs all tests on the minified release version of built Cesium.
- `test-all` - Runs all tests with Karma using all browsers installed on the current system
- `test-non-webgl` - Runs only non-WebGL tests
- `test-webgl` - Runs only WebGL tests
- `test-webgl-stub` - Runs all tests using the WebGL stub, which WebGL calls a noop and ignores related test expectations
- `test-webgl-validation` - Runs all tests with Karma and enables low-level WebGL validation
- `test-release` - Runs all tests on the minified release version of built Cesium
- **Deployment scripts**
- `deploy-s3` - Deploys the built CesiumJS files, the npm package, and the zip file to Amazon S3. This requires having credentials set up for the S3 bucket to which you are deploying.
- `deploy-status` - Set the deployment statuses in GitHub, for use with Travis.
- `deploy-set-version` - Sets the version of `package.json`, for use with Travis.
- `deploy-s3` - Deploys the built CesiumJS files, the npm package, and the zip file to Amazon S3. This requires having credentials set up for the S3 bucket to which you are deploying
- `deploy-status` - Sets the deployment statuses in GitHub, for use with Travis
- `deploy-set-version` - Sets the version of `package.json`, for use with Travis

## Travis and Continuous Integration

Cesium uses [Travis](https://travis-ci.com/) for continuous integration. The Travis configuration and all the steps of the build process are defined in `travis.yml`. The blog post [Cesium Continuous Integration](http://cesium.com/blog/2016/04/07/Cesium-Continuous-Integration/) contains an in-depth explaination of the travis build process.
Cesium uses [Travis](https://travis-ci.com/) for continuous integration. The Travis configuration and all the steps of the build process are defined in `travis.yml`. The blog post [Cesium Continuous Integration](http://cesium.com/blog/2016/04/07/Cesium-Continuous-Integration/) contains an in-depth explanation of the travis build process.
ggetz marked this conversation as resolved.
Show resolved Hide resolved

Travis triggers a build whenever someone opens a pull request or pushes code to the Cesium repository. After the build has completed, at the bottom on the pull request, the status of the build is shown and you can access the build by clicking the "Details" link.

Expand Down
10 changes: 5 additions & 5 deletions Documentation/Contributors/TestingGuide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,9 @@ These tests run quickly (for example, 15 seconds compared to 60) and are very re

#### Run All Tests Against the Minified Release Version of CesiumJS

Most test options load CesiumJS using the individual source files in the `Source` directory, which is great for debugging.
Most test options load CesiumJS with the unminified build plus a source map, which is great for debugging.

However, many users build apps using the built Cesium.js in `Build/Cesium` (which is created, for example, by running `npm run minifyRelease`). This option runs the tests using this instead of individual CesiumJS source files. The release version has `DeveloperError` exceptions optimized out so this test option makes `toThrowDeveloperError` always pass. See the [Build Guide](https://github.com/CesiumGS/cesium/blob/main/Documentation/Contributors/BuildGuide/README.md#build-scripts) for all the CesiumJS build options. When testing against built Cesium.js, the specs need to be built as well with `npm run build-specs`.
However, many users build apps using the built Cesium.js in `Build/Cesium` (which is created, for example, by running `npm run release`). This option runs the tests using this instead of the unminified build. The release version has `DeveloperError` exceptions optimized out so this test option makes `toThrowDeveloperError` always pass. See the [Build Guide](https://github.com/CesiumGS/cesium/blob/main/Documentation/Contributors/BuildGuide/README.md#build-scripts) for all the CesiumJS build options.

`npm run test-release`

Expand Down Expand Up @@ -187,11 +187,11 @@ These tests run quickly (for example, 15 seconds compared to 60) and are very re

#### Run All Tests against Combined File (Run All Tests against Combined File with Debug Code Removed)

Most test options load CesiumJS using the individual source files in the `Source` directory, which is great for debugging.
Most test options load CesiumJS with the unminified build plus a source map, which is great for debugging.

However, many users build apps using the built Cesium.js in `Build/Cesium` (which is created, for example, by running `npm run combine`). This option runs the tests using this instead of individual CesiumJS source files.
However, many users build apps using the built Cesium.js in `Build/Cesium` (which is created, for example, by running `npm run release`). This option runs the tests using this instead of the unminified build.

The **Run All Tests against Combined File with Debug Code Removed** is the same except it is for use with the release version of the built Cesium.js (which is created, for example, by running `npm run combineRelease`). The release version has `DeveloperError` exceptions optimized out so this test option makes `toThrowDeveloperError` always pass.
The **Run All Tests against Combined File with Debug Code Removed** is the same except it is for use with the release version of the built Cesium.js (which is created, for example, by running `npm run release`). The release version has `DeveloperError` exceptions optimized out so this test option makes `toThrowDeveloperError` always pass.

See the [Build Guide](https://github.com/CesiumGS/cesium/blob/main/Documentation/Contributors/BuildGuide/README.md#build-scripts) for all the CesiumJS build options.

Expand Down
Loading