build immutable bundles for new platform plugins #53976
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spalger
force-pushed
the
implement/kbn-optimizer
branch
13 times, most recently
from
374f3b1
to
0dc7d68
Compare
mshustov
reviewed
Jan 10, 2020
mshustov
reviewed
Jan 10, 2020
spalger
force-pushed
the
implement/kbn-optimizer
branch
15 times, most recently
from
dd74deb
to
7c5be03
Compare
joshdover
approved these changes
Feb 12, 2020
There was a problem hiding this comment.
LGTM - thanks for refactoring this for readability.
I haven't reviewed all 12,000 lines 😬 but I don't have any fundamental issues with how it works and I'm able to follow along. This is going to lay a great foundation for even further improvements going forward!
tylersmalley
approved these changes
Feb 13, 2020
There was a problem hiding this comment.
LGTM - looking forward to this shaving more off the production builds as plugins move over to the new platform. As of right now, this is reducing our HTTP assets on load from 134 to 128 (still crazy high).
Development looks good as well.
One thing which needs a follow-up is that the building of 3rd party plugins was removed in #55281 (comment) so could not be tested as part of this PR.
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Feb 13, 2020
* master: add `absolute` option to `getUrlForApp` (elastic#57193) [Telemetry] Migrate public to NP (elastic#56285) address flaky test where instances might have different start… (elastic#57506) fix(NA): support legacy plugins path in plugins (elastic#57472) build immutable bundles for new platform plugins (elastic#53976) [SIEM] [Detection Engine] Reject if duplicate rule_id in request payload (elastic#57057) Add autocomplete="off" for input type="password" to appease the scanners (elastic#56922) Use default spaces suffix for signals index if spaces disabled (elastic#57244) [Alerting] Create alert design cleanup (elastic#56929)
mbondyra
added a commit
to mbondyra/kibana
that referenced
this pull request
Feb 13, 2020
* master: (22 commits) Use log4j pattern syntax (elastic#57433) [ML] Categorization field example endpoint tests (elastic#57471) [Lens] Filter out pinned filters from saved object of Lens (elastic#57197) Lens client side shim cleanup (elastic#56976) [Maps] do not show border color for icon in legend when border width is zero (elastic#57501) refactors 'data-providers' tests (elastic#57474) add `absolute` option to `getUrlForApp` (elastic#57193) [Telemetry] Migrate public to NP (elastic#56285) address flaky test where instances might have different start… (elastic#57506) fix(NA): support legacy plugins path in plugins (elastic#57472) build immutable bundles for new platform plugins (elastic#53976) [SIEM] [Detection Engine] Reject if duplicate rule_id in request payload (elastic#57057) Add autocomplete="off" for input type="password" to appease the scanners (elastic#56922) Use default spaces suffix for signals index if spaces disabled (elastic#57244) [Alerting] Create alert design cleanup (elastic#56929) Management Api - add to migration guide (elastic#56892) fixing maps (elastic#56706) [Maps] Autocomplete for custom color palettes and custom icon palettes (elastic#56446) [Alerting] make actionGroup name's i18n-able (elastic#57404) fixed flaky test (elastic#57490) ... # Conflicts: # src/legacy/core_plugins/telemetry/public/components/__snapshots__/telemetry_form.test.js.snap # src/plugins/telemetry/public/components/telemetry_management_section.tsx
spalger
pushed a commit
that referenced
this pull request
Feb 14, 2020
* build immutable bundles for new platform plugins (#53976) * build immutable bundles for new platform plugins * only inspect workers if configured to do so * [navigation] use an index.scss file * add yarn.lock symlink * set pluginScanDirs in test so fixtures stay consistent * cleanup helpers a little * fix type error * support KBN_OPTIMIZER_MAX_WORKERS for limiting workers via env * test support for KBN_OPTIMIZER_MAX_WORKERS * expand the available memory for workers when only running one or two * add docs about KBN_OPTIMIZER_MAX_WORKERS environment variable * fix README link * update kbn/pm dist * implement bundle caching/reuse * update kbn/pm dist * don't check for cache if --no-cache is passed * update renovate config * standardize on index.scss, move console styles over * add support for --no-cache to cli * include worker config vars in optimizer version * ignore concatenated modules * update integration test * add safari to browserslist to avoid user-agent warnings in dev * update docs, clean up optimizer message/misc naming * always handle initialized messages, don't ignore states that are attached to specific events * reword caching docs, add environment var to disable caching * tweak logging and don't use optimizer.useBundleCache as that's disabled in dev * handle change notifications * batch changes for 1 second * rename CompilerState type to CompilerMsg * getChanges() no longer needs to assign changes to dirs * remove unused deps * split up run_worker.ts and share cacheKey generation logic * add a couple docs * update tests and remove unused imports * specify files when creating bundle cache key * remove one more unused import * match existing dev cli output more closely * update kbn/pm dist * set KBN_NP_PLUGINS_BUILT to avoid warning in CI * avoid extending global window type * add note to keep pluginScanDirs in sync * pass browserslistEnv in workerConfig so it is used for cache key * load commons.bundle.js in parallel too * emit initialized+success states if all bundles are cached * load bootstraps as quickly as possible * skip flaky suite * bump * update jest snapshots * remove hashing from cache key generation * remove unnecessary non-null assertion * improve docs and break up Optimizer#run() * remove unused import * refactor kbn/optimizer to break up observable logic, implement more helpful cache invalidation logic with logging * fix tests * add initializing phase * avoid rxjs observable constructor * remove unnecessary rxjs helper, add tests for bundle cache * update consumers of optimizer * update readme with new call style * replace "new platform" with "kibana platform" * fix a couple more renames * add support for several plain-text file formats * fix naming of OptimizerMsg => OptimizerUpdate, use "store" naming too * one more OptimizerMsg update * ensure bundles are not cached when cache config is false * test for initializing states and bundle cache events * remove unnecessary timeout change * Remove unnecessary helpers * Add tests for BundleCache class * Add tests for Bundle class * test summarizeEvent$ * missing paths are no longer listed in mtimes map * add tests for optimizer/cache_keys * Add some extra docs * Remove labeled loop * add integration test for kbn-optimizer watcher components * querystring-browser removed * tweak logging a smidge, improve info and final message * remove unused imports * remove duplication of getModuleCount() method * move type annotation that validates things * clear up the build completion message Co-authored-by: Elastic Machine <[email protected]> * [kbn/optimizer] Fix windows support (#57592) * [kbn/optimizer] simplify run_workers.ts a smidge * use Path.resolve() to create windows paths from normalized ones Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 343bc9c) * remove istanbul/code coverage references * fix webpack config syntax * removal of querystring-browser was backported to 7.x Co-authored-by: Elastic Machine <[email protected]>
|
7.x/7.7: 9f94c39 |
This was referenced Feb 14, 2020
Merged
lizozom
pushed a commit
to lizozom/kibana
that referenced
this pull request
Feb 18, 2020
* Generate NP plugin * Added tsconfig * tsconfig * Adjust sao test * Add server side to plugin gen * Added navigation * add empty element * eslint * platform team CR * design CR improvements * text updates * temp disable plugin gen tests * eslint * Code review fixes * Add scss support - requires elastic#53976 to be merged to work * CR fixes * comment fixes * Don't generate eslint for internal plugins by default * Update tests * reenable jest test for sao * Fix regex * review comments * code review Co-authored-by: Elastic Machine <[email protected]>
lizozom
pushed a commit
that referenced
this pull request
Feb 23, 2020
* Update plugin generator to generate NP plugins (#55281) * Generate NP plugin * Added tsconfig * tsconfig * Adjust sao test * Add server side to plugin gen * Added navigation * add empty element * eslint * platform team CR * design CR improvements * text updates * temp disable plugin gen tests * eslint * Code review fixes * Add scss support - requires #53976 to be merged to work * CR fixes * comment fixes * Don't generate eslint for internal plugins by default * Update tests * reenable jest test for sao * Fix regex * review comments * code review Co-authored-by: Elastic Machine <[email protected]> * Update plugin creation string Co-authored-by: Elastic Machine <[email protected]>
💔 Build FailedTest FailuresKibana Pipeline / kibana-intake-agent / Jest Integration Tests.packages/kbn-plugin-generator/integration_tests.running the plugin-generator via 'node scripts/generate_plugin.js plugin-name' with default config then running with es instance 'yarn start' should result in the spec plugin being initialized on kibana's stdoutStandard OutStack TraceHistory
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Creates a decoupled optimizer that can build new platform plugins so that they can be distributed as static assets, created at build time, loaded independently in the browser, and never rebuilt in the distributable. See the
@kbn/optimizerreadme for some more details on the new optimizer.styles
.scssfiles can now be imported directly from JS/TS module. node-sass is used to build the linked file as if it was built using node-sass directly (using node-sass module resolution, not webpack's). An import for thesrc/legacy/ui/public/styles/_styling_constants.scssfile is automatically injected at the beginning of each.scssfile imported by a JS/TS module so that users can access the styling constants without explicitly pulling in the styles (as requested by design). The styles required by a module are a part of the JS bundle (both light and dark mode variants) and are injected into the page when the module is loaded based on the theme selected for the current page. Light/dark mode variants are built the same way they used to be, by replacingeui_colors_lightwitheui_colors_darkwhen it's found in any imported path. A page reload is still required to switch themes.changes made to plugins
uiExports.styleSheetPaths(ie src/plugins/expressions/public/index.ts)performance
A primary performance goal of the new optimizer is to build everything as quickly as possible, it does this by running webpack compilers in parallel. This can cause the CPU usage to spike for a bit while the initial build completes. Most people I've talked to prefer that to longer build times, but if you prefer not pegging your CPU you can lower the maximum number of workers using the
KBN_OPTIMIZER_MAX_WORKERSenvironment variable. Setting it to 1 will make the new build take a while, but will cause the least amount of load on your system.caching
When the files in a bundle haven't changed since the last time it was built, the webpack compiler for that bundle will not be started and instead a watcher will be created to watch the files for changes. Checkout the caching section of the readme
Dev Docs
Plugins that have been migrated to the "new platform" are built with a new system into their own
target/publicdirectory. To build third-party plugin's with this new system either pass--plugin-pathtonode scripts/build_new_platform_pluginsor use the@kbn/optimizerpackage in the Kibana repo to build your plugin as described in the readme for that package.