Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration of a static filesystem for the node_modules #47998

Merged
merged 151 commits into from
May 5, 2020

Conversation

mistic
Copy link
Member

@mistic mistic commented Oct 11, 2019

Closes #43357
Closes #17176

That is the so awaited very first integration of a static filesystem for the node_modules on Kibana.
I really have the impression that would be huge, thank you very much for the support team!

== Benefits ==

  • decrease number of files we ship within the distributable (preventing us from create a windows service for Kibana)
  • avoid problems like the max path length of a file (Windows)
  • decrease the amount of time needed for decompress the distributable

== Benchmark ==

Collected when using a static-fs with Kibana for the node_modules.
The hardware used was:

Computer: Custom Windows Box
OS: Windows 10 64bits
Processor: 3,7 GHz AMD Ryzen 2700x
Memory: 16 GB 2933 MHz DDR4
Disk: SSD 500GB NVMe

Stats (on distributable) Without Static-Fs With Static-Fs Difference
Unzip (windows default tool) 47m 8m -83%
Number of files 137476 22803 -83%
Used memory 298MB 335MB +12%
Load time (cold start) ~3000ms 3253ms +8%

Along the normal usage of the application no performance issue was found.

mistic added 30 commits August 13, 2018 17:49
…ode_modules and monkey patch the bin start files to include the static file system loader on begining.
@mistic
Copy link
Member Author

mistic commented Apr 27, 2020

@elasticmachine merge upstream

@mistic
Copy link
Member Author

mistic commented Apr 29, 2020

@elasticmachine merge upstream

@mistic
Copy link
Member Author

mistic commented May 5, 2020

@elasticmachine merge upstream

@mistic mistic merged commit 33376fc into elastic:master May 5, 2020
@mistic
Copy link
Member Author

mistic commented May 5, 2020

7.x/7.8.0: 6c732a9

mistic added a commit to mistic/kibana that referenced this pull request May 5, 2020
* feat(NA): added new build step to generate a static file system for node_modules and monkey patch the bin start files to include the static file system loader on begining.

* chore(NA): updated lockfile.

* chore(NA): replace import to the correct static-fs files.

* fix(NA): resolve for a correct static fs loader.

* chore(NA): last static-fs working version.

* feat(NA): changed patched entryPoints.

* chore(NA): update to last static fs.

* fix(NA): fix imports on webpack shims. chore(NA): added new static-fs version.

* chore(NA): update to static-fs with stat and statSync.

* fix(NA): vendor chunk filtering in order to include default node modules and static node modules.

* refact(NA): major refact on create static modules fs task.

* fix(NA): remove global from clean empty folders task.

* chore(NA): change args order for deleteAll.

* chore(NA): move to the new static fs

* feat(NA): migrate to last apis of static fs

* fix(NA): patch correct entry file.

* chore(NA): last stable integration with static-fs task

* chore(NA): rollback some unecessary changes

* chore(NA): remove changes on sinon webpackshim

* chore(NA): integrate with official static-fs version

* chore(NA): integrate last thread-loader version

* chore(NA): added last static-fs version

* chore(NA): upgrade to last static-fs version

* chore(NA): upgrade to last static-fs version 1.1.1

* chore(NA): rever static-fs version to 1.1.0

* chore(NA): upgrade static-fs to 1.1.1

* chore(NA): upgrade static-fs for last 1.2.0 version

* chore(NA): update package.json

* chore(NA): run apm after patched environment

* chore(NA): specify rule to disable eslint for

* chore(NA): remove changes in the src/apm file

* chore(NA): change apm order in the dev scripts

* chore(na): update static-fs versin

* chore(NA): bump static fs version

* chore(NA): correctly lint task file

* chore(NA): move away from rimraf to del

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump to last static-fs version

* chore(NA): apply changed according PR review feedback

* chore(NA): remove changes to base optimizer

* chore(NA): fix discover pattern

* chore(NA): bump static-fs version

* chore(NA): bump static fs to 1.6.3

* chore(NA): bump static-fs to 1.6.4

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs for 1.8.3

* chore(NA): bump static-fs to 1.9.0

* chore(NA): bump static-fs to 1.9.1

* chore(NA): update to last static-fs version 1.10.0

* chore(NA): moving to @elastic/static-fs package

* fix(NA): change import to the new package

* chore(NA): bump elastic static-fs to 1.0.1

Co-authored-by: spalger <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
@mistic mistic removed the review label May 5, 2020
mistic added a commit that referenced this pull request May 5, 2020
)

* feat(NA): added new build step to generate a static file system for node_modules and monkey patch the bin start files to include the static file system loader on begining.

* chore(NA): updated lockfile.

* chore(NA): replace import to the correct static-fs files.

* fix(NA): resolve for a correct static fs loader.

* chore(NA): last static-fs working version.

* feat(NA): changed patched entryPoints.

* chore(NA): update to last static fs.

* fix(NA): fix imports on webpack shims. chore(NA): added new static-fs version.

* chore(NA): update to static-fs with stat and statSync.

* fix(NA): vendor chunk filtering in order to include default node modules and static node modules.

* refact(NA): major refact on create static modules fs task.

* fix(NA): remove global from clean empty folders task.

* chore(NA): change args order for deleteAll.

* chore(NA): move to the new static fs

* feat(NA): migrate to last apis of static fs

* fix(NA): patch correct entry file.

* chore(NA): last stable integration with static-fs task

* chore(NA): rollback some unecessary changes

* chore(NA): remove changes on sinon webpackshim

* chore(NA): integrate with official static-fs version

* chore(NA): integrate last thread-loader version

* chore(NA): added last static-fs version

* chore(NA): upgrade to last static-fs version

* chore(NA): upgrade to last static-fs version 1.1.1

* chore(NA): rever static-fs version to 1.1.0

* chore(NA): upgrade static-fs to 1.1.1

* chore(NA): upgrade static-fs for last 1.2.0 version

* chore(NA): update package.json

* chore(NA): run apm after patched environment

* chore(NA): specify rule to disable eslint for

* chore(NA): remove changes in the src/apm file

* chore(NA): change apm order in the dev scripts

* chore(na): update static-fs versin

* chore(NA): bump static fs version

* chore(NA): correctly lint task file

* chore(NA): move away from rimraf to del

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump static-fs version

* chore(NA): bump to last static-fs version

* chore(NA): apply changed according PR review feedback

* chore(NA): remove changes to base optimizer

* chore(NA): fix discover pattern

* chore(NA): bump static-fs version

* chore(NA): bump static fs to 1.6.3

* chore(NA): bump static-fs to 1.6.4

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs to last version

* chore(NA): bump static-fs for 1.8.3

* chore(NA): bump static-fs to 1.9.0

* chore(NA): bump static-fs to 1.9.1

* chore(NA): update to last static-fs version 1.10.0

* chore(NA): moving to @elastic/static-fs package

* fix(NA): change import to the new package

* chore(NA): bump elastic static-fs to 1.0.1

Co-authored-by: spalger <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>

Co-authored-by: spalger <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kobelb added a commit to kobelb/kibana that referenced this pull request May 20, 2020
kobelb added a commit that referenced this pull request May 20, 2020
kobelb added a commit to kobelb/kibana that referenced this pull request May 20, 2020
jloleysens added a commit to jloleysens/kibana that referenced this pull request May 20, 2020
…ent/add-support-in-url-for-hidden-toggle

* 'master' of github.com:elastic/kibana: (49 commits)
  [Uptime] Improve responsiveness details page (elastic#67034)
  skip flaky suite (elastic#66669)
  Revert "Integration of a static filesystem for the node_modules (elastic#47998)" (elastic#67124)
  Support api_integration/kibana/stats against remote hosts (elastic#53000)
  chore(NA): add module name mapper for src plugins on x-pack (elastic#67103)
  Change the error message on TSVB in order to be more user friendly (elastic#67090)
  [kbn/optimizer] poll parent process to avoid zombie processes (elastic#67059)
  [Visualize] Lazy load default editor, fix duplicated styles (elastic#66732)
  Bump styled-component dependencies (elastic#66611)
  Bump react-markdown dependencies (elastic#66615)
  Fix Core docs links (elastic#66977)
  Timelion graph is not refreshing content after searching or filtering (elastic#67023)
  Remove `--xpack.endpoint.enabled=true` from README.md file (elastic#67053)
  Move apm tutorial from apm plugin into apm_oss plugin (elastic#66432)
  [Logs UI] Restore call to `UsageCollector.countLogs` (elastic#67051)
  Remove unused license check result from LP Security plugin (elastic#66966)
  [Saved Objects] adds support for including hidden types in saved objects client (elastic#66879)
  [Discover] Deangularize timechart header (elastic#66532)
  [Discover] Improve and unskip a11y context view test (elastic#66959)
  [SIEM] Refactor Timeline.timelineType draft to Timeline.status draft (elastic#66864)
  ...

# Conflicts:
#	x-pack/plugins/index_management/__jest__/client_integration/helpers/home.helpers.ts
kobelb added a commit that referenced this pull request May 20, 2020
kobelb added a commit that referenced this pull request May 20, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request May 21, 2020
* master:
  [apm] Annotation API documentation (elastic#65963)
  [Uptime] Improve responsiveness details page (elastic#67034)
  skip flaky suite (elastic#66669)
  Revert "Integration of a static filesystem for the node_modules (elastic#47998)" (elastic#67124)
  Support api_integration/kibana/stats against remote hosts (elastic#53000)
  chore(NA): add module name mapper for src plugins on x-pack (elastic#67103)
  Change the error message on TSVB in order to be more user friendly (elastic#67090)
  [kbn/optimizer] poll parent process to avoid zombie processes (elastic#67059)
  [Visualize] Lazy load default editor, fix duplicated styles (elastic#66732)
  Bump styled-component dependencies (elastic#66611)
  Bump react-markdown dependencies (elastic#66615)
  Fix Core docs links (elastic#66977)
  Timelion graph is not refreshing content after searching or filtering (elastic#67023)
  Remove `--xpack.endpoint.enabled=true` from README.md file (elastic#67053)
  Move apm tutorial from apm plugin into apm_oss plugin (elastic#66432)
  [Logs UI] Restore call to `UsageCollector.countLogs` (elastic#67051)
  Remove unused license check result from LP Security plugin (elastic#66966)
gmmorris added a commit to gmmorris/kibana that referenced this pull request May 21, 2020
* master: (21 commits)
  [Alerting] Hides the `alert` SavedObjects type (elastic#66719)
  skip flaky suite (elastic#66869)
  fix visual baseline tests
  [kbn/optimizer] require fsevents on macos (elastic#67147)
  [APM] Fix obscured service map connections (elastic#67129)
  [apm] Annotation API documentation (elastic#65963)
  [Uptime] Improve responsiveness details page (elastic#67034)
  skip flaky suite (elastic#66669)
  Revert "Integration of a static filesystem for the node_modules (elastic#47998)" (elastic#67124)
  Support api_integration/kibana/stats against remote hosts (elastic#53000)
  chore(NA): add module name mapper for src plugins on x-pack (elastic#67103)
  Change the error message on TSVB in order to be more user friendly (elastic#67090)
  [kbn/optimizer] poll parent process to avoid zombie processes (elastic#67059)
  [Visualize] Lazy load default editor, fix duplicated styles (elastic#66732)
  Bump styled-component dependencies (elastic#66611)
  Bump react-markdown dependencies (elastic#66615)
  Fix Core docs links (elastic#66977)
  Timelion graph is not refreshing content after searching or filtering (elastic#67023)
  Remove `--xpack.endpoint.enabled=true` from README.md file (elastic#67053)
  Move apm tutorial from apm plugin into apm_oss plugin (elastic#66432)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Integrate a static filesystem for node modules with kibana Investigate slow package installs
8 participants