new plugins pack strategy proposal

[colinsurprenant]

this idea applies to both offline plugins installation and eventual pre-packaged plugins packs themes.

This is an extract of the conversation in #4779 which derails the original PR intent.

The problem we are facing with our offline installation process is that by disrupting Bundler's state by either cleaning *.gem files or simply by installing or updating plugins on one side and trying to pack the resulting installation and move it to another target installation to provide the ability to install these new plugins in an offline way, we create an impedance mismatch between 2 different bundler states. The Gemfile and Gemfile.jruby-1.9.lock does not match the current state as just unpacked on the target installation. This result in problems as described in #4651 and #4779.

One of the original problem is the cleaning of the Bundler cache at artifact generation:

1. For reference, this is where the *.gem file are discarded from the artifact creation.
2. I think that what we do in (1) made sense from a package size perspective before we decided to support offline plugin installation but now it does not make sense because it creates a state problem in Bundler.
3. I will argue that the real problem here is our default plugins pre-installation & packaging strategy. the idea of providing a readily available default plugins set makes sense from a getting-started perspective but I think we should revisit how we could achieve this without disrupting the installed plugins state.
4. One idea would be that the bin/plugin pack and bin/plugin unpack commands freezes and restores the complete plugins state, including Gemfile and Gemfile.jruby-1.9.lock and this would allows just running logstash on the target system exactly as it was on the source system without having to do any bin/plugin install or bin/plugin update command after the bin/plugin unpack. We would just document that bin/plugin install or bin/plugin update require connectivity. I don't thinks it actually make sense to support bin/plugin install or bin/plugin update offline!
5. Similar to (4), instead of pre installing plugins and deleting the *.gem files at packaging time, we could ship the core logstash package without any plugins or just a very limited set of core plugins including their .gem files and provide a few additional plugins packages that can be downloaded and installed the same way as (4). Obviously one of these additional package would be the infamous "default" plugins package that would be pretty big since it would also include all the .gem files but since its an optional and non essential download, it would be acceptable and we would also provide other smaller plugins combinations packages like a "getting started" that would cover the courses and workshop, etc.
6. by doing this we would:
   • stop confusing bundler state and avoid more dangerous and non future-proof monkey patching
   • move forward with the plugin-based vision of not having to pre-package all plugins with the base logstash package
   • still provide the facility for someone to download and easily install the whole default plugins package which would be a bigger file but with a very simple installation since it would just involve copying files at the right place and not call any bundler install/update.

---

With such a pack, there would be no installing or updating individual plugins, to update or install a new plugin, the pack plugins state could be (recreated and) moved back to another connected machine, unpacked, perform new plugin install or update, repacked and moved back to the disconnected machine.

Let me sequence the new workflow under that idea:

• On connected machine with logstash
  • install, update, remove plugins to satisfaction
  • run bin/plugin pack
  • copy pack file to disconnected machine
• On disconnected machine with logstash
  • stop logstash
  • run bin/plugin unpack
  • start logstash
  • now disconnected logstash run exactly with same set of plugins as connected logstash

Now if you need to update a single plugins. there are 2 possibilities:

1. You still have the disconnected logstash pack file around:
   • move disconnected pack to a connected logstash
   • run bin/plugin unpack
   • install, update, remove plugins to satisfaction
   • run bin/plugin pack
   • copy pack file to disconnected machine
2. You do not have the disconnected logstash pack file around:
   • run bin/plugin pack on disconnected logstash
   • move disconnected pack to a connected logstash
   • run bin/plugin unpack
   • install, update, remove plugins to satisfaction
   • run bin/plugin pack
   • copy pack file to disconnected machine

[colinsurprenant]

After discussing this with @purbon, @ph and @suyograo we think this strategy for basic offline plugins installation could potentially be a good alternative and solve what I call the impedance problems, or dependencies conflicts, we potentially introduce when manipulating the bundler files directly.

One use-case where this will not work is with x-packs where the requirement is to provide plugin packs that support offline installation but in an additive way. This mean that it should be possible to install a logstash x-pack on-top of a current logstash installation that has installed plugins and also support this in offline mode. The above strategy completely swaps the bundler state so if we apply this strategy to x-packs, the current logstash installed plugins would be lost.

It seems that the only way we think this will be possible is to still support injecting bundler's cache to install new plugins on top of a current installation.To minimize potential dependency conflicts, which is the initial problem we are trying to solve above by avoiding directly playing with Bundler's state, we think that enforcing stricter dependencies versions constrains for all core gems and all plugins within a released version and creating a more relaxed conflict resolution class in Bundler (as supported in recent/upcoming versions) we would probably avoid dependency conflicts.

[ph]

One use-case where this will not work is with x-packs where the requirement is to provide plugin packs that support offline installation but in an additive way. This mean that it should be possible to install a logstash x-pack on-top of a current logstash installation that has installed plugins and also support this in offline mode. The above strategy completely swaps the bundler state so if we apply this strategy to x-packs, the current logstash installed plugins would be lost.

The PR #6404 and the issue #6393 align with the your proposed additive strategy and also don't impact the ruby cache.

The code in #6404 does the following (high level view):

Logstash with internet

1. Take the specified plugins we want to pack
2. Fetch all their dependencies, this operation is recursive.
3. If the gem is in the cache folder we use that, if not we download it from rubygems.
4. Create a new zip archive with 2 scopes
   • direct plugins to install
   • dependencies of theses plugins

Offline Logstash

1. Zip is uncompressed
2. Direct plugins to install are added to the gemfile with a specific version.
3. If a dependencies is present in the gemfile the version is updated.
4. gems are added to a rubygems local index.
5. Bundler do a local resolution.
6. The gem code and the specification are directly installed in the local directory.
7. Gemfile/lock are updated on disk.

Generating an X-packs and generating an offline pack uses the same creation code in https://github.com/elastic/logstash/tree/master/tools/paquet

Both artifact uses the same file structure and so it goes with the same install path.

more relaxed conflict resolution class in Bundler (as supported in recent/upcoming versions) we would probably avoid dependency conflicts.

This is still an open question that might come in the real world usage.

I'll will close this issue thanks @colinsurprenant for the write up and the discussion that lead to #6404.