Add bin/logstash-plugin prepare-offline-pack command

This new command replace the old workflow of `pack`, `unpack` and the `install --local`, and wrap all the logic into one uniform way of installing plugins.
The work is based on the flow developed for installing an x-pack inside Logstash, when you call prepare-offline-pack, the specified plugins and their dependencies will be packaged in a zip.
And this zip can be installed with the same flow as the pack.

Definition:

Source Logstash: Where you run the prepare-offline-pack.
Target Logstash: Where you install the offline package.

PROS:
- If you install a .gem in the source logstash, the .gem and his dependencies will be bundled.
- The install flow doesn't need to have access to the internet.
- Nothing special need to be setup in the target logstash environment.

CONS:
- The is one minor drawback, the plugins need to have their JARS bundled with them for this flow to work, this is currently the case for all the official plugins.
- The source Logstash need to have access to the internet when you install plugins before packaging them.

Usage examples:
bin/logstash-plugin prepare-offline-pack logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-jdbc logstash-input-beats
bin/logstash-plugin prepare-offline-pack logstash-filter-*
bin/logstash-plugin prepare-offline-pack logstash-filter-* logstash-input-beats

How to install:
bin/logstash-plugin install file:///tmp/logstash-offline-plugins-XXXX.zip

Fixes #6404

Gemfile

@@ -6,6 +6,7 @@ gem "logstash-core", :path => "./logstash-core"
 gem "logstash-core-queue-jruby", :path => "./logstash-core-queue-jruby"
 gem "logstash-core-event-java", :path => "./logstash-core-event-java"
 gem "logstash-core-plugin-api", :path => "./logstash-core-plugin-api"
+gem "paquet", "~> 0.2.0"
 gem "ruby-progressbar", "~> 1.8.1"
 gem "builder", "~> 3.2.2"
 gem "file-dependencies", "0.1.6"

ci/ci_integration.sh

@@ -16,6 +16,7 @@ rake artifact:tar
 cd build
 echo "Extracting logstash tar file in build/"
 tar xf *.tar.gz
+
 cd ../qa/integration
 # to install test dependencies
 bundle install

ci/travis_integration_install.sh

@@ -18,8 +18,9 @@ rake artifact:tar
 cd build
 echo "Extracting logstash tar file in build/"
 tar xf *.tar.gz
+
 cd ../qa/integration
 pwd
 echo $BUNDLE_GEMFILE
 # to install test dependencies
-bundle install --gemfile="Gemfile"
+bundle install --gemfile="Gemfile"

lib/pluginmanager/bundler/logstash_injector.rb

@@ -5,6 +5,8 @@
 require "bundler/dependency"
 require "bundler/dsl"
 require "bundler/injector"
+require "pluginmanager/gemfile"
+
 
 # This class cannot be in the logstash namespace, because of the way the DSL
 # class interact with the other libraries
@@ -14,17 +16,24 @@ def self.inject!(new_deps, options = { :gemfile => LogStash::Environment::GEMFIL
       gemfile = options.delete(:gemfile)
       lockfile = options.delete(:lockfile)
 
-      bundler_format = Array(new_deps).collect { |plugin|  ::Bundler::Dependency.new(plugin.name, "=#{plugin.version}")}
+      bundler_format = new_deps.plugins.collect(&method(:dependency))
+      dependencies = new_deps.dependencies.collect(&method(:dependency))
 
       injector = new(bundler_format)
-      injector.inject(gemfile, lockfile)
+      injector.inject(gemfile, lockfile, dependencies)
     end
 
+    def self.dependency(plugin)
+      ::Bundler::Dependency.new(plugin.name, "=#{plugin.version}")
+    end
 
     # This class is pretty similar to what bundler's injector class is doing
     # but we only accept a local resolution of the dependencies instead of calling rubygems.
     # so we removed `definition.resolve_remotely!`
-    def inject(gemfile_path, lockfile_path)
+    #
+    # And managing the gemfile is down by using our own Gemfile parser, this allow us to
+    # make it work with gems that are already defined in the gemfile.
+    def inject(gemfile_path, lockfile_path, dependencies)
       if Bundler.settings[:frozen]
         # ensure the lock and Gemfile are synced
         Bundler.definition.ensure_equivalent_gemfile_and_lockfile(true)
@@ -33,16 +42,31 @@ def inject(gemfile_path, lockfile_path)
       end
 
       builder = Dsl.new
-      builder.eval_gemfile(gemfile_path)
+      gemfile = LogStash::Gemfile.new(File.new(gemfile_path, "r+")).load
 
-      @new_deps -= builder.dependencies
+      begin
+        @new_deps.each do |dependency|
+          gemfile.update(dependency.name, dependency.requirement)
+        end
 
-      builder.eval_gemfile("injected gems", new_gem_lines) if @new_deps.any?
-      definition = builder.to_definition(lockfile_path, {})
-      append_to(gemfile_path) if @new_deps.any?
-      definition.lock(lockfile_path)
+        # If the dependency is defined in the gemfile, lets try to update the version with the one we have
+        # with the pack.
+        dependencies.each do |dependency|
+          if gemfile.defined_in_gemfile?(dependency.name)
+            gemfile.update(dependency.name, dependency.requirement)
+          end
+        end
 
-      return @new_deps
+        builder.eval_gemfile("bundler file", gemfile.generate())
+        definition = builder.to_definition(lockfile_path, {})
+        definition.lock(lockfile_path)
+        gemfile.save
+      rescue => e
+        # the error should be handled elsewhere but we need to get the original file if we dont
+        # do this logstash will be in an inconsistent state
+        gemfile.restore!
+        raise e
+      end
     ensure
       Bundler.settings[:frozen] = "1" if frozen
     end

lib/pluginmanager/command.rb

@@ -6,15 +6,15 @@ def gemfile
 
   # If set in debug mode we will raise an exception and display the stacktrace
   def report_exception(readable_message, exception)
-    if ENV["DEBUG"]
+    if debug?
       raise exception
     else
       signal_error("#{readable_message}, message: #{exception.message}")
     end
   end
 
   def display_bundler_output(output)
-    if ENV['DEBUG'] && output
+    if debug? && output
       # Display what bundler did in the last run
       $stderr.puts("Bundler output")
       $stderr.puts(output)
@@ -35,4 +35,8 @@ def relative_path(path)
     require "pathname"
     ::Pathname.new(path).relative_path_from(::Pathname.new(LogStash::Environment::LOGSTASH_HOME)).to_s
   end
+
+  def debug?
+    ENV["DEBUG"]
+  end
 end

lib/pluginmanager/custom_gem_indexer.rb

@@ -0,0 +1,63 @@
+# encoding: utf-8
+require "pluginmanager/ui"
+require "stud/temporary"
+
+module LogStash module PluginManager
+  class CustomGemIndexer
+    GEMS_DIR = "gems"
+
+    class << self
+      # Copy the file to a specific format that `Gem::Indexer` can understand
+      # See `#update_in_memory_index`
+      def copy_to_local_source(temporary_directory)
+        local_source = Stud::Temporary.pathname
+        local_source_gems = ::File.join(local_source, GEMS_DIR)
+
+        FileUtils.mkdir_p(local_source_gems)
+        PluginManager.ui.debug("Creating the index structure format from #{temporary_directory} to #{local_source}")
+
+        Dir.glob(::File.join(temporary_directory, "**", "*.gem")).each do |file|
+          destination = ::File.join(local_source_gems, ::File.basename(file))
+          FileUtils.cp(file, destination)
+        end
+
+        local_source
+      end
+
+      # *WARNING*: Bundler need to not be activated at this point because it wont find anything that
+      # is not defined in the gemfile/lock combo
+      #
+      # This takes a folder with a special structure, will generate an index
+      # similar to what rubygems do and make them available in the local program,
+      # we use this **side effect** to validate theses gems with the current gemfile/lock.
+      # Bundler will assume they are system gems and will use them when doing resolution checks.
+      #
+      #.
+      # ├── gems
+      # │   ├── addressable-2.4.0.gem
+      # │   ├── cabin-0.9.0.gem
+      # │   ├── ffi-1.9.14-java.gem
+      # │   ├── gemoji-1.5.0.gem
+      # │   ├── launchy-2.4.3-java.gem
+      # │   ├── logstash-output-elasticsearch-5.2.0-java.gem
+      # │   ├── logstash-output-secret-0.1.0.gem
+      # │   ├── manticore-0.6.0-java.gem
+      # │   ├── spoon-0.0.6.gem
+      # │   └── stud-0.0.22.gem
+      #
+      # Right now this work fine, but I think we could also use Bundler's SourceList classes to handle the same thing
+      def update_in_memory_index!(local_source)
+        PluginManager.ui.debug("Generating indexes in #{local_source}")
+        indexer = ::Gem::Indexer.new(local_source, { :build_modern => true})
+        indexer.ui = ::Gem::SilentUI.new unless ENV["DEBUG"]
+        indexer.generate_index
+      end
+
+      def index(path)
+        local_source = copy_to_local_source(path)
+        update_in_memory_index!(local_source)
+        local_source
+      end
+    end
+  end
+end end

lib/pluginmanager/errors.rb

@@ -1,6 +1,8 @@
 # encoding: utf-8
 module LogStash module PluginManager
     class PluginManagerError < StandardError; end
+    class PluginNotFoundError < PluginManagerError; end
+    class UnpackablePluginError < PluginManagerError; end
     class FileNotFoundError < PluginManagerError; end
     class InvalidPackError < PluginManagerError; end
     class InstallError < PluginManagerError

lib/pluginmanager/gem_installer.rb

@@ -2,6 +2,7 @@
 require "pluginmanager/ui"
 require "pathname"
 require "rubygems/package"
+require "fileutils"
 
 module LogStash module PluginManager
   # Install a physical gem package to the appropriate location inside logstash
@@ -12,11 +13,13 @@ class GemInstaller
     GEM_HOME = Pathname.new(::File.join(LogStash::Environment::BUNDLE_DIR, "jruby", "1.9"))
     SPECIFICATIONS_DIR = "specifications"
     GEMS_DIR = "gems"
+    CACHE_DIR = "cache"
 
     attr_reader :gem_home
 
     def initialize(gem_file, display_post_install_message = false, gem_home = GEM_HOME)
-      @gem = ::Gem::Package.new(gem_file)
+      @gem_file = gem_file
+      @gem = ::Gem::Package.new(@gem_file)
       @gem_home = Pathname.new(gem_home)
       @display_post_install_message = display_post_install_message
     end
@@ -26,6 +29,7 @@ def install
       extract_files
       write_specification
       display_post_install_message
+      copy_gem_file_to_cache
     end
 
     def self.install(gem_file, display_post_install_message = false, gem_home = GEM_HOME)
@@ -41,6 +45,10 @@ def spec_dir
       gem_home.join(SPECIFICATIONS_DIR)
     end
 
+    def cache_dir
+      gem_home.join(CACHE_DIR)
+    end
+
     def spec_file
       spec_dir.join("#{spec.full_name}.gemspec")
     end
@@ -69,10 +77,16 @@ def display_post_install_message?
       @display_post_install_message && !spec.post_install_message.nil?
     end
 
+    def copy_gem_file_to_cache
+      destination = ::File.join(cache_dir, ::File.basename(@gem_file))
+      FileUtils.cp(@gem_file, destination)
+    end
+
     def create_destination_folders
       FileUtils.mkdir_p(gem_home)
       FileUtils.mkdir_p(gem_dir)
       FileUtils.mkdir_p(spec_dir)
+      FileUtils.mkdir_p(cache_dir)
     end
   end
 end end

lib/pluginmanager/gemfile.rb

@@ -71,6 +71,10 @@ def restore
       @gemset = @original_backup
     end
 
+    def defined_in_gemfile?(name)
+      @gemset.find_gem(name)
+    end
+
     def restore!
       restore
       save

lib/pluginmanager/install.rb

@@ -158,13 +158,17 @@ def install_gems_list!(install_list)
   # Bundler 2.0, will have support for plugins source we could create a .gem source
   # to support it.
   def extract_local_gems_plugins
+    FileUtils.mkdir_p(LogStash::Environment::CACHE_PATH)
     plugins_arg.collect do |plugin|
       # We do the verify before extracting the gem so we dont have to deal with unused path
       if verify?
         puts("Validating #{plugin}")
         signal_error("Installation aborted, verification failed for #{plugin}") unless LogStash::PluginManager.logstash_plugin?(plugin, version)
       end
 
+      # Make the original .gem available for the prepare-offline-pack,
+      # paquet will lookup in the cache directory before going to rubygems.
+      FileUtils.cp(plugin, ::File.join(LogStash::Environment::CACHE_PATH, ::File.basename(plugin)))
       package, path = LogStash::Rubygems.unpack(plugin, LogStash::Environment::LOCAL_GEM_PATH)
       [package.spec.name, package.spec.version, { :path => relative_path(path) }]
     end

lib/pluginmanager/main.rb

@@ -21,6 +21,7 @@ module PluginManager
 require "pluginmanager/pack"
 require "pluginmanager/unpack"
 require "pluginmanager/generate"
+require "pluginmanager/prepare_offline_pack"
 
 module LogStash
   module PluginManager
@@ -31,10 +32,11 @@ class Main < Clamp::Command
       subcommand "install", "Install a Logstash plugin", LogStash::PluginManager::Install
       subcommand "remove", "Remove a Logstash plugin", LogStash::PluginManager::Remove
       subcommand "update", "Update a plugin", LogStash::PluginManager::Update
-      subcommand "pack", "Package currently installed plugins", LogStash::PluginManager::Pack
-      subcommand "unpack", "Unpack packaged plugins", LogStash::PluginManager::Unpack
+      subcommand "pack", "Package currently installed plugins, Deprecated: Please use prepare-offline-pack instead", LogStash::PluginManager::Pack
+      subcommand "unpack", "Unpack packaged plugins, Deprecated: Please use prepare-offline-pack instead", LogStash::PluginManager::Unpack
       subcommand "generate", "Create the foundation for a new plugin", LogStash::PluginManager::Generate
       subcommand "uninstall", "Uninstall a plugin. Deprecated: Please use remove instead", LogStash::PluginManager::Remove
+      subcommand "prepare-offline-pack", "Create an archive of specified plugins to use for offline installation", LogStash::PluginManager::PrepareOfflinePack
     end
   end
 end