The Conflict With youtube_dl and ruby youtube dl

The short answer is because oversight. The slightly longer answer is documented here and here, but I'll repost what goes on and go more in depth:

This gem, youtube-dl.rb, searches for a youtube-dl binary in your PATH before falling back to the built in youtube-dl. But the youtube_dl gem includes a binary named youtube-dl which from what I can tell is basically a really old version of youtube-dl. When youtube_dl (the gem) is installed, rbenv will make a Ruby file called youtube-dl to correspond with the executable youtube_dl (the gem) includes, which basically figures out the Ruby environment you are using at the moment and redirects it to the appropriate directory. That's where the Gem::LoadError is coming from.

For the sake of this document, I'm going to refer to each gem by their exact name, and any file in code form. For the tool itself, I will link to the original project. Examples:

• youtube_dl
• ruby-youtube-dl
• youtube-dl.rb
• youtube-dl
• youtube_dl.rb
• youtube-dl

Let's peep this out

Executables, outside the scope of this gem

youtube_dl and ruby-youtube-dl include a really old version of youtube-dl as an executable file. This is really bad for several reasons.

1. It's really old and outdated, and the maintainers aren't making an effort to keep them up-to-date.
2. It completely breaks Rubygems.

As for point one, yes you can technically do youtube-dl --update to get the latest version, but it's still sort of shitty. Point two is where the trouble starts. You know how in your Gemspecs you have to specify the executables explicitly? There's a reason for that. Rubygems doesn't expose your raw files in the bin/ or exe/ directory. It creates the following file:

#!/usr/bin/env ruby_executable_hooks
#
# This file was generated by RubyGems.
#
# The application 'my-awesome-gem' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

version = ">= 0"

if ARGV.first
  str = ARGV.first
  str = str.dup.force_encoding("BINARY") if str.respond_to? :force_encoding
  if str =~ /\A_(.*)_\z/ and Gem::Version.correct?($1) then
    version = $1
    ARGV.shift
  end
end

gem 'my-awesome-gem', version
load Gem.bin_path('my-awesome-gem', 'my-awesome-gem', version)

Now you will notice on the last line it loads your executable straight into that Ruby instance. Last time I checked, MRI can't parse Python. The documentation for the spec.executables method even states:

These files must be executable Ruby files. Files that use bash or other interpreters will not work.

That's why when you try to run youtube-dl this will happen:

$ youtube-dl 
/usr/local/rvm/gems/ruby-2.2.1/bin/youtube-dl:23:in `load': /usr/local/rvm/gems/ruby-2.2.1/gems/youtube_dl-0.0.2/bin/youtube-dl:2: Invalid char `\x03' in expression (SyntaxError)
	from /usr/local/rvm/gems/ruby-2.2.1/bin/youtube-dl:23:in `<main>'
	from /usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
	from /usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `<main>' 

This is why, sadly, youtube_dl and ruby-youtube-dl must be completely removed and not accessible from your PATH at all, unless you have a system version of youtube-dl installed. Even then you must make sure you have your PATH set correctly because most Ruby version managers like to take precedence over everything else. For example, even when I have youtube-dl installed with pip, whenever I try to run it, I will hit rbenv's shim first.

$ pip install youtube-dl
Downloading/unpacking youtube-dl
  Downloading youtube_dl-2016.03.14-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: youtube-dl
Successfully installed youtube-dl
Cleaning up...
$ youtube-dl
bash: /root/.rbenv/shims/youtube-dl: No such file or directory

!! IMPORTANT NOTE: Also when you do uninstall the offending gems, ALSO UNINSTALL EXECUTABLES

$ gem uninstall youtube_dl
Remove executables:
	youtube-dl

in addition to the gem? [Yn]  n
Executables and scripts will remain installed.
Successfully uninstalled youtube_dl-0.0.2
$ youtube-dl
/usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/site_ruby/2.2.0/rubygems/dependency.rb:315:in `to_specs': Could not find 'youtube_dl' (>= 0) among 25 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=/usr/local/rvm/gems/ruby-2.2.1:/usr/local/rvm/gems/ruby-2.2.1@global', execute `gem env` for more information
	from /usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/site_ruby/2.2.0/rubygems/dependency.rb:324:in `to_spec'
	from /usr/local/rvm/rubies/ruby-2.2.1/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:64:in `gem'
	from /usr/local/rvm/gems/ruby-2.2.1/bin/youtube-dl:22:in `<main>'
	from /usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `eval'
	from /usr/local/rvm/gems/ruby-2.2.1/bin/ruby_executable_hooks:15:in `<main>'

What this means for youtube-dl.rb

When we were designing youtube-dl.rb we wanted it to be largely future proof. Any API changes to youtube-dl wouldn't require much if any work on our part. We wanted our users to be able to install youtube-dl.rb very easily, with minimal work to get running, so we followed what Pygments.rb was doing and including a vendor youtube-dl that we could reference directly. But youtube-dl is a very rapidly-changing project. There's a new version every few days, so we had to plan for when we couldn't keep updating our gem with the latest version of youtube-dl.

When looking for a youtube-dl executable, youtube-dl.rb will first search your path using a pure Ruby implementation of which taken from StackOverflow:

def which(cmd)
  exts = ENV['PATHEXT'] ? ENV['PATHEXT'].split(';') : ['']
  ENV['PATH'].split(File::PATH_SEPARATOR).each do |path|
    exts.each do |ext|
      exe = File.join(path, "#{cmd}#{ext}")
      return exe if File.executable?(exe) && !File.directory?(exe)
    end
  end
  nil
end

This insures compatibility on all platforms, not just OS X and Linux. If there's a youtube-dl found, then youtube-dl.rb will only use that installation of youtube-dl. On the other hand, if it doesn't detect youtube-dl, it will revert to using the one provided with the gem, in the vendor/ directory. Only youtube-dl.rb can use this vendor youtube-dl as it addresses it directly and doesn't modify your PATH in any way. Now of course given the evidence above, if youtube-dl.rb finds some executable called youtube-dl in your PATH it's going to automatically assume that that is youtube-dl. We could introduce checks to see if it is actually youtube-dl but that would slow it down, and it's already slow enough shelling out to an external executable, which is why we make every last attempt to only do things in one go and cache excessively. We are currently looking toward alternatives to shelling out (borrowing from Pygments.rb again and using a Mentos-like system?) but have no plans to drastically change anything any time soon.

TLDR: Because of questionable design choices, youtube_dl and ruby-youtube-dl needs to be uninstalled completely, executables included.

If anyone has any workarounds or suggestions because telling people they have to uninstall something to make our product work feels incredibly wrong, please open a PR or Issue.