Update Installation Documentation

[bcoles]

The installation procedure and documentation is a mess.

• Two install scripts:
  • https://github.com/beefproject/beef/blob/master/install-beef
  • https://github.com/beefproject/beef/blob/master/install
    Edit: These have been merged into one install file.
• Wiki Installation page:
  • https://github.com/beefproject/beef/wiki/Installation
    Edit: Done. The installation page now mirrors INSTALL.txt file.
• Two README files:
  • https://github.com/beefproject/beef/blob/master/README
  • https://github.com/beefproject/beef/blob/master/README.mkd
  • The README file should be dropped in favor of the README.mkd (which should probably be renamed to README.md)
    Edit: These have been merged into one README.md file.
  • The READMEs make reference to:
    • https://raw.githubusercontent.com/beefproject/beef/a6a7536e736e7788e12df91756a8f132ced24970/install-beef
      Edit: This reference has been removed.

These all need to be updated and consolidated.

[plumpNation]

Why not remove the rvm instructions and just point to the rvm home page?

[bcoles]

@plumpNation That's a good idea, but doesn't fix the install scripts, and inconsistency between install scripts.

[plumpNation]

Ah yeah, I didn't read the issue properly and mistook it for the installation documentation needing work. My bad. Thanks for all your work.

[0xmachos]

Hi,

I've been working on an install script:
https://github.com/AbertayHackers/BeEF/blob/master/install.sh

Currently it only supports Debian 9 and Kali Rolling 2018.1.

Let me know if it's useful or if I can help with updating the scripts already in the beef repo.

[bcoles]

@0xmachos cool. It looks like you've gone to a lot of effort already.

There are some dependencies missing for your script. The wiki page upon which you based the script doesn't mention most of the OS level dependencies, which can be found in the install scripts. Specifically:

sudo apt-get install build-essential openssl libreadline6 libreadline6-dev zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev autoconf libc6-dev libncurses5-dev automake libtool bison subversion nodejs

I have no idea which of the above are still required. The install script is ancient.

Fortunately the installation procedure for these dependencies is pretty standard for most Linux systems, simply depending on the package manager (yum, apt, etc)

rvm is a problem. For example, if the user already has rbenv, there's no point in using rvm. It would be nice if we could ditch rvm / rbenv from the install process entirely, but the state of OS ruby distributions means that's just not going to happen any time soon.

Ideally the script should check for rbenv or rvm first, then check if the installed Ruby versions are compatible, then, if a compatible version isn't found, install a compatible version of Ruby (or prompt to do so). We're doing something similar now, but only for rvm :

if [ -e $HOME/.rvm/scripts/rvm ]; then
	source $HOME/.rvm/scripts/rvm
elif [ -e /usr/local/rvm/scripts/rvm ]; then
	source /usr/local/rvm/scripts/rvm
else
	source /etc/profile.d/rvm.sh
fi

Next comes bundler, which is meant to be used as a development tool rather than an installer. At least, that's the theory the developers subscribe to - in practice bundle install for installation is fairly common.

Currently the Gemfile makes use of environment variables to prevent installation of development gems. That's kind of janky. Rather than rely on environment variables, I would prefer if the install script invoked bundle install --without development. An alternative is shipping a .bundler file with the default install command (--without development).

With that out of the way, the biggest issue with writing stable install scripts is testing to ensure they work across many OS.

If you or anyone else is keen to help, the best thing to do would be to review the existing scripts and documentation described in this issue and condense it down into one script and one wiki page.

From there, the installation steps can be tested on various OS and updated as necessary.

As an aside, I'd really like to ditch therubyracer gem, if possible. It causes problems.

The second biggest issue is trying to be overly controlling. Trying to handle every edge case for every dependency for every distribution on every platform is a maintenance nightmare that we don't have the developer bandwidth to support. Instead, the installation should be as generic as possible, bailing out with useful error messages, and instructions if it's a known issue. As an example, there's a known issue with the psych gem on Kali.

Finally: colors. The script has to use colors. For no reason other than I said so.

[bcoles]

Related #1356

[0xmachos]

@bcoles

There are some dependencies missing for your script. The wiki page upon which you based the script doesn't mention most of the OS level dependencies, which can be found in the install scripts.

A quick check shows that Debian 9 needs sqlite3 to be installed however the package libreadline6 has no install candidate. When I get some time I'll check for other packages missing install candidates for Debian 9 and Kali 2018.1. Once I work this out i'll add to the OS level deps to my script.

Ideally the script should check for rbenv or rvm first, then check if the installed Ruby versions are compatible, then, if a compatible version isn't found, install a compatible version of Ruby (or prompt to do so). We're doing something similar now, but only for rvm 👍

My script also has rvm covered. I've never heard of rbenv before. I don't code in Ruby and BeEF is the only Ruby based project I interact with regularly. I'll have a look at adding checks for rbenv though, I can't imagine that it's too much work.

Currently the Gemfile makes use of environment variables to prevent installation of development gems. That's kind of janky. Rather than rely on environment variables, I would prefer if the install script invoked bundle install --without development. An alternative is shipping a .bundler file with the default install command (--without development).

Had a quick read of the bundle install man page, looks like the easiest option here is your first suggestion. Again i'm not familiar with Ruby, the man page indicates that --without development will skip the install of gems in the development group as stated in the Gemfile. I see the ENV check you mention however I'm confused as I don't see any gems marked as being in the development group. Could you correct me here or point me at something I can read to correct myself?

With that out of the way, the biggest issue with writing stable install scripts is testing to ensure they work across many OS.

I'm happy to cover testing for Kali Rolling and Debian Stable.

If you or anyone else is keen to help, the best thing to do would be to review the existing scripts and documentation described in this issue and condense it down into one script and one wiki page.

Once my script matures a little bit more I'll have a look at opening a PR to merge it with install-beef. My initial idea is to replace the existing Debian install functionality with my script and preserve the macOS and RedHat install functionality. Does that sound acceptable?

The second biggest issue is trying to be overly controlling.

I'll take another look at what I'm doing with this in mind, cheers.

[bcoles]

Regarding Debian, you'll probably find that many of the dependencies listed in the install file are required, it's just that you already had them installed.

I had a look at Kali - it looks like the dep is libreadline-dev rather than libreadline6 - I don't know where the latter came from and whether it was ever relevant.

Regarding development Gem group, it's the test group wrapped in the ENV check, not development, although development would probably be a better name.

[0xmachos]

I had a look at Kali - it looks like the dep is libreadline-dev rather than libreadline6 - I don't know where the latter came from and whether it was ever relevant.

Thanks for that.

Regarding development Gem group, it's the test group wrapped in the ENV check, not development, although development would probably be a better name.

Sorry if I'm being an idiot.
Does this mean that as the Gem group for Gems required for dev/testing is currently called test, bundle install should be invoked with --without test instead of --without development?

[bcoles]

That's correct. bundle install --without test development would make more sense, as a development group may or may not be used in the future.

[0xmachos]

That's correct. bundle install --without test development would make more sense, as a development group may or may not be used in the future.

Perfect, cheers for clarifying that.

[bcoles]

To clarify the Ruby / rvm / rbenv process flow, it should look something like this:

Is ruby installed? Is it larger than or equal to the version required?
|
|_ No _ are rvm or rbenv installed?
|       |
|       |_ No _ Warn the user that no Ruby version management system is available, and give a brief
|       |       description with links to install either of them, and advise to restart the install process once complete.
|       |
|       |_ Yes _ Notify the user a new version of Ruby will be installed, then install a suitable version of Ruby
|                (optionally, prompt to proceed before installing Ruby).
|                |
|                V
|_ Yes _ proceed with install

[0xmachos]

I've raised an issue rbenv/rbenv#1068 with the rbenv project about updating the Debian package.

Much easier to do apt install benv then cloning their repo and manually installing.

Given that they don't release new versions regularly I think apt install should work okay.

[bcoles]

I've updated the install script. It doesn't handle rvm / rbenv - opting instead to detect only the version of Ruby - allowing the user to manage their own Ruby environment.

I've removed the README.txt file in favor of the README.md.

Remaining TODO:

• Use bundle install --without test development
• Test the install script on various systems. In particular, check whether it works on various Linux distributions.
• Update the install_mac function in the install script (for Mac OSX systems)
• Add colored output
• Remove Gemfile.lock
• Update the contents of README.md
• Update the wiki

It's worth noting that Ruby 2.2 will be end-of-life in a couple weeks. The documentation and source will need to be updated to reflect the move to Ruby 2.3.

• Update BeEF and documentation to reflect minimum Ruby version of 2.3.

[bcoles]

I've updated the documentation to reflect the upgrade to Ruby 2.3. The .ruby-version file has had 2.3.0 for almost two years now, so hopefully the upgrade won't cause any problems.

I've tested the new install script on Linux Mint 18, Kali 2018, and Parrot 3.11. These are all Debian based.

json gem / ruby-dev package

It appears the ruby-dev operating system package is required for the json gem to install. I was under the impression that we don't even need the json gem in the Gemfile, because it's in standard lib, and could therefor be removed. I could be wrong. More investigation required.

Rainbow gem

The rubydns library continues to cause issues. The rake gem needs to be installed first, as rubydns depends on rainbow which depends on rake.

Workaround: gem install rake before bundle install

[0xmachos]

Are there any plans to move BeEF to a newer version of Ruby?
Installing 2.3.0 via RVM gives this warning

Install of ruby-2.3.0 - #complete 
Please be aware that you just installed a ruby that requires 3 patches just to be compiled on an up to date linux system.
This may have known and unaccounted for security vulnerabilities.
Please consider upgrading to ruby-2.4.1 which will have all of the latest security patches.

[bcoles]

@0xmachos Ruby 2.3+ is supported.

2.4.x is supported, but hasn't been tested as thoroughly.

2.5.x hasn't been tested at all.

Unfortunately, rbenv does not support fuzzy version matching, which is why the .ruby-version file contains 2.3.0, see:

• Support for abbreviated ruby version in .ruby-version file rbenv/rbenv#328

Is there a better way to specify a version? Currently BeEF checks if RUBY_VERSION < 2.3 on start.

[0xmachos]

The Ruby version check appears to be broken.

From install Line 91:

if [ `ruby -e "puts RUBY_VERSION.to_f >= ${MIN_RUBY_VER}"` = 'false' ]

This checks the version of the system Ruby install. Even if no Ruby version has been installed by RVM or rbenv the version check will pass and produce the following error:

[INFO]  rvm is installed
[INFO]  Detecting Ruby environment...
[INFO]  Ruby version 2.3.3 is installed
[INFO]  Detecting bundler gem...
[INFO]  Installing bundler gem
Fetching: bundler-1.16.1.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /var/lib/gems/2.3.0 directory.
machos@GLaDOS:~/Desktop/beef3$ rvm list

rvm rubies

# No rvm rubies installed yet. Try 'rvm help install'.

To reproduce:

• Run install on a fresh Debian 9 install
• install returns fatal as no ruby version manger
• Install RVM via the instructions install points to
  • In my case curl -sSL https://get.rvm.io | bash -s stable
• Re run install

It needs to check if a Ruby version has been installed by RVM or rbenv and if so what that version is.

For RVM this looks something like:

if rvm list default | grep -q "${ruby_version}" ; then

[bcoles]

@0xmachos Welp. I was hoping RVM would be smart enough to read the .ruby-version file.

Did you reload the terminal after installing RVM?

[bcoles]

Even if no Ruby version has been installed by RVM or rbenv the version check will pass and produce the following error:

The output you provided indicates that the install failed due to insufficient privileges to install Bundler. The code is worked as intended, as the version of Ruby installed is sufficient.

Using rvm or rbenv is not recommended. Instead, it's an alternative to using the rubyng ppa for operating systems with outdated Ruby.

install returns fatal as no ruby version manger

That code block is commented out...

[bcoles]

gem install bundler should probably be sudo gem install bundler but I have no idea whether that will play nicely with rvm/rbenv.

Edit: Nevermind, the user is prompted if sudo privileges are required.

[0xmachos]

It doesn't handle rvm / rbenv - opting instead to detect only the version of Ruby - allowing the user to manage their own Ruby environment.

Sorry I didn't see this comment about abandoning rvm/ rbenv support. I enabled that section of code to test it.

Googling the error

ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /var/lib/gems/2.3.0 directory.

Turns up a bunch of results which suggest using rvm or rbenv instead of using sudo to install gems.

I think it's probably a mistake to use sudo gem install bundler and sudo bundle install --without test development, there's a good chance this will break peoples setups.

It'd be more robust to detect if a version manager is present and if not fail.

[bcoles]

@0xmachos yeah; ditching rbenv / rvm for now.

Adding in proper detection for the environment, and offering the choice of installing system ruby, or rbvenv, or rvm, will take some effort. I opted for the simplest method: allowing the user to set up their own Ruby environment, rather than force installing RVM and Ruby and gems which is rather impolite.

I probably shouldn't have included the commented out section. I left it in for whoever takes over as I have limited time to contribute.

I've tested the installer with system Ruby on Linux Mint 18, Parrot, and Kali 2018 - the latter two seemingly being the most popular Linux operating systems for BeEF users (based on bug reports). It worked fine without prefacing bundle with sudo, as it asks for privs when required.

I have no idea how well it will work with rvm / rbenv installed. In theory, if they're installed, the .ruby-version file should be automatically detected and used. I've tested on a couple systems, one with rbenv and one with rvm, however they weren't clean systems.

I've also removed the Gemfile.lock file. From now on, gem version requirements should be specified in the Gemfile.

The remaining items for this issue are:

• Update the install_mac function in the install script (for Mac OSX systems)
• Update the wiki

Unfortunately I don't have a Mac, so someone else will need to do this.

[0xmachos]

Adding in proper detection for the environment, and offering the choice of installing system ruby, or rbvenv, or rvm, will take some effort. I opted for the simplest method: allowing the user to set up their own Ruby environment, rather than force installing RVM and Ruby and gems which is rather impolite.

Okay no worries, that also makes sense.

I have no idea how well it will work with rvm / rbenv installed. In theory, if they're installed, the .ruby-version file should be automatically detected and used. I've tested on a couple systems, one with rbenv and one with rvm, however they weren't clean systems.

@0xmachos Welp. I was hoping RVM would be smart enough to read the .ruby-version file.

I think RVM was detecting the required version from .ruby-version when I was testing. I just didn't realise it, again not used to Ruby.

Update the install_mac function in the install script (for Mac OSX systems)

I've got a Mac, I can have a look at implementing the install_mac function.

[bcoles]

@0xmachos

I've got a Mac, I can have a look at implementing the install_mac function.

That would be awesome, thanks! We need to know what needs to be installed with brew install, if anything, for the required gems to install cleanly.

Edit: XCode is likely to be required. Selenium is potentially required,

[0xmachos]

@bcoles

I'll try and have a look at install_mac today.

Has anymore work been done on figuring out which of the Linux dependancies are actually needed?
Per #1477 (comment)

I've got an issue tracking this in our BeEF script repo AbertayHackers/BeEF#2 however I've not had a chance to do anymore testing.

[bcoles]

@0xmachos unfortunately, no.

I have no idea why subversion is in the list of Ubuntu packages. The rest of the packages look sane.

The SQLite related packages are only required as the default database is SQLite. If you were using PostgreSQL or MySQL, then these dependencies would need to be included instead, along with some Gems (which are currently commented out in the Gemfile).

I've just tested the installer on Fedora 27, and it failed to install iconv-devel as the package couldn't be found. A workaround would be to install glibc-common instead - however I'm not sure that iconv is required at all, and so far it doesn't seem to be required from my minimal testing.

The second issue was related to the json gem (again). Running sudo yum install redhat-rpm-config resolved the issue.

The third and final issue related to the use of the uglifier gem. Same as #1504.
Bumping the version of therubyracer in Gemfile from 0.12.2 to 0.12.3, then bundle install, resolved the issue.

[bcoles]

CentOS 7 failed to find nodejs and iconv-devel packages. It also seems to package Ruby version 2.0.0.

[bcoles]

@0xmachos I've removed the subversion and iconv requirements from the installer.

The git dependency is only required for updating. ie, if users are using BeEF from the Git repository and want to be able to git pull to update.

The remaining dependencies are likely required; with the exception of SQLite as per above.

[0xmachos]

I've done some testing on macOS High Sierra 10.13.1. Using homebrew to install the dependencies I successfully installed BeEF, cloned a site and deployed the fake flash update module. I've not tested any other functionality. I'll open a PR to add the code for install_mac.

The following dependencies are required:

curl 
git 
nodejs
python3
openssl
readline 
libyaml
sqlite3
libxml2
autoconf
ncurses
automake
libtool
bison
wget

Removed

• python3-pip included with python3
• build-essential included with xcode
• zlib1g & zlib1g-dev included with xcode
• libssl-dev included with openssl
• libsqlite3-0 and libsqlite3-dev included with sqlite3
• libxslt1-dev incldued with xcode
• libc6-dev no brew formula for this. Likely included with xcode. It's total removal didn't throw any errors during install or cloning.

Changed

• libreadline6-dev to readline
• libyaml-dev to libyaml
• libxml2-dev to libxml2
• libncurses5-dev toncurses

Added

• wget

[bcoles]

Thanks @0xmachos

Looks good to me.