Skip to content

Disaster Recovery

Jump to bottom
rsiddall edited this page Sep 3, 2017 · 5 revisions

How to bring up a replacement BigBlueButton server (disaster recovery)

  1. Create a server or virtual machine with Ubuntu 14.04.05 64 bit, with at least 4GB memory with swap enabled. BBB 1.0 requires this distribution and attempting to install on a different distribution may fail.

  2. Create a non-privileged user to do the install as, e.g. "ubuntu". (This is done automatically when creating the server for some cloud providers.) The user should have a home directory as you will be installing the customized version of BBB into a subfolder of the home directory. The user on BBB is called 'ubuntu', so restoring the backup will be more complicated if you use a different user name. You'll want to give the user rights to run just about everything via sudo without entering a password.

    This is something along the lines of:

    useradd -m ubuntu
sudo visudo

    and then add to the bottom of the file:

    ubuntu ALL=(ALL) NOPASSWD: ALL

  3. If desired, add your public key to the ubuntu user's ~/.ssh/authorized_keys so you can log in via SSH without entering a password.

  4. Make sure the host name is correct

    hostname

  5. Log in as the un-privileged user, and install Git.

    sudo apt-get install git

  6. Clone the bbb-install repository from https://github.com/rsiddall/bbb-install:

    git clone https://github.com/rsiddall/bbb-install.git

    (You may need to select the BBB 1.0 branch to get the correct installation script if the script has been updated for a later version of BBB.)

  7. Copy in files you'll need to run the BBB installation script.

    • SSL certificate, key, and certificate authority files for use with nginx. These must be in ~/bbb-install/ssl.crt, ssl.key, and ssl.ca.

  8. Look at /etc/hosts and ensure that either the server's hostname and FQDN do not appear, or are set to its public IP address, not 127.0.1.1.

  9. Check that the server's FQDN resolves to the correct IP address in public DNS.

  10. Run the BBB installation script (which just implements the steps at the BBB 1.0 installation instructions at [http://docs.bigbluebutton.org/10install.html])

    cd bbb-install
./bbb-install -n `hostname -f` -s `hostname -f` -y # -e if installing on a virtual machine

  11. Generally answer yes when APT asks you if you want to install something. (Using the -y flag answers most of the following for you.)

    • Answer yes to upgrading Ubuntu to latest version (of 14.04)
    • Answer yes to upgrading bind9, etc.
    • Hit ENTER to accept adding libreoffice to software sources
    • Answer yes to installing many A/V libraries
    • Wait while FFMPEG gets installed. Ignore the warnings. (Use a PPA to speed this up when just testing.)
    • Answer yes to installing BBB itself
    • Answer yes to installing the libreoffice packages without verification (change in LibreOffice since instructions were written)
    • Wait for BBB to install
    • Hit tab to select OK, then hit space, then hit tab to select YES then hit space to accept Microsoft font license
    • Note warning about "Don't run Bundler as root" (weakness in BBB install instructions)
    • Note warning about "Rubygems 1.8.23 is not threadsafe" (weakness in BBB install instructions)
    • Wait for ruby gems to install
    • Wait for initialization to complete
    • Wait for DH parameters to generate
    • Hit return to accept OK to create iptables ipv4 rules
    • Hit return to accept OK to create iptables ipv6 rules
    • Wait for FreeSWITCH to start

  12. Test your BBB installation. There's no point in proceeding if BBB is not working at this point

    sudo bbb-conf --check # And inspect output
netstat -nlp # Check listening on correct ports

  13. Install duply so you can restore our duplicity backups from Amazon S3

    sudo apt-get install -y duply

  14. Create a duply profile called 'bbb', and then overwrite it with your saved version

    sudo mkdir /etc/duply # To get profiles in /etc/duply and not ~/.duply
sudo duply bbb create
sudo scp -pr me@remoteserver:/path/to/profile /etc/duply/bbb
sudo duply bbb status # Check validity and ability to access backups

  15. Install the daily backup cron job only if you want to update the S3 backup bucket from this server. Only one server should be updating this bucket, so disable it on any existing servers when enabling it on this server.

    sudo echo "#!/bin/sh\nduply bbb backup" > /etc/cron.daily/backup-bbb

  16. Run the dev-bbb-install script to install the tools required for BBB development, AND clone our custom BBB installation into /dev/bigbluebutton:

    cd ~/bbb-install
./dev-bbb-install -b https://[email protected]/someproject/bbb-server.git

  17. Restore the most recent version of our custom BBB installation. (duply sets files to be owned by the same user and group as on the original system, even if the UID and GID of those users have changed. So, restore has to be done after the users have been created on the new system during initial setup or by installing the BBB packages.)

    sudo duply bbb restore /home/ubuntu/dev 1d

  18. Start our customized BBB

    sudo bbb-conf --clean

  19. Check for correct operation:

    • Firewall
    • BBB itself (should have failed server's meetings and banned word list)
    sudo bbb-conf --check # And inspect output

  20. Grab the secret for this BBB installation and paste it into the control web site settings (or set this server's secret to a known value):

    sudo bbb-conf --secret
Clone this wiki locally