MongoDB Provisioning

On August 13th, 2019, OpenFarm switched off of our previous MongoDB vendor (Compose) and began self-hosting the database. This was in response to Compose deprecating their MongoDB Heroku addon.

Below are notes for future developers that must perform database maintenance.

Create a VM

• Create an Ubuntu droplet in Digital Ocean
• Using Digital Ocean's firewall service, block all ports except SSH (port 22) and MongoDB (port 3001). We will use the non-standard TCP port of 3001 instead of the MongoDB default.

Install MongoDB

Start with a fresh Ubuntu 18.04 installation. Run the following commands:

apt-get update
apt-get install -y mongodb
snap install micro --classic # Micro4Lyfe

Verify Mongo installation:

mongo --eval 'db.runCommand({ connectionStatus: 1 })'

Ensure you see "OK": 1 in the terminal output.

Creating Database Users

First, the admin account:

use admin

db.createUser({
  user: "admin",
  pwd: "password123", // Use a real password, please
  roles: [{ role: "userAdminAnyDatabase", db: "admin" }]
});

Then an account for the Rails app:

use openfarm_production

db.createUser({
  user: "openfarm_production",
  pwd: "password123", // Use a real password, please
  roles: [{ role: "readWrite", db: "openfarm_production" }]
});

Lock Down MongoDB config

We need to disallow anonymous database access. We also need to allow the Rails server to connect to the database node.

Overwrite /etc/mongodb.conf with this:

dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
port=3001
journal=true
auth=true
bind_ip=0.0.0.0

Update Systemd

Ubuntu uses Systemd to keep MongoDB running at all times. We need to change some of those configs. Do a google search for "Systemd unit files" if you are unfamiliar with systemd.

We must add an --auth flag to the ExecStart entry of the MongoDB unit file, usually found at /lib/systemd/system/mongodb.service. Replace the existing ExecStart line with this:

ExecStart=/usr/bin/mongod --auth --unixSocketPrefix=${SOCKETPATH} --config ${CONF} $DAEMON_OPTS

Restart the Database

systemctl daemon-reload # Because we changed the unit file
service mongodb restart

Ensure It Worked

Ensure you are able to log in with credentials. Run the following commands from a remote machine:

mongo --port 3001 -u "openfarm_admin" -p "password123" --authenticationDatabase "admin"
mongo --port 3001 -u "openfarm_production" -p "password123" --authenticationDatabase "openfarm_production"

Ensure you can't log in without credentials:

# You should not be able to do stuff with this account.
# Commands like `db.foo.insert({bar: "baz"})` should fail.
mongo --port 3001

Put Old Database Into Maintenance Mode

If we are migrating between database nodes, we need to backup the old database. This requires downtime, otherwise records created during the backup will be lost forever and consistency will be lost.

Use the "Maintenance Mode" toggle in Heroku's settings tab to bring the application offline while performing the next steps.

Dump Old Database from Host

mongodump \
  --authenticationDatabase DATABASE_HERE \
  -d DATABASE_HERE_TOO \
  -h my.database.com:1234 \
  -u admin456 \
  -p password123 \
  --out path/to/latest_dump

Restore New Database to Target

mongorestore \
  -d database567 \
  -h localhost:3001 \
  -u admin456 \
  -p password123 \
  path/to/latest_dump/foo

Perform Regular Backups

You will want to create database dumps regularly for emergency planning.

mongodump \
  --authenticationDatabase DATABASE_HERE \
  -d openfarm_production \
  -h my.database.com:1234 \
  -u admin456 \
  -p password123 \
  --out path/to/latest_dump

Perform Security Audits

Use automated tools to find vulns in the instance, such as MongoAudit.