Remove Flyway and move migrations to Sequelize (#97)

* Ensure sequalizer cli can create models

Without passing in additional configuration options.

* Add instructions for connecting to db w/dotenv

You can use dotenv to load the enviornment variable and it is much
cleaner than exporting than manually.

* Write migration in Sequelize

I looked into a bunch of ways of doing this automatically but none of
them worked with the current version of sequelize but it is the most
asked for plus-oned feature there is.

I just ended up writting it up manually.

* Switch JSON to JSONB

https://www.depesz.com/2014/03/25/waiting-for-9-4-introduce-jsonb-a-structured-format-for-storing-json/

* Use database dump to create migration file

* Add initial data with seeds

* Deploy without flyway

* Fix deploy to work without flyway

Additionally remove flyway migraions and add deploy documentation.

* Add a Vagrant up message to help users find app

* Remove long hardcoded paths in ansible command

* Add a new github oauth app just for Vagrant

This has the correct redirects.

* Change datatype in initalial migration to jsonb

* Use the dotenv-cli to load enviornment vars

https://www.npmjs.com/package/dotenv-cli

* Fix Addd typo

* Fix the undoing the original migration

Now `yarn run sequelize -- db:migrate:undo` works.

* Rename Database.md to database.md

resolving merge conflict due to capitalization file name change (Database.md -> database.md)

* update documentation for users who will migrate from flyway to sequelize migration system

* Delete .gitignore

Co-authored-by: Erika Miguel <[email protected]>

.gitignore

@@ -117,3 +117,4 @@ deploy/ansible-vault-password.txt
 
 # Vagrant
 /.vagrant/
+deploy/provision.retry

.sequelizerc

@@ -0,0 +1,7 @@
+var path = require('path')
+
+module.exports = {
+  'models-path': path.resolve('server', 'models'),
+  'migrations-path': path.resolve('server', 'migrations'),
+  'seeders-path': path.resolve('server', 'seeders')
+}

Vagrantfile

@@ -13,6 +13,8 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
   # conflict with that of another project.
   config.vm.network :private_network, ip: '192.168.10.40'
 
+  config.vm.post_up_message = 'The app is now running at http://192.168.10.40/'
+
   # A specific name looks much better than "default" in ansible output.
   config.vm.define 'vagrant'
 

config/config.js

@@ -0,0 +1,17 @@
+module.exports = {
+    development: {
+        database: process.env.PGDATABASE,
+        username: process.env.PGUSER,
+        password: process.env.PGPASSWORD,
+        host: process.env.PGHOST,
+        port: process.env.PGPORT,
+        dialect: 'postgres',
+        dialectOption: {
+            ssl: true,
+            native: true,
+            multipleStatements: true
+        },
+        seederStorage: 'sequelize',
+        logging: console.log // eslint-disable-line no-console
+    }
+};

db/scripts/db_init.sh

@@ -6,7 +6,7 @@ source $1
 
 result="$(psql -tAc "SELECT 1 FROM pg_database WHERE datname='${PGDATABASE}'")"
 
-if [ $result = '1' ]; then
+if [[ $result = '1' ]]; then
   echo "Database ${PGDATABASE} already exists."
 else
   echo "Creating ${PGDATABASE} database..."

deploy/files/config-development.env

@@ -1,5 +1,4 @@
 API_SERVER=http://192.168.10.40
-GITHUB_CLIENT_ID=c7e16364e8663f8b2ae7
 PORT=80
 PGDATABASE=aria_at_report
 PGUSER=atr
@@ -8,8 +7,8 @@ PGHOST=localhost
 PGPORT=5432
 ENVIRONMENT=dev
 
-GITHUB_CLIENT_ID=c7e16364e8663f8b2ae7
-GITHUB_CLIENT_SECRET=2ebd9e96e2b3287ab274726c196ebf0020fdeb7c
+GITHUB_CLIENT_ID=7190d52e3c3ef8e48e70
+GITHUB_CLIENT_SECRET=a5eebe76bdaaf126e34a9b5a5a9686cf1cdbb71f
 GITHUB_ORGANIZATION=bocoup
 GITHUB_TEAM_QUERY="ARIA-AT Report"
 GITHUB_TEAM_TESTER="ARIA-AT Report Testers"

deploy/roles/application/tasks/main.yml

@@ -41,7 +41,12 @@
     chdir: '{{source_dir}}'
 
 - name: Migrate database
-  command: ./db/scripts/db_migrate.sh {{environment_config.dest}}
+  shell: DOTENV_CONFIG_PATH={{environment_config.dest}} node -r ./node_modules/dotenv/config $(npm bin)/sequelize-cli db:migrate --config ./config/config.js --migrations-path ./server/migrations/
+  args:
+    chdir: '{{source_dir}}'
+
+- name: Seed database
+  shell: DOTENV_CONFIG_PATH={{environment_config.dest}} node -r ./node_modules/dotenv/config $(npm bin)/sequelize-cli db:seed:all --config ./config/config.js --seeders-path ./server/seeders/
   args:
     chdir: '{{source_dir}}'
 

deploy/roles/database/tasks/main.yml

@@ -4,33 +4,3 @@
     name:
       - postgresql
       - postgresql-contrib
-
-- name: Download Flyway
-  get_url:
-    url: https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/6.4.1/flyway-commandline-6.4.1-linux-x64.tar.gz
-    checksum: md5:f2553075b96f2114e1a170d8765f4437
-    dest: /root/flyway-6.4.1.tar.gz
-  register: download_flyway
-
-- name: Define fact
-  set_fact:
-    flyway_root: /opt/{{download_flyway.dest | basename | regex_replace('\.tar\.gz$', '')}}
-
-- name: Extract Flyway
-  unarchive:
-    src: '{{download_flyway.dest}}'
-    dest: /opt
-    creates: '{{flyway_root}}'
-    remote_src: True
-
-- name: Allow execution
-  file:
-    state: file
-    mode: 0755
-    dest: '{{flyway_root}}/flyway'
-
-- name: Install Flyway
-  file:
-    state: link
-    dest: /usr/local/bin/flyway
-    src: '{{flyway_root}}/flyway'

deploy/roles/nodejs/tasks/main.yml

@@ -7,7 +7,7 @@
     - https://dl.yarnpkg.com/debian/pubkey.gpg
     - https://deb.nodesource.com/gpgkey/nodesource.gpg.key
 
-- name: Addd software repositories
+- name: Add software repositories
   apt_repository:
     state: present
     repo: '{{item}}'

docs/Deploy.md

@@ -0,0 +1,45 @@
+# Deploying
+
+## Prerequisite
+
+- Install [ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-ansible) which is used to deploy the application.
+- Get a copy of `ansible-vault-password.txt`
+- Get a copy of the private key used to access the server get your own public
+  key placed on the server
+
+## Testing the deploy with Vagrant
+
+You can test that a deploy will function by running the deploy in a vagrant that
+closely matches that of staging and production.
+
+First install [vagrant](https://www.vagrantup.com/docs/installation/).
+
+Copy your `ansible-vault-password.txt` to `deploy/ansible-vault-password.txt`.
+
+Start the vagrant by running `vagrant up`.
+
+If you make any changes locally and want to run them again `vagrant rsync &&
+vagrant up --provision`.
+
+If you want to debug you can run `vagrant ssh` to ssh into the vagrant box. You
+can view logging from ansible with `sudo -i cat /var/log/messages`.
+
+Once the vagrant box is up you can test by running by going to the ip configured
+in the `Vagrantfile` [192.168.10.40](192.168.10.40).
+
+## Updating passwords
+
+If you need to update a password to the staging server use ansible vault like:
+`ansible-vault decrypt deploy/files/config-staging.env`.
+
+## How to deploy
+
+0. Get the prerequisites.
+
+1. Get a copy of the ansible vault password and place it at `deploy/ansible-vault-password.txt`.
+
+2. `cd deploy`
+
+3. `ansible-playbook provision.yml --inventory inventory/staging.yml`
+
+4. Check if this has been working `http://aria-at-staging.w3.org/`