Update project deployment tooling, add docker-compose development environment #226

tnation14 · 2017-08-30T19:23:19Z

Overview

This PR brings this project into alignment with our standard setup by updating the deployment tooling to use Terraform and AWS instead of the s3_website gem, and replacing the Vagrant development environment with a docker-compose based setup.

Notable Changes

Repace npm package manager with yarn
Replace Vagrant setup with docker-compose
Replace s3_website deployment with Terraform/AWS CLI
Add docker-based deployment tooling

Demo

~~Optional. Screenshots, curl examples, etc.~~

Notes

I couldn't verify that package-lock.json is exactly the same as yarn.lock. However, I created yarn.lock using the following steps:

$ npm install # create node_modules from package.json
$ yarn import # create yarn.lock from the contents of the node_modules folder

Initially, dockerized Karma tests were failing in CI (Ubuntu 16.04) with the following error:

ChromeHeadless stderr: Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted

I was able to get things working by creating a Karma custom launcher that passes --no-sandbox to Chomium. The custom launcher overrode the default launcher flags, so I had to include --disable-gpu, --headless, and --remote-debugging-port as well.

Testing Instructions

Rebuild the app VM, ensure the process completes successfully.
Run script/server, make sure the site is visible at http://localhost:4200
See Jenkins output
See https://lab.staging.climate.azavea.com

Closes #223

tnation14 · 2017-08-30T19:24:25Z

@CloudNiner could you weigh in on the switch to yarn and my modifications to the Karma configuration? Just want to be sure that it's not going to break anything.

tnation14 · 2017-08-30T19:42:42Z

deployment/terraform/cdn.tf

@@ -0,0 +1,75 @@
+resource "aws_cloudfront_distribution" "cdn" {


Reminder to tear down the old infrastructure once this PR is merged.

CloudNiner · 2017-08-31T15:09:19Z

Is there any significant advantage to yarn now that npm has the package-lock.json file?

I'll defer to @flibbertigibbet regarding the karma conf, she was just updating dependencies in #228 so it's probably worth rebasing this against develop once that's merged to ensure you've got the most up to date packages and the testing config here still works.

tnation14 · 2017-08-31T15:36:07Z

The main reasons for using yarn here are that it alleviates the node-sass compilation issue that was being fixed in ansible/climate-change-lab.app, and it's already installed in the node Docker image. NPM's installed as well, but it's an older version so we'd have to update it in the Dockerfile.

Re: Karma, I'll try testing again after #228 is merged. Thanks!

CloudNiner · 2017-08-31T15:40:15Z

👍 to yarn then.

CloudNiner · 2017-08-31T15:51:51Z

I built the VM and then ran ./scripts/server. The docker containers were pulled/built successfully, but then I got this error:

vagrant@app:/vagrant$ ./scripts/server
yarn run v0.24.4
warning You don't appear to have an internet connection. Try the --offline flag to use the cache for registry queries.
$ ng serve
sh: 1: ng: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It occurs regardless of whether I use the --vm option.

tnation14 · 2017-08-31T16:04:28Z

Did you build the VM using scripts/setup, or vagrant up?

CloudNiner · 2017-08-31T18:55:12Z

I just rebuilt after pulling down the latest commit via ./scripts/setup. Then ran ./scripts/server in the VM. The webpack server started.

However, I can't hit localhost:4200 from my browser. Unsure if related, but I'm still seeing this yarn warning:

warning You don't appear to have an internet connection. Try the --offline flag to use the cache for registry queries.

tnation14 · 2017-08-31T19:36:05Z

I was able to hit localhost:4200, even with the internet connection warning. Did you run scripts/server --vm, or just scripts/server?

hectcastro · 2017-09-01T14:14:54Z

deployment/terraform/storage.tf

+#
+# S3 resources
+#
+data "template_file" "read_only_bucket_policy" {


I believe that we have parts of this modularized in https://github.com/azavea/terraform-aws-s3-origin.

hectcastro · 2017-09-01T14:35:41Z

GitHub isn't letting me leave comments on these lines:

CloudNiner · 2017-09-05T19:12:07Z

@tnation14 sorry if you were waiting on me for this, I was able to hit the application after reprovisioning and doing ./scripts/server --vm. Although I still see the "no internet" error that you're seeing.

CloudNiner

This LGTM once this has been rebased against current develop, tests pass and we validate that the yarn lockfile is up to date.

- Install docker-compose and NTP in the VM - cd to /vagrant on login - Add docker container with Node, NPM, and Karma dev dependencies - scripts/update now installs node dependencies

hectcastro · 2017-09-07T11:26:10Z

I'm not super worried about the "no internet" error. We see it in all of the other Yarn + Docker projects, but it doesn't appear to have any significant negative effects.

See: yarnpkg/yarn#749

- Read-only static site bucket - Logs bucket - CloudFront distribution with S3 origin - DNS entry for CloudFront distribution - Remove s3_website gem

- Add docker-compose.ci.yml, with a terraform 0.9.11 container for deployments - Add STRATA script to deploy terraform changes and upload files to site bucket - Don’t run STRTA scripts in the VM - Make linter output available outside of app container

tnation14 added the in review label Aug 30, 2017

tnation14 requested a review from hectcastro August 30, 2017 19:23

tnation14 self-assigned this Aug 30, 2017

tnation14 requested a review from CloudNiner August 30, 2017 19:28

tnation14 commented Aug 30, 2017

View reviewed changes

tnation14 force-pushed the feature/tnation/update-deployment-tooling branch from 029199b to 0e451ce Compare August 31, 2017 12:38

hectcastro reviewed Sep 1, 2017

View reviewed changes

tnation14 force-pushed the feature/tnation/update-deployment-tooling branch from 42c454d to faae188 Compare September 1, 2017 15:26

CloudNiner reviewed Sep 5, 2017

View reviewed changes

Manage development dependencies with docker-compose

71f6065

- Install docker-compose and NTP in the VM - cd to /vagrant on login - Add docker container with Node, NPM, and Karma dev dependencies - scripts/update now installs node dependencies

tnation14 force-pushed the feature/tnation/update-deployment-tooling branch 2 times, most recently from e407d2f to 828e0a3 Compare September 6, 2017 12:58

hectcastro approved these changes Sep 7, 2017

View reviewed changes

CloudNiner approved these changes Sep 7, 2017

View reviewed changes

tnation14 added 3 commits September 7, 2017 08:33

Use Yarn instead of NPM

195be88

Add terraform configuration for static site

1f8719b

- Read-only static site bucket - Logs bucket - CloudFront distribution with S3 origin - DNS entry for CloudFront distribution - Remove s3_website gem

tnation14 force-pushed the feature/tnation/update-deployment-tooling branch from 79d267f to 840052d Compare September 7, 2017 12:33

tnation14 merged commit 6067bc4 into develop Sep 7, 2017

hectcastro removed the in review label Sep 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update project deployment tooling, add docker-compose development environment #226

Update project deployment tooling, add docker-compose development environment #226

tnation14 commented Aug 30, 2017 •

edited

Loading

tnation14 commented Aug 30, 2017 •

edited

Loading

tnation14 Aug 30, 2017

CloudNiner commented Aug 31, 2017

tnation14 commented Aug 31, 2017

CloudNiner commented Aug 31, 2017

CloudNiner commented Aug 31, 2017

tnation14 commented Aug 31, 2017

CloudNiner commented Aug 31, 2017

tnation14 commented Aug 31, 2017 •

edited

Loading

hectcastro Sep 1, 2017

hectcastro commented Sep 1, 2017

CloudNiner commented Sep 5, 2017

CloudNiner left a comment

hectcastro commented Sep 7, 2017

		@@ -0,0 +1,75 @@
		resource "aws_cloudfront_distribution" "cdn" {

Update project deployment tooling, add docker-compose development environment #226

Update project deployment tooling, add docker-compose development environment #226

Conversation

tnation14 commented Aug 30, 2017 • edited Loading

Overview

Notable Changes

Demo

Notes

Testing Instructions

tnation14 commented Aug 30, 2017 • edited Loading

tnation14 Aug 30, 2017

Choose a reason for hiding this comment

CloudNiner commented Aug 31, 2017

tnation14 commented Aug 31, 2017

CloudNiner commented Aug 31, 2017

CloudNiner commented Aug 31, 2017

tnation14 commented Aug 31, 2017

CloudNiner commented Aug 31, 2017

tnation14 commented Aug 31, 2017 • edited Loading

hectcastro Sep 1, 2017

Choose a reason for hiding this comment

hectcastro commented Sep 1, 2017

CloudNiner commented Sep 5, 2017

CloudNiner left a comment

Choose a reason for hiding this comment

hectcastro commented Sep 7, 2017

tnation14 commented Aug 30, 2017 •

edited

Loading

tnation14 commented Aug 30, 2017 •

edited

Loading

tnation14 commented Aug 31, 2017 •

edited

Loading