Certs, Build Events, Log Reliability

SSL Certs #568

SSL certs are now managed on the Rack or AWS account level with the convox certs tool.

You can add a label to your application manifest to automatically configure a self-signed cert in your load balancer:

web:
  labels:
    - convox.port.443.protocol=https
  ports:
    - 80:3000
    - 443:3000

Then you can replace this with a higher-quality certificate for your domain. You can generate a free AWS Certificate Manager (ACM) cert:

$ convox certs generate foo.example.org
Requesting certificate... OK, acm-01234567890

Or upload one of your existing certs:

$ convox certs create example.org.pub example.org.key
Uploading certificate... OK, cert-1234567890

Then you can attach these certificates to an app's process.

$ convox ssl update web:443 cert-1234567890
Updating certificate... OK

See the SSL docs for more details.

Build Events #572

Failed builds were failing to send webhook notifications. This behavior is now restored.

Power users can look at the Rack Event Logs with:

$ convox logs --app=convox --filter=EventSend --follow=false --since=20m
2016-04-25T20:58:16Z web:20160425202355/1eb3f413602b aws EventSend msg="{\"action\":\"build:create\",\"status\":\"error\",\"data\":{\"app\":\"httpd\",\"id\":\"BEAIOGUJGTU\",\"message\":\"exit status 1\"},\"timestamp\":\"2016-04-25T20:58:16.428582907Z\"}"
2016-04-25T20:58:16Z web:20160425202355/1eb3f413602b ns=kernel at=EventSend message-id="a0266efc-1cd8-54f6-bc46-21649bf06b5a"
2016-04-25T21:00:59Z web:20160425202355/1eb3f413602b aws EventSend msg="{\"action\":\"build:create\",\"status\":\"success\",\"data\":{\"app\":\"httpd\",\"id\":\"BIRNSOLKVUC\"},\"timestamp\":\"2016-04-25T21:00:59.002442447Z\"}"
2016-04-25T21:00:59Z web:20160425202355/1eb3f413602b ns=kernel at=EventSend message-id="1976c1a2-d4cd-5db6-a8c7-8e3a5306b920"
2016-04-25T21:01:02Z web:20160425202355/7971b3dd9ee6 aws EventSend msg="{\"action\":\"release:create\",\"status\":\"success\",\"data\":{\"app\":\"httpd\",\"id\":\"RQYRSVEXGLD\"},\"timestamp\":\"2016-04-25T21:01:02.065567824Z\"}"
2016-04-25T21:01:02Z web:20160425202355/7971b3dd9ee6 ns=kernel at=EventSend message-id="313b7ec7-5b3f-51a6-ba28-315fdcf1c150"
2016-04-25T21:01:04Z web:20160425202355/7971b3dd9ee6 models EventSend msg="{\"action\":\"release:promote\",\"status\":\"success\",\"data\":{\"app\":\"httpd\",\"id\":\"RQYRSVEXGLD\",\"rack\":\"convox\"},\"timestamp\":\"2016-04-25T21:01:04.657552667Z\"}"

Log Reliability convox/agent#22

Some errors in synchronizing and retrying around getting container logs out of Docker and into CloudWatch Logs have been root-caused and fixed.

Syslog, SSL, Cron

Run convox rack update to install.

Syslog service

You can now forward your logs to a generic syslog drain.

$ convox services create syslog --url tcp+tls://logs1.papertrailapp.com:12345
Creating syslog-3785 (syslog)... CREATING

Link the app with

$ convox services link syslog-3785 --app example-app
Linked syslog-3786 to example-app

Syslog Docs

SSL

SSL and load balancer protocols are now configured using labels in docker-compose.yml.

web:
  labels:
    - convox.port.443.protocol=https
    - convox.port.443.secure=true
  ports:
    - 443:5001

convox ssl create and convox ssl delete and associated API endpoints have been removed. The API endpoint for convox ssl update has been updated and is the sole command to manage SSL certs going forward.

convox start has also been updated to simulate the behavior of a production load balancer locally for better dev/prod parity.

See the SSL and Load Balancer docs

Cron

Convox now supports scheduled jobs that are configured via labels in docker-compose.yml.

For example,to run the command rake myjob every hour on the web process, you would configure the label like this:

web:
  labels:
    - convox.cron.myjob=0 * * * ? rake myjob

Scheduled Tasks Docs

Log Delivery and Tail Reliability

Infstructure

• convox/agent:0.67
  • Move to golang:1.6-alpine (3.3) base image. It's smaller in general and now layers are shared with the Rack API image
  • Write to CloudWatch logs with much simpler format: web:RXZMCQEPDKO/1d11a78279e0 Hello from Docker.
  • Emit custom metric for how many Docker logs lines were read. This offers new operational visibility into the log pipeline:

App Runtime

• Inject APP, RACK, and AWS_REGION into app process environment just like PROCESS and RELEASE. Re-deploy an app to pick these variables up.

API

• convox logs now streams from CloudWatch Logs which is more reliable and easier to use
• convox logs has new options. Try --since 2h, --follow false and --filter /auth

Roadmap Notes

Now that we're tailing from CloudWatch Logs, Kinesis isn't really needed. You can expect more updates that stop using then delete Kinesis streams from every rack and app. It'll come back as something you can optionally add with convox services add kinesis

20160415191544

• Terminating instances are removed from ELBs cleanly
• convox.port.80.protocol: "proxy" label to configure ELB ProxyProtocol Parameter
• convox start does file sync without volumes by default

Go 1.6 and vendor/

Migrates rack to Go 1.6 and vendor/ directory. Greatly improves rack build times by removing vendor redundancies.

Build Bugfixes

• Validate manifest process names are deployable. Fixes #364
• Build from docker-compose Dockerfile directive. Fixes #431

Build Refactor

This release continues the Convox tradition of offering a simple, reliable, private build service.

We further this goal with a massive simplification of the build code. See PR #507 for code.

Most notably it removes the one build at a time limit. This limit was put in place to avoid transient errors that occur in the Docker registry when pushing two images at once. The occurrence of this error was already greatly reduced with the migration to ECR. The error is eliminated by adding a retry/backoff to the image push step.

The build script is now greatly simplified to match the fact that Convox builds are a very simple pipeline:

• create a new container
• extract source
• setup docker authentication
• docker pull
• docker build
• docker tag
• docker push
• callback to rack API with success or a failure reason

See https://github.com/convox/rack/blob/master/api/cmd/build/main.go for the simple pipeline implementation.

It also refactors the Rack build operations into a clearly defined interface:

• BuildCopy
• BuildCreateIndex
• BuildCreateRepo
• BuildCreateTar
• BuildDelete
• BuildGet
• BuildList
• BuildRelease
• BuildSave

These small units of build functionality are easier to write, easier to test, and easy to compose together for a clearly defined build API:

Description	Method
List builds	GET /apps/{app}/builds
Create new build	POST /apps/{app}/builds
Get build info	GET /apps/{app}/builds/{build}
Update build info	PUT /apps/{app}/builds/{build}
Delete a build	DELETE /apps/{app}/builds/{build}
Copy a build	POST /apps/{app}/builds/{build}/copy
Get build logs	GET /apps/{app}/builds/{build}/logs

Critical Update for Custom Resource Handlers

A change in CloudFormation on AWS side started triggering the update path on some custom resources that don't expect it. This makes the update path safe while we work with AWS to root cause the change in expected behavior.

Papertrail Service Log History

This changes the behavior of convox services add papertrail. It no longer replays some log history into papertrail, fixing confusing log duplication.