Standardize on Docker as the primary deployment option

[jeffwidman]

@arikfr mentioned that he'd like to standardize on Docker as the primary deployment option. Supporting multiple options requires either spending already-stretched maintainer time keeping them updated, or letting them get outdated and having users complain. Similarly, having a single primarily deployment path means we need less documentation.

As part of this, what do you think about removing the following?

• Heroku procfiles.
• Vagrant box - I'm unsure whether this is still helpful for dev or not. It's quick to rebuild the docker container, but it might be a pain to debug what's happening inside the docker container, I'm not sure.
• Google Cloud Images
• AWS image
• Fabric script - this seems to do a fair bit, when all it should be handling is DB migrations. Could switch to using flask-alembic to handle migrations.

While removing options seems like it'd make it more difficult for users to get started, it'd actually make it less confusing because there'd be a single "officially-blessed" path for getting up and running.

[arikfr]

Just to clarify what I meant by standardizing on Docker for deployment: we will still have AWS/Google Cloud images, but they will use the Docker images, which will simplify their maintenance.

The cloud images had been a great help with adoption, and having just Docker images won't serve the same audience. The Vagrant box is still used for development environments, and probably will only be replaced by Docker when Docker is more adopted (and has better support for OS/X).

The Fabric script is used to upgrade the cloud images.

I'm cool with dropping Procfile and Procfile.heroku as Heroku is not supported anymore (although I'm thinking of supporting it again).

So the path forward should be:

1. Finish the work on the [Ubuntu/docker deployment[(https://github.com/getredash/redash/tree/master/setup/ubuntu_docker).
2. Have clear instructions and script for upgrading a Docker based setup. It's going to be similar to our current Fabric file, but simpler: stop containers -> grab latest -> run migrations (if needed) -> start containers.
3. Switch to creating cloud images with the Ubuntu/docker deployment script.

The upgrade process is the main reason I didn't move forward with the Docker deployment yet. I didn't have the time to invest in researching a good way to run the upgrade process with Docker images.

[fredfalcon]

Would love to see Heroku support keeping with the theme. "Run where you run - Easily launch Re:dash on-premise in your environment." which is Heroku

[jeffwidman]

Heroku supports docker container deployments, although you have to use their base image for the Dockerfile. Not sure if @arikfr has the time to support Heroku, but if so he may use those instead of the procfiles used previously.

In my opinion, there's currently too many deployment options in the core repo leading to a combination of user confusion and outdated deployment scripts. I'd rather see Heroku, along with the AWS and GCE images, moved out of core and into a sister repo(s) that are dedicated to alternative deployment options.

[adamlwgriffiths]

Anyone requiring Heroku support can use my branch until a Heroku Docker deployment process is made.
https://github.com/someones/redash/tree/heroku
All I really did was update requirements.txt, drop the npm scripts to the root level and update the paths and add bower to package.json.

[arikfr]

@adamlwgriffiths it's a good start and I'm open to merging this in, if you willing to do the extra work required --

1. Instead of adding requirements_all_ds.txt into requirements.txt, use a post-compile step (basically adding a one line script in ./bin).
2. Update documentation and Makefile where we reference the Node.js stuff, that it no longer requires changing directory into rd_ui.

Although the biggest challenge is offering people simple upgrade path. It's a challenge for Docker deployments as well, and requires us adding real migrations mechanism. There is related discussion in #876.

[adamlwgriffiths]

I'm not sure if post-compile will work. Heroku looks for cffi in requirements.txt. If found, it will include the cffi library in the build pack.
That said, I haven't had to delve too deeply into Heroku deployments yet.
When I get time, I'll try a new deploy with post-compile and see if it does.

I am confused however. I thought Heroku support was specifically removed (#903)?

[arikfr]

I'm not sure if post-compile will work. Heroku looks for cffi in requirements.txt. If found, it will include the cffi library in the build pack.

We can either reproduce what the official build pack does in the post compile step or just include cffi in requirements.txt. While I would prefer just to include it (to keep things simple), we need to verify it won't have unwanted complications.

I am confused however. I thought Heroku support was specifically removed (#903)?

What was removed are leftovers of previous support for Heroku. As you found out yourself supporting Heroku is more than just having the Procfile. Because the other "components" were missing there was no point in keeping the Procfiles.

When I get time, I'll try a new deploy with post-compile and see if it does.

Basically you need to add the following file in bin/post_compile:

#!/bin/bash

pip install requirements_all_ds.txt

[jeffwidman]

FYI: Rackspace is also demoing a very nifty docker-based PAAS called Carina here at Pycon, and playing with it reminded me a lot of the simplicity of heroku... they've done a good job with it, but using dockerfiles etc that we already have for redash and not having to fiddle with maintaing as much provider-specific config like heroku's current process. Still in beta, but something worth watching

[arikfr]

I've been experimenting with using Docker Compose for running Redash as an alternative for our complicated images setup and it seems to work quite nicely.

If time allows, v4's images will use Docker.