Initial commit

LICENSE

@@ -0,0 +1,13 @@
+Copyright 2016 Cluster Labs, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

README.md

@@ -0,0 +1,153 @@
+LaunchKit
+=========
+
+This repo contains an unbranded version of all the code that powers [LaunchKit](https://launchkit.io/). We have packaged things up to run easily with [Vagrant](https://www.vagrantup.com) and [Ansible](https://www.ansible.com), so getting started running LaunchKit's services locally should be fairly straightforward.
+
+* [Getting Started](#getting-started) → HOWTO install & run your own instance of LK.
+* [System Configuration](#system-configuration) → Easily editable settings to enable LK to work with third-party providers.
+* [Architecture Overview](#architecture-overview) → Overview of how LK works.
+* [Code Organization](#code-organization) → Overview of how code is organized in this repository.
+
+If anything in this guide is not accurate or if you run into any issues installing & running LaunchKit, please file a bug.
+
+## Getting Started
+
+Getting your LK instance up and running is fairly simple. This process has been tested thoroughly on **Mac OS 10.11**, but should also work on other systems compatible with Vagrant, VirtualBox and Ansible.
+
+### STEP 1
+
+Install **Xcode dev tools** if you don’t have them yet. Running `cc` from the command line on OS X Terminal should prompt you to install them.
+
+    $ cc
+
+If the command complains about clang input, you're all set.
+
+### STEP 2
+
+Install **Vagrant**. You can find the installer here: https://www.vagrantup.com/downloads.html
+
+Once the installer finishes, you don't need to do anything else.
+
+### STEP 3
+
+Install **VirtualBox** 5.0. You can find 5.0.x here: https://www.virtualbox.org/wiki/Download_Old_Builds_5_0 (The latest 5.1.x versions are not yet compatible for some reason, so use 5.0.x.)
+
+Once the installer finishes, you don't need to open the VirtualBox app. (You can close it if it opens.)
+
+### STEP 4
+
+Install **ansible**:
+
+    # You need pip and a newer version of setuptools to use ansible
+    $ sudo easy_install pip
+    $ pip install --upgrade setuptools --user python
+
+    # Install ansible globally
+    $ sudo pip install ansible
+
+`ansible` should now work as a command:
+
+    $ ansible
+
+(Output should be an error message about missing targets, that's fine.)
+
+### STEP 5
+
+Get the LaunchKit code and configure your LK settings.
+
+    $ git clone [email protected]:clusterinc/launchkit-web.git
+    $ cd launchkit-web
+    $ git checkout oss
+
+Edit your [System Configuration](#system-configuration) according to the various configuration detailed in the next section. (If you do this after the server is started, you will have to reboot.)
+
+### STEP 6
+
+Start LaunchKit (this might take awhile):
+
+    $ vagrant up --provision
+
+After that command finishes, your LK instance should be up and running at `http://localhost:9100/` — woohoo! (The instance might not be ready right away, check 30 seconds after provision is done.)
+
+If you're **all done** using LaunchKit, you can stop the machine by running:
+
+    $ vagrant halt
+
+If you're never going to use LaunchKit again, you can destroy the machine altogether:
+
+    $ vagrant destroy
+
+## System Configuration
+
+LaunchKit will work largely out of the box, but each service has some **external dependencies** that you will need to configure if you wish the service to work properly.
+
+We have moved the most common configuration bits for LaunchKit into `launchkit-web/backend/settings.py` so you can easily find and reconfigure your local instance. After changes are made to this file, you should restart your local instance using `vagrant reload` — changes will not be reflected immediately on a running system.
+
+
+### Global
+
+If you want your LK instance to send emails, you will need to update the following settings:
+
+* **EMAIL_SMTP_HOST** → You can set up an account with a service like Sendgrid, and enter their SMTP endpoint here, eg. `smtp.sendgrid.com`. By default TLS on port 587 is used, this is editable further down in the file.
+* **EMAIL_SMTP_USER** and **EMAIL_SMTP_PASSWORD** → Your SMTP account credentials.
+* **EMAIL_FROM_DOMAIN** → This should be the email address you with to send email _from_, eg. `yourdomain.com`.
+
+
+### Screenshot Builder
+
+Screenshot Builder runs locally and in your browser, but has one key dependency: .zip files are uploaded and hosted from Amazon S3. In order to download Screenshot Builder bundles, you must configure an S3 bucket.
+
+In `launchkit-web/backend/settings.py`, update:
+
+* **BUNDLES_S3_BUCKET_NAME** → Set this to the name of your S3 bucket, eg. `my-screenshot-bundles`
+* **READWRITE_S3_ACCESS_KEY_ID** → Create an IAM role for a user with write access to your S3 bucket, and set the ID here.
+* **READWRITE_S3_SECRET_ACCESS_KEY** → ... and set the SECRET here.
+* **READONLY_S3_ACCESS_KEY_ID** → When serving bundles for download, you may wish to use a different IAM role if this server is public. Set this key to an IAM role with read access to your S3 bucket.
+* **READONLY_S3_SECRET_ACCESS_KEY** → ... and set the SECRET for that user here.
+
+
+### Review Monitor & Sales Monitor
+
+If you would like Slack or Twitter integrations to work in your local instance, you must create a [Slack App](https://api.slack.com/slack-apps) or [Twitter App](https://apps.twitter.com/) and enter the key pairs in `backend/settings.py`. The relevant keys are:
+
+* **SLACK_CLIENT_ID** and **SLACK_CLIENT_SECRET** → Credentials for your Slack App
+* **TWITTER_APP_KEY** and **TWITTER_APP_SECRET** → Credentials for your Twitter App
+
+If you would like Twitter preview images to work properly, you will need to configure a [URL2PNG](https://www.url2png.com/) account and set **URL2PNG_URL_KEY** and **URL2PNG_SECRET_KEY**.
+
+### App Websites
+
+If you want to use App Websites to host an actual website, you will need to expose the hosted frontend webserver externally. This webserver is accessible locally on `http://localhost:9105/` and works by loading the website configured for the current domain ("localhost" in this case) in order to render it.
+
+To test it locally, you can create an App Website and set your domain to "localhost" — then your website should show up on `http://localhost:9105/` just like how you made it.
+
+### Super Users & Cloud Config
+
+These products use our [LaunchKit iOS SDK](https://github.com/launchkit/launchkit-ios) to send events to the backend. In order to use them, our API webserver — located at `http://localhost:9101/` — must be accessible to the network your phone client is on. You can then update the iOS SDK to communicate with your instance of the API webserver, at whichever address you end up hosting it on. If you configure a domain for it, eg. `hosted.yourdomain.com`, you can use that domain as a CNAME endpoint to host many App Websites.
+
+## Architecture Overview
+
+LaunchKit spawns several different *processes* in order to work:
+
+* Skit frontend — `http://localhost:9100/` → Renders all of our frontend HTML, JavaScript and CSS. This server communicates with our backend API over HTTP in order to load content, and does not access the database itself. Daemon: `ansible/roles/lk-skit`, code: `launchkit-web/skit/...`
+* API backend — `http://localhost:9101/` → Authenticates users, loads content, renders JSON over a REST API for all services. Daemon: `ansible/roles/lk-django/files/init.lk-django.conf`, code: `launchkit-web/backend/...`
+* Celery task worker → Executes async tasks, spread throughout python codebase. Sends emails, fetches data from iTunes, creates Screenshot Bundles, etc. `ansible/roles/lk-django/files/init.celery.conf`, code: `launchkit-web/backend/...`
+* Review ingester → Loads reviews from iTunes periodically. `ansible/roles/lk-review-ingester`, code: `launchkit-web/backend/review_ingester.py`
+* Skit hosted frontend — `http://localhost:9105/` → Loads and renders custom App Websites according to the current domain (provided in the Host: HTTP header) and is not used by the other products. Daemon: `ansible/roles/lk-skit`, code: `launchkit-web/skit/lk/public_hosted/...`
+* App Engine images host — `http://localhost:9103/` → Our GAE server handles all image hosting, image uploading and image resizing for LaunchKit products. Daemon: `ansible/roles/lk-google-app-engine`, code: `launchkit-web/gae/...`
+* Dev proxy — `http://localhost:9102/` → A hack to enable App Engine to work with CORS locally. Daemon: `ansible/roles/lk-go-devproxy`, code: `launchkit-web/devproxy.go`
+
+## Code Organization
+
+LaunchKit is largely two codebases: a large python app called `backend`, and a large JavaScript app called `skit`. There are other parts that are less important, but those are the two primary codebases.
+
+* `ansible` → All configuration for this local runtime environment, which spawns all services and loads all dependencies.
+* `backend` → All python code, which is organized in a vaguely Django-like way, for our backend API and async task queues. This Django project has only one "app", called `lk`, which can be found in `backend/lk/`. Settings are in `backend/settings.py`.
+* `backend/lk/views` → Backend API HTTP handlers.
+* `backend/lk/logic` → Backend business logic for creating and managing various LK products.
+* `backend/lk/models` → Backend database models for storing everything in LaunchKit.
+* `gae` → A simple App Engine python application which handles all LaunchKit images. In production, Google App Engine's image service is simply incredible.
+* `skit` → Our frontend web app and hosted App Websites webservers, based on [Skit](https://skitjs.com/).
+* `skit/lk/library` → Library code used throughout the LK frontend & hosted frontend.
+* `skit/lk/public` → Code for the public website, located at https://launchkit.io/
+* `skit/lk/public_hosted` → Code for the hosted App Website endpoint, located at http://domains.launchkit.io/

Vagrantfile

@@ -0,0 +1,40 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+#
+# Copyright 2016 Cluster Labs, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+Vagrant.configure(2) do |config|
+  config.vm.box = "ubuntu/trusty64"
+
+  # LaunchKit services
+  config.vm.network "forwarded_port", guest: 9100, host: 9100 # skit web app
+  config.vm.network "forwarded_port", guest: 9101, host: 9101 # backend api
+  config.vm.network "forwarded_port", guest: 9102, host: 9102 # dev proxy
+  config.vm.network "forwarded_port", guest: 9103, host: 9103 # un-proxied app engine
+  config.vm.network "forwarded_port", guest: 9104, host: 9104 # (intentionally left blank)
+  config.vm.network "forwarded_port", guest: 9105, host: 9105 # hosted websites
+
+  config.vm.network "private_network", ip: "192.168.42.10"
+
+  config.vm.provider "virtualbox" do |vb|
+    vb.memory = "1024"
+  end
+
+  config.vm.provision :ansible do |ansible|
+    ansible.playbook = "ansible/vagrant.yml"
+  end
+end

ansible/roles/common-python/tasks/main.yml

@@ -0,0 +1,7 @@
+---
+- name: install python dev library
+  apt: name=python-dev
+- name: install pip
+  apt: name=python-pip
+- name: install python virtualenv
+  pip: name=virtualenv

ansible/roles/common/files/init.launchkit.conf

@@ -0,0 +1,12 @@
+# Triggers the start of all launchkit tasks
+start on vagrant-mounted MOUNTPOINT=/vagrant
+
+# perform no action, but unless there's something with a PID it behaves weird
+script
+ echo "[`date`] Starting launchkit (runlevel `runlevel`)"
+ sleep 999d
+end script
+
+pre-stop script
+  echo "[`date`] Stopping launchkit (runlevel `runlevel`)"
+end script

ansible/roles/common/files/workaround-vagrant-bug-6074.conf

@@ -0,0 +1,10 @@
+# workaround for https://github.com/mitchellh/vagrant/issues/6074
+start on filesystem
+task
+
+env MOUNTPOINT=/vagrant
+
+script
+  until mountpoint -q $MOUNTPOINT; do sleep 1; done
+  /sbin/initctl emit --no-wait vagrant-mounted MOUNTPOINT=$MOUNTPOINT
+end script

ansible/roles/common/tasks/install-unzip.yml

@@ -0,0 +1,3 @@
+---
+- name: install the unzip tool
+  apt: name=unzip

ansible/roles/common/tasks/main.yml

@@ -0,0 +1,16 @@
+---
+- include: install-unzip.yml
+
+- name: install launchkit dummy service
+  copy:
+    src=../files/init.launchkit.conf
+    dest=/etc/init/launchkit.conf
+    owner=root
+    group=root
+
+- name: install vagrant start bug workaround
+  copy:
+    src=../files/workaround-vagrant-bug-6074.conf
+    dest=/etc/init/workaround-vagrant-bug-6074.conf
+    owner=root
+    group=root

ansible/roles/lk-django/files/init.celery.conf

@@ -0,0 +1,15 @@
+start on starting launchkit
+stop on stopped launchkit
+
+respawn
+respawn limit 10 90
+
+setuid vagrant
+
+script
+  cd /vagrant
+  rm -f celerybeat-schedule # this is created at runtime if file not present
+  celery worker -A backend.celery_app -Q \
+  	celery,email,ingestion,archive,gae,slack,itunes,itunesux,itunesfetch,appstore,sessions \
+  	-lINFO -B -Ofair --concurrency=1
+end script

ansible/roles/lk-django/files/init.lk-django.conf

@@ -0,0 +1,10 @@
+start on (starting launchkit and started postgres)
+stop on stopped launchkit
+
+respawn
+respawn limit 10 90
+
+script
+  cd /vagrant
+  python manage.py runserver 0.0.0.0:9101
+end script

ansible/roles/lk-django/tasks/main.yml

@@ -0,0 +1,25 @@
+---
+- include: python-libraries.yml
+
+- name: upgrade pip
+  pip: name=pip extra_args='--upgrade'
+
+- name: install python requirements
+  pip: requirements=/vagrant/requirements.txt
+
+- include: setup_db.yml
+- include: migrate_db.yml
+
+- name: setup celery upstart script
+  copy:
+    src=../files/init.celery.conf
+    dest=/etc/init/celery.conf
+    owner=root
+    group=root
+
+- name: setup django upstart script
+  copy:
+    src=../files/init.lk-django.conf
+    dest=/etc/init/lk-django.conf
+    owner=root
+    group=root

ansible/roles/lk-django/tasks/migrate_db.yml

@@ -0,0 +1,5 @@
+---
+- name: perform django migration
+  django_manage:
+    command=migrate
+    app_path=/vagrant

ansible/roles/lk-django/tasks/python-libraries.yml

@@ -0,0 +1,9 @@
+---
+- name: install python lxml library
+  apt: name=python-lxml
+- name: install jpeg library for pillow
+  apt: name=libjpeg62
+- name: install jpeg-dev library for pillow
+  apt: name=libjpeg62-dev
+- name: install db adapter
+  apt: name=python-psycopg2

ansible/roles/lk-django/tasks/setup_db.yml

@@ -0,0 +1,17 @@
+---
+- name: create cluster db
+  sudo: yes
+  sudo_user: postgres
+  postgresql_db: name=lk
+- name: add hstore db extension
+  sudo: yes
+  sudo_user: postgres
+  postgresql_ext: name=hstore db=lk
+- name: add {{ db_user }} user to cluster db as superuser
+  sudo: yes
+  sudo_user: postgres
+  postgresql_user: name={{ db_user }} role_attr_flags=SUPERUSER db=lk
+- name: add {{ db_user }} user to postgres as superuser
+  sudo: yes
+  sudo_user: postgres
+  postgresql_user: name={{ db_user }} role_attr_flags=SUPERUSER

ansible/roles/lk-go-devproxy/files/init.go-devproxy.conf

@@ -0,0 +1,10 @@
+start on starting launchkit
+stop on stopped launchkit
+
+respawn
+respawn limit 10 90
+
+script
+  cd /vagrant
+  go run devproxy.go 0.0.0.0:9102 localhost:9103
+end script

ansible/roles/lk-go-devproxy/tasks/main.yml

@@ -0,0 +1,9 @@
+---
+- name: install go
+  apt: name=golang
+- name: install go devproxy as service
+  copy:
+    src=../files/init.go-devproxy.conf
+    dest=/etc/init/go-devproxy.conf
+    owner=root
+    group=root

ansible/roles/lk-google-app-engine/files/google_app_engine.sh

@@ -0,0 +1,2 @@
+# Insert gae path to find the gae files
+export PATH="$PATH:/usr/local/google_appengine"

ansible/roles/lk-google-app-engine/files/home.appcfg_nag

@@ -0,0 +1,2 @@
+opt_in: false
+timestamp: 1446322504.937726