From 47eaeab394f1279002b7a3a4cd68e7c1fa63411e Mon Sep 17 00:00:00 2001 From: Will Kahn-Greene Date: Thu, 17 Aug 2017 11:39:56 -0400 Subject: [PATCH] Update and clean up some getting started docs (#3916) * Update and clean up some getting started docs I made some changes based on Jonathan's suggestions. The big issue was that the documentation about running "make dockerupdatedata" was after the link sending you off to the webapp documentation, so he missed it. This fixes that flow and adds some more details which are probably helpful to know. * Improve Quickstart by adding more context explanation * Clean up tools install section of Getting Started * Add warnings about outdatedness --- docs/components/crashstorage.rst | 4 + docs/components/crontabber.rst | 4 + docs/components/processor.rst | 5 ++ docs/components/webapp.rst | 6 +- docs/gettingstarted.rst | 124 ++++++++++++++++++++++++------- 5 files changed, 115 insertions(+), 28 deletions(-) diff --git a/docs/components/crashstorage.rst b/docs/components/crashstorage.rst index dc999c72db..c11857f6ab 100644 --- a/docs/components/crashstorage.rst +++ b/docs/components/crashstorage.rst @@ -11,6 +11,10 @@ Base class implemented in ``socorro/external/crashstorage_base.py`` These are our base classes for all crash storage for Socorro. +.. Warning:: + + August 17th, 2017: These docs are outdated. + socorro.external.crashstorage_base ================================== diff --git a/docs/components/crontabber.rst b/docs/components/crontabber.rst index 880fa51ce7..e2baa6332e 100644 --- a/docs/components/crontabber.rst +++ b/docs/components/crontabber.rst @@ -22,6 +22,10 @@ The configuration is done using ``configman`` and it looks something like this:: The default config lives in ``socorro/cron/crontabber_app.py`` as ``DEFAULT_JOBS``. +.. Warning:: + + August 17th, 2017: Everything below this point is outdated. + crontab runs crontabber ======================= diff --git a/docs/components/processor.rst b/docs/components/processor.rst index a2eeb37b83..b867c0848a 100644 --- a/docs/components/processor.rst +++ b/docs/components/processor.rst @@ -34,6 +34,11 @@ Then you can start and stop the processor and tweak files and all that jazz. The stackwalk binaries are in ``/stackwalk`` in the container. +.. Warning:: + + August 17th, 2017: Everything below this point is outdated. + + Troubleshooting =============== diff --git a/docs/components/webapp.rst b/docs/components/webapp.rst index 69ff4845a7..14e2c4b855 100644 --- a/docs/components/webapp.rst +++ b/docs/components/webapp.rst @@ -40,8 +40,6 @@ Then you can start and stop the webapp, adjust files, and debug. https://docs.docker.com/compose/extends/ -FIXME(willkg): Review everything after this line. - Setting up authentication and a superuser ========================================= @@ -88,6 +86,10 @@ need to get a set of oauth credentials. After that, Google Sign-In should work. +.. Warning:: + + August 17th, 2017: Everything below this point is outdated. + About Permissions, User and Groups ================================== diff --git a/docs/gettingstarted.rst b/docs/gettingstarted.rst index 32ddef53a7..1c77d93f0a 100644 --- a/docs/gettingstarted.rst +++ b/docs/gettingstarted.rst @@ -12,83 +12,155 @@ If you're interested in running Socorro in a server environment, then check out Quickstart ========== -1. Install `Docker `_. +1. Install required software: Docker, docker-compose (1.10+), make, and git. -2. Install `docker-compose `_. You need - 1.10 or higher. + **Linux**: - .. Note:: + Use your package manager. - It helps to add an alias to your shell:: + **OSX**: - alias dc=docker-compose + Install `Docker for Mac `_ which + will install Docker and docker-compose. - because it's way easier to type "dc" and I do it a lot. + Use `homebrew `_ to install make and git:: -3. Install `make `_ using either your - system's package manager (Linux) or homebrew (OSX). + $ brew install make git - FIXME(willkg): Windows instructions? + **Other**: -4. From the root of this repository, run:: + Install `Docker `_. + + Install `docker-compose `_. You need + 1.10 or higher. + + Install `make `_. + + Install `git `_. + +2. Clone the repository so you have a copy on your host machine. Instructions + are `on GitHub `_. + +3. From the root of this repository, run:: $ make dockerbuild - That will build the containers required for development: test, processor, and - webapp. + That will build the Docker images required for development: processor, + webapp, and crontabber. + + Each of these images covers a single Socorro component: processor, webapp, + and crontabber. -5. Then you need to set up Postgres and Elasticssearch. To do that, run:: +4. Then you need to set up the Postgres database and Elasticssearch. To do that, + run:: $ make dockersetup + This creates the Postgres database and sets up tables, stored procedures, + integrity rules, types, and a bunch of other things. It also adds a bunch of + static data to lookup tables. + + For Elasticsearch, it sets up Supersearch fields and the index for raw and + processed crash data. + + You can run ``make dockersetup`` any time you want to wipe the Postgres + database and Elasticsearch to pick up changes to those storage systems or + reset your environment. - You can run ``make dockersetup`` any time you want to wipe the database, pick - up changes in static data, stored procedures, types, migrations, etc. +5. Then you need to pull in product release and some other data that makes + Socorro go. - .. Warning:: + To do that, run:: - This is a work in progress and might be fussy about some things. + $ make dockerupdatedata - Pull requests welcome! + This adds data that changes day-to-day. Things like product builds and + normalization data. + + Depending on what you're working on, you might want to run this weekly or + maybe even daily. At this point, you should have a basic functional Socorro development environment. +See :ref:`gettingstarted-chapter-updating` for how to maintain and update your environment. + See :ref:`webapp-chapter` for additional setup and running the webapp. See :ref:`processor-chapter` for additional setup and running the processor. +See :ref:`crontabber-chapter` for additional setup and running crontabber. + + +.. _gettingstarted-chapter-updating: Updating data in a dev environment ================================== -Wiping the db -------------- +Updating the code +----------------- + +Any time you want to update the code in the repostory, run something like this from +the master branch:: + + $ git pull -Any time you want to wipe the database and start over, run ``make dockersetup``. + +It depends on what you're working on and the state of things. + +After you do that, you'll need to update other things. + +If there were changes to the requirements files or setup scripts, you'll need to +build new images:: + + $ make dockerbuild + + +If there were changes to the database tables, stored procedures, types, +migrations, or anything like that, you'll need to wipe the Postgres database and +Elasticsearch:: + + $ make dockersetup + + +After doing that, you'll definitely want to update data:: + + $ make dockerupdatedata + + +Wiping the database +------------------- + +Any time you want to wipe the database and start over, run:: + + $ make dockersetup Updating release data --------------------- -Data on releases comes from running ftpscraper. After you run ftpscraper, you +Release data and comes from running ftpscraper. After you run ftpscraper, you have to run featured-versions-automatic which will update the featured versions -list. +list. Additionally, there's other data that changes day-to-day that you need to +pick up in order for some views in the webapp to work well. -We put all that in a single shell script. +Updating that data is done with a single make rule. Run:: $ make dockerupdatedata -.. Warning:: +.. Note:: This can take a long while to run and if you're running it against an existing database, then ftpscraper will "catch up" since the last time it ran which can take a long time, maybe hours. + If you don't have anything in the database that you need, then it might be + better to wipe the database and start over. + General architecture ====================