update configuration.md (#1663)

* update configuration.md

* change STATIC_FILES_CLIENT_PORT to
 STATIC_FILES_PARTICIPATION_PORT;
typo and organization of configuration.md.

* Creating an upgrading.md document with config changes

* update upgrading.md

* use BOLD MONO style for env vars

* CACHE_MATH_RESULTS default is now true

* PR feedback

* Add STATIC_FILES_PORT as backup for specific port settings

---------

Co-authored-by: Christopher Small <[email protected]>

client-participation/README.md

@@ -40,7 +40,7 @@ assets in another way.
 ### embed.js
 
 Among the assets built into the `dist/` directory is `embed.js` which is used when deploying a polis client
-embedded into another website. Set the **EMBED_SERVICE_HOSTNAME** environment variable to your API Service hostname
+embedded into another website. Set the **`EMBED_SERVICE_HOSTNAME`** environment variable to your API Service hostname
 when you build this app. In the top-level `docker compose` configuration, this variable is read from the `.env` file
 there. e.g. `EMBED_SERVICE_HOSTNAME=api.mypolis.org`.
 
@@ -49,7 +49,7 @@ there. e.g. `EMBED_SERVICE_HOSTNAME=api.mypolis.org`.
 Optionally, you can [register with Facebook](https://developers.facebook.com/docs/development) and get a Facebook App ID
 to use the Facebook auth features.
 
-If you do so, set the **FB_APP_ID** environment variable in the top level `.env` file, or manually pass it in
+If you do so, set the **`FB_APP_ID`** environment variable in the top level `.env` file, or manually pass it in
 when building and running this application.
 
 ## Building the Application

client-report/README.md

@@ -38,7 +38,7 @@ get very far. Still it can be useful for developing and debugging builds.
 
 The folowing environment variable can be set when building and running this application. If using the top-level `docker compose` configuration, it can be found in the `.env` file there.
 
-**SERVICE_URL**: (Optional) The URL of your API Server. Set this value if your API Server is not on the same domain as your client-report.
+**`SERVICE_URL`**: (Optional) The URL of your API Server. Set this value if your API Server is not on the same domain as your client-report.
 Currently this is only used in development.
 
 You will also need to have AWS credentials set up at `.polis_s3_creds_client.json` if you are using S3

docs/configuration.md

@@ -28,6 +28,12 @@ the individual READMEs for more detailed descriptions of how to configure these
 
 ## Environment variables and .env
 
+> **Quickstart**
+>
+> In development, the default values of example.env should work as-is.
+>
+> `cp example.env .env`
+
 By default, `docker compose` will look for and use an `.env` file if one exists. However, any value present in your
 environment or passed in on the command line will overwrite those in the file. Thus you should be able to set your
 configuration values in whatever way suits your given scenario. (A plain text `.env` file is not always appropriate in
@@ -37,12 +43,96 @@ If you are running these applications without Docker, just make sure that any en
 the environment where the application is running.
 
 If you are doing development on a url other than `localhost` or `localhost:5000`, you need to update the
-API_DEV_HOSTNAME value to your development hostname:port, e.g. `myhost:8000` or `api.testserver.net`.
-DEV_MODE should be `true`.
+**`API_DEV_HOSTNAME`** value to your development hostname:port, e.g. `myhost:8000` or `api.testserver.net`.
+**`DEV_MODE`** should be `true`.
+
+If you are deploying to a custom domain (not pol.is) then you need to update both the **`API_PROD_HOSTNAME`** and
+**`DOMAIN_OVERRIDE`** values to your custom hostname (omitting http(s):// protocol).
+**`DEV_MODE`** should be `false`.
+
+### General Settings
+
+- **`ADMIN_UIDS`** an array of user UUIDs for site admins. These users will have moderator capabilities on all conversations hosted on the site.
+- **`EMAIL_TRANSPORT_TYPES`** comma-separated list of email services to use (see [Email Transports](#email-transports) below)
+- **`GIT_HASH`** Set programatically using `git rev-parse HEAD` (e.g. in Makefile) to tag docker container versions and other release assets. Can be left blank.
+- **`MATH_ENV`** Set to prod (default), preprod, or dev. In cases where a single database is used for multiple environments, this value is used by the API service to request the correct data. (Using a single DB for multiple environments is no longer recommended.)
+- **`SERVER_ENV_FILE`** The name of an environment file to be passed into the API Server container by docker compose. Defaults to ".env" if left blank. Used especially for building a "test" version of the project for end-to-end testing.
+- **`SERVER_LOG_LEVEL`** Used by Winston.js in the API server to determine how much logging to output. Reasonable values are "debug", "info", and "error". Defaults to "info".
+
+### Database
+
+- **`READ_ONLY_DATABASE_URL`** (optional) Database replica for reads.
+- **`POSTGRES_DB`** database name (e.g. "polis-dev")
+- **`POSTGRES_HOST`** database host (e.g. postgres:5432 if using docker compose, localhost:5432 if using local db)
+- **`POSTGRES_PASSWORD`** database password
+- **`POSTGRES_PORT`** typically 5432
+- **`POSTGRES_USER`** typically "postgres". Any username will be used by the docker container to create a db user.
+- **`DATABASE_URL`** should be the combination of above values, `postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}/${POSTGRES_DB}`
+
+### Docker Concerns
+
+- **`TAG`** used by **`COMPOSE_PROJECT_NAME`** below. Defaults to "dev".
+- **`COMPOSE_PROJECT_NAME`** Used by docker compose to label containers and volumes. Useful in development if you are (re)-building and deleting groups of docker assets.
+
+### Ports
+
+- **`API_SERVER_PORT`** typically 5000. Used internally within a docker network and/or behind a proxy. A `PORT` value is used as a fallback if `API_SERVER_PORT` is not set (for Heroku comptability).
+- **`HTTP_PORT`** typically 80. Port exposed by Nginx reverse proxy.
+- **`HTTPS_PORT`** typically 443. Port exposed by Nginx reverse proxy.
+- **`STATIC_FILES_PORT`** typically 8080. Used internally within a docker network and/or behind a proxy.
+- **`STATIC_FILES_ADMIN_PORT`** same as **`STATIC_FILES_PORT`** unless you are hosting client-admin separately from file-server. Useful in local development.
+- **`STATIC_FILES_PARTICIPATION_PORT`** same as **`STATIC_FILES_PORT`** unless you are hosting client-participation separately from file-server. Useful in local development.
+
+### Email Addresses
+
+- **`ADMIN_EMAIL_DATA_EXPORT`** email address from which data export emails are sent.
+- **`ADMIN_EMAIL_DATA_EXPORT_TEST`** email address to receive periodic export test results, if configured below.
+- **`ADMIN_EMAIL_EMAIL_TEST`** email address to receive backup email system test.
+- **`ADMIN_EMAILS`** array of email addresses to receive team notifications.
+- **`POLIS_FROM_ADDRESS`** email address from which other emails are sent.
+
+### Boolean Flags
+
+(All can be left blank, or `false`)
+
+- **`BACKFILL_COMMENT_LANG_DETECTION`** Set to `true`, if Comment Translation was enabled, to instruct the server upon the next initialization (reboot) to backfill detected language of stored comments. Default `false`.
+- **`CACHE_MATH_RESULTS`** Set this to `true` to instruct the API server to use LRU caching for results from the math service. Default is `true` if left blank.
+- **`DEV_MODE`** Set this to `true` in development and `false` otherwise. Used by API Server to make a variety of assumptions about HTTPS, logging, notifications, etc.
+- **`RUN_PERIODIC_EXPORT_TESTS`** Set this to `true` to run periodic export tests, sent to the **`ADMIN_EMAIL_DATA_EXPORT_TEST`** address.
+- **`SERVER_LOG_TO_FILE`** Set this to `true` to tell Winston.js to also write log files to server/logs/. Defaults to `false`. *Note that if using docker compose, server/logs is mounted as a persistent volume.*
+- **`SHOULD_USE_TRANSLATION_API`** Set this to `true` if using Google translation service. See [Enabling Comment Translation](#enabling-comment-translation) below.
+
+### URL/Hostname Settings
+
+- **`API_DEV_HOSTNAME`** typically "localhost" unless you are running a development instance elsewhere.
+- **`API_PROD_HOSTNAME`** the hostname of your site (e.g. pol.is, or example.com). Should match **`DOMAIN_OVERRIDE`**. (In the future these two options may be combined into one.)
+- **`DOMAIN_OVERRIDE`** the hostname of your site. Should match **`API_PROD_HOSTNAME`**.
+- **`DOMAIN_WHITELIST_ITEM_01`** - **`08`** up to 8 possible additional whitelisted domains for client applications to make API requests from. Typical setups that use the same URL for the API service as for the public-facing web sites do not need to configure these.
+- **`EMBED_SERVICE_HOSTNAME`** should match **`API_DEV_HOSTNAME`** in production, or **`API_DEV_HOSTNAME`** in development. Embedded conversations make API requests to this host.
+- **`SERVICE_URL`** used by client-report to make API calls. Only necessary if client-report is hosted separately from the API service. Can be left blank.
+- **`STATIC_FILES_HOST`** Used by the API service to fetch static assets (the compiled client applications) from a static file server. Within a docker compose network this is "file-server", but could be an external hostname, such as a CDN.
+
+### Third Party API Credentials
+
+(All are optional, and omitting them will disable the related feature.)
+
+- **`AKISMET_ANTISPAM_API_KEY`** Comment spam detection and filtering.
+- **`AWS_REGION`** Used for S3 data import/export.
+- **`ENABLE_TWITTER_WIDGETS`** set to `true` to enable twitter widgets on the client-admin authentication pages.
+- **`FB_APP_ID`** Must register with Facebook to get an ID to enable Facebook App connectivity.
+- **`GA_TRACKING_ID`** For using Google Analytics on client pages.
+- **`GOOGLE_CREDENTIALS_BASE64`** Required if using Google Translate API. (See below).
+- **`GOOGLE_CREDS_STRINGIFIED`** Alternative to **`GOOGLE_CREDENTIALS_BASE64`** (See below).
+- **`MAILGUN_API_KEY`**, **`MAILGUN_DOMAIN`** If using Mailgun as an email transport.
+- **`MAXMIND_LICENSEKEY`**, **`MAXMIND_USERID`** If using IP Geolocation service Maxmind.
+- **`TWITTER_CONSUMER_KEY`**, **`TWITTER_CONSUMER_SECRET`** For Twitter integration.
+- **`AWS_ACCESS_KEY_ID`**, **`AWS_SECRET_ACCESS_KEY`** If using Amazon SES as an email transport.
+
+### Deprecated
 
-If you are deploying to a custom domain (not pol.is) than you need to update both the API_PROD_HOSTNAME and
-DOMAIN_OVERRIDE values to your custom hostname (omitting http(s):// protocol).
-DEV_MODE should be `false`.
+- **`ENCRYPTION_PASSWORD_00001`** (deprecated) a password used to encrypt and decrypt participants' IP addresses. Can be left blank.
+- **`WEBSERVER_PASS`** (deprecated) basic auth setting for certain requests sent between math and api services.
+- **`WEBSERVER_USERNAME`** (deprecated) basic auth setting for certain requests sent between math and api services.
 
 ## Enabling Comment Translation
 
@@ -64,7 +154,7 @@ browser's language.
       (inspect the simple JS code), or
     - using your workstation terminal: `cat path/to/My-Project-abcdef0123456789.json | base64` (linux/mac)
 
-4. Set `GOOGLE_CREDENTIALS_BASE64` in `.env`
+4. Set **`GOOGLE_CREDENTIALS_BASE64`** in `.env`
 
 5. Set `SHOULD_USE_TRANSLATION_API=true` in `.env`
 
@@ -77,15 +167,15 @@ translate strings can be found in: `client-participation/js/strings/en_us.js`
 ## Email Transports
 
 We use [Nodemailer] to send email. Nodemailer uses various built-in and
-packaged _email transports_ to send email via SMTP or API, either directly or
+packaged *email transports* to send email via SMTP or API, either directly or
 via third-party platforms.
 
 Each transport needs a bit of hardcoded scaffold configuration to make it work,
 which we welcome via code contribution. But after this, others can easily use
 the same email transport by setting some configuration values via environment
 variable or otherwise.
 
-We use `EMAIL_TRANSPORT_TYPES` to set email transports and their fallback
+We use **`EMAIL_TRANSPORT_TYPES`** to set email transports and their fallback
 order. Each transport has a keyword (e.g., `maildev`). You may set one or more
 transports, separated by commas. If you set more than one, then each transport
 will "fallback" to the next on failure.
@@ -100,7 +190,7 @@ the email will not be sent.
 
 Note: The [MailDev][] email transport is for **development purposes only**. Ensure it's disabled in production!
 
-1. Add `maildev` into the `EMAIL_TRANSPORT_TYPES` configuration.
+1. Add `maildev` into the **`EMAIL_TRANSPORT_TYPES`** configuration.
 
 This transport will work automatically when running via Docker Compose with the development overlay, accessible on port
 1080.
@@ -109,13 +199,13 @@ This transport will work automatically when running via Docker Compose with the
 
 ### Configuring transport: `aws-ses`
 
-1. Add `aws-ses` into the `EMAIL_TRANSPORT_TYPES` configuration.
-2. Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` configuration.
+1. Add `aws-ses` into the **`EMAIL_TRANSPORT_TYPES`** configuration.
+2. Set the **`AWS_ACCESS_KEY_ID`** and **`AWS_SECRET_ACCESS_KEY`** configuration.
 
 ### Configuring transport: `mailgun`
 
-1. Add `mailgun` into the `EMAIL_TRANSPORT_TYPES` configuration.
-2. Set the `MAILGUN_API_KEY` and `MAILGUN_DOMAIN` configuration.
+1. Add `mailgun` into the **`EMAIL_TRANSPORT_TYPES`** configuration.
+2. Set the **`MAILGUN_API_KEY`** and **`MAILGUN_DOMAIN`** configuration.
 
 ### Adding a new transport
 

docs/upgrading.md

@@ -0,0 +1,30 @@
+# Upgade Guide
+
+## Configuration Changes (Q1 2023)
+
+`polis.config.template.js` and `polis.config.js` files are removed and no longer used.
+`docker-dev.env` and `docker-db-dev.env` files are removed and no longer used.
+`.env` and/or `prod.env` are now treated as the source of truth for the application and are ignored by git.
+See `example.env` for default values.
+
+Please read [configuration.md](./configuration.md) for more information and a complete list of configuration values.
+
+Values that have been renamed or replaced:
+
+- **`DATABASE_FOR_READS_NAME`** has been replaced by **`READ_ONLY_DATABASE_URL`**
+- **`PORT`** has been renamed **`API_SERVER_PORT`**
+- **`SERVICE_HOSTNAME`** has been renamed to **`EMBED_SERVICE_HOSTNAME`**
+- **`STATIC_FILES_ADMINDASH_PORT`** has been renamed to **`STATIC_FILES_ADMIN_PORT`**
+
+New values:
+
+- **`API_DEV_HOSTNAME`**
+- **`API_PROD_HOSTNAME`**
+- **`ENABLE_TWITTER_WIDGETS`**
+- **`GA_TRACKING_ID`**
+- **`POSTGRES_HOST`**
+- **`POSTGRES_PORT`**
+- **`SERVER_ENV_FILE`**
+- **`SERVER_LOG_LEVEL`**
+- **`SERVER_LOG_TO_FILE`**
+- **`STATIC_FILES_PARTICIPATION_PORT`**

example.env

@@ -7,25 +7,14 @@ ADMIN_UIDS=[]
 # Options: maildev, aws-ses, mailgun
 # Example: `aws-ses,mailgun` would try sending via AWS SES first, and fallback to Mailgun on error.
 EMAIL_TRANSPORT_TYPES=maildev
-# (Deprecated) Used internally by Node.Crypto.
-ENCRYPTION_PASSWORD_00001=
 # Optionally set this manually or use `git rev-parse HEAD`. It can be useful for debugging.
 GIT_HASH=
 # Options: prod, preprod, dev:
 MATH_ENV=prod
-# Optionally set this to whatever you want. It is a convenience for docker assets.
-TAG=dev
-COMPOSE_PROJECT_NAME=polis-${TAG}
 # Optionally give the server container a distinct env_file. Useful for CI tests.
 SERVER_ENV_FILE=.env
 # Used by winston via server/utils/logger. Defaults to "info".
 SERVER_LOG_LEVEL=
-# When true, logs are written to server/logs in addition to stdout.
-# If docker compose is used, the logs directory is mounted as a persistent volume.
-SERVER_LOG_TO_FILE=
-# (Deprecated) Settings for submitting web requests to the math worker.
-WEBSERVER_PASS=ws-pass
-WEBSERVER_USERNAME=ws-user
 
 
 ###### DATABASE ######
@@ -39,6 +28,12 @@ POSTGRES_USER=postgres
 DATABASE_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}/${POSTGRES_DB}
 
 
+###### DOCKER CONCERNS ######
+# Optionally set this to whatever you want. It is a convenience for docker assets.
+TAG=dev
+COMPOSE_PROJECT_NAME=polis-${TAG}
+
+
 ###### PORTS ######
 API_SERVER_PORT=5000
 HTTP_PORT=80
@@ -47,7 +42,7 @@ STATIC_FILES_PORT=8080
 # These ports could actually be different in development or if you are hosting clint-admin,
 # client-participation, and file-server on different servers or containers.
 STATIC_FILES_ADMIN_PORT=${STATIC_FILES_PORT}
-STATIC_FILES_CLIENT_PORT=${STATIC_FILES_PORT}
+STATIC_FILES_PARTICIPATION_PORT=${STATIC_FILES_PORT}
 
 
 ###### EMAIL ADDRESSES ######
@@ -59,12 +54,16 @@ POLIS_FROM_ADDRESS="Example <[email protected]>"
 
 
 ###### BOOLEAN FLAGS ######
-# (false by default):
-BACKFILL_COMMENT_LANG_DETECTION=
+# Instructs the API server to cache the results from the math service. Default is true if not set.
 CACHE_MATH_RESULTS=
+# The following flags will all default to false if not set.
+BACKFILL_COMMENT_LANG_DETECTION=
 # Set to `false` for production:
 DEV_MODE=true
 RUN_PERIODIC_EXPORT_TESTS=
+# When true, logs are written to server/logs in addition to stdout.
+# If docker compose is used, the logs directory is mounted as a persistent volume.
+SERVER_LOG_TO_FILE=
 SHOULD_USE_TRANSLATION_API=
 
 
@@ -118,3 +117,11 @@ AWS_ACCESS_KEY_ID=
 AWS_SECRET_ACCESS_KEY=
 # This value is written by the server app if SHOULD_USE_TRANSLATION_API is true.
 GOOGLE_APPLICATION_CREDENTIALS=
+
+
+###### DEPRECATED ######
+# (Deprecated) Used internally by Node.Crypto to encrypt/decrypt IP addresses.
+ENCRYPTION_PASSWORD_00001=
+# (Deprecated) Basic Auth settings for certain requests between math and api services.
+WEBSERVER_PASS=ws-pass
+WEBSERVER_USERNAME=ws-user

math/doc/configuration.md

@@ -10,18 +10,18 @@ Please see [`src/polismath/components/config.clj`](https://github.com/pol-is/pol
 
 The ones you're most frequently to need to tweak for one reason or another:
 
-* `MATH_ENV`: This defaults to `dev`, for local development environments.
+* **`MATH_ENV`**: This defaults to `dev`, for local development environments.
   Traditionally we've set this to `prod` and `preprod` for our production and pre-production deployments specifically.
   This value is used in keying the math export json blobs as found in the `math_main` and other tables in the database.
   This makes it possible to run multiple math environments (dev, testing, prod, preprod) all on the same database of votes.
   This setting is something of a relic from an old architecture where prod and preprod environments ran off of the same database, and with the docker infrastructure is generally no longer needed.
-  Nevertheless, when you start the math server, you will need to run it with the same `MATH_ENV` setting as you ran the math worker with.
-* `POLL_FROM_DAYS_AGO`: This defaults to 10 (at the time of this writing).
+  Nevertheless, when you start the math server, you will need to run it with the same **`MATH_ENV`** setting as you ran the math worker with.
+* **`POLL_FROM_DAYS_AGO`**: This defaults to 10 (at the time of this writing).
   Conversations which have had vote or moderation activity in the specified range will be loaded into memory, and will be updated.
   This prevents old inactive conversations from being loaded into memory every time the poller starts.
 
 You'll also need to pass database credentials. If using docker compose, this will be inherited from the `.env` file or process environment in which docker is being run.
 
-  `DATABASE_URL`: url for the database:
+  **`DATABASE_URL`**: url for the database:
 
   `postgres://<username>:<password>@<url>:<port>/<database-id>`