chore(repo-structure): Docker images and stacks organized and clarified

.env.example

@@ -74,9 +74,9 @@ POSTGRES_USE_PGVECTOR=true
 # POSTGRES_SSL_DIR='/var/lib/postgresql'
 REDIS_PASSWORD=''
 REDIS_URL='redis://localhost:6379'
-# REDIS_TLS_CERT_FILE=./certs/redis.crt
-# REDIS_TLS_KEY_FILE=./certs/redis.key
-# REDIS_TLS_CA_FILE=./certs/redisCA.crt
+# REDIS_TLS_CERT_FILE=./run-it/development/redis/certs/redis.crt
+# REDIS_TLS_KEY_FILE=./run-it/development/redis/certs/redis.key
+# REDIS_TLS_CA_FILE=./run-it/development/redis/certs/redisCA.crt
 # REDIS_TLS_REJECT_UNAUTHORIZED='false'
 RETHINKDB_URL='rethinkdb://localhost:28015/actionDevelopment'
 RETHINKDB_SSL='false'
@@ -119,9 +119,9 @@ RETHINKDB_SSL='false'
 # RECALL_AI_KEY=''
 # SLACK_CLIENT_ID='key_SLACK_CLIENT_ID'
 # SLACK_CLIENT_SECRET='key_SLACK_CLIENT_SECRET'
-# STRIPE_SECRET_KEY='sk_test_4eC39HqLyjWDarjtT1zdp7dc'
-# STRIPE_PUBLISHABLE_KEY='pk_test_TYooMQauvdEDq54NiTphI7jx'
-# STRIPE_WEBHOOK_SECRET='sk_test_4eC39HqLyjWDarjtT1zdp7dc'
+# STRIPE_SECRET_KEY=''
+# STRIPE_PUBLISHABLE_KEY=''
+# STRIPE_WEBHOOK_SECRET=''
 
 # MAIL
 # MAIL GLOBALS. PROVIDER: mailgun | google | debug | smtp

.github/workflows/README.md

@@ -2,7 +2,7 @@
 
 ## Runners
 
-To run `docker-build.yml`, GitHub requires using a larger runner.
+To run `build.yml`, GitHub requires using a larger runner.
 This is because we're webpackifying all the code in node_modules into a single `.js.`.
 Doing all that transpiling can take a LOT of memory. 8GB+.
 At this time, large GitHub-hosted runners are in beta.

.github/workflows/build.yml

@@ -12,8 +12,8 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
   cancel-in-progress: true
 env:
-  PARABOL_DOCKERFILE: ./docker/parabol-ubi/docker-build/dockerfiles/basic.dockerfile
-  PARABOL_BUILD_ENV_PATH: docker/parabol-ubi/docker-build/environments/pipeline
+  PARABOL_DOCKERFILE: ./docker/parabol-ubi/dockerfiles/basic.dockerfile
+  PARABOL_BUILD_ENV_PATH: docker/parabol-ubi/environments/pipeline
 jobs:
   build:
     runs-on: ubuntu-8cores
@@ -22,8 +22,7 @@ jobs:
       id-token: "write"
     services:
       postgres:
-        # Image is pinned to v15, OK since it's just for testing
-        image: ankane/pgvector
+        image: pgvector/pgvector:pg15
         # This env variables must be the same in the file PARABOL_BUILD_ENV_PATH
         env:
           POSTGRES_PASSWORD: "temppassword"

.github/workflows/test.yml

@@ -8,7 +8,7 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 env:
-  PARABOL_BUILD_ENV_PATH: docker/parabol-ubi/docker-build/environments/pipeline
+  PARABOL_BUILD_ENV_PATH: docker/parabol-ubi/environments/pipeline
 jobs:
   test:
     runs-on: ubuntu-8cores

docker/parabol-ubi/docker-build/README.md → docker/parabol-ubi/README.md

@@ -21,9 +21,9 @@ Recommended:
 | `postgresql_tag`     | PostgreSQL version from the [Docker image](https://hub.docker.com/_/postgres)                                           | `Any tag`                                             | `15.4`                                                              |
 | `rethinkdb_tag`      | RethinkDB version from the [Docker image](https://hub.docker.com/_/rethinkdb)                                           | `Any tag`                                             | `2.4.2`                                                             |
 | `redis_tag`          | Redis version from the [Docker image](https://hub.docker.com/_/redis)                                                   | `Any tag`                                             | `7.0-alpine`                                                        |
-| `_BUILD_ENV_PATH`    | File `.env` used by the application during the build process                                                            | `Relative path from the root level of the repository` | `docker/parabol-ubi/docker-build/environments/basic-env`             |
+| `_BUILD_ENV_PATH`    | File `.env` used by the application during the build process                                                            | `Relative path from the root level of the repository` | `docker/parabol-ubi/environments/basic-env`             |
 | `_NODE_VERSION`      | Node version, used by Docker to use the Docker image node:\_NODE_VERSION as base image to build                         | `Same as in root package.json`                        |                                                                     |
-| `_DOCKERFILE`        | Dockerfile used to build the image                                                                                      | `Relative path from the root level of the repository` | `./docker/parabol-ubi/docker-build/dockerfiles/basic.dockerfile` |
+| `_DOCKERFILE`        | Dockerfile used to build the image                                                                                      | `Relative path from the root level of the repository` | `./docker/parabol-ubi/dockerfiles/basic.dockerfile` |
 | `_DOCKER_REPOSITORY` | The destination repository                                                                                              | `String`                                              | `parabol`                                                           |
 | `_DOCKER_TAG`        | Tag for the produced image                                                                                              | `String`                                              |                                                                     |
 
@@ -33,9 +33,9 @@ Example of variables:
 export postgresql_tag=15.4; \
 export rethinkdb_tag=2.4.2; \
 export redis_tag=7.0-alpine; \
-export _BUILD_ENV_PATH=docker/parabol-ubi/docker-build/environments/basic-env; \
+export _BUILD_ENV_PATH=docker/parabol-ubi/environments/basic-env; \
 export _NODE_VERSION=$(jq -r -j '.engines.node|ltrimstr("^")' package.json); \
-export _DOCKERFILE=./docker/parabol-ubi/docker-build/dockerfiles/basic.dockerfile; \
+export _DOCKERFILE=./docker/parabol-ubi/dockerfiles/basic.dockerfile; \
 export _DOCKER_REPOSITORY=parabol; \
 export _DOCKER_TAG=test-image
 ```

docker/parabol-ubi/docker-build/dockerfiles/basic.dockerfile → docker/parabol-ubi/dockerfiles/basic.dockerfile

@@ -7,8 +7,8 @@ ENV HOME=/home/node \
 ENV NPM_CONFIG_PREFIX=/home/node/.npm-global
 ENV PORT=3000
 
-COPY --chown=node --chmod=755 docker/parabol-ubi/docker-build/entrypoints/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
-COPY --chown=node docker/parabol-ubi/docker-build/tools/ip-to-server_id ${HOME}/tools/ip-to-server_id
+COPY --chown=node --chmod=755 docker/parabol-ubi/entrypoints/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
+COPY --chown=node docker/parabol-ubi/tools/ip-to-server_id ${HOME}/tools/ip-to-server_id
 
 # Required for pushToCDN to work with FILE_STORE_PROVIDER set to 'local'
 RUN mkdir -p ${HOME}/parabol/self-hosted && \

docker/parabol-ubi/docker-build/environments/basic-env → docker/parabol-ubi/environments/basic-env

@@ -4,7 +4,7 @@ NODE_ENV='production'
 NODE_EXTRA_CA_CERTS=''
 PROTO='https'
 PORT='3000'
-# Database configurations must be the same used in the docker-build.yml Github workflow
+# Database configurations must be the same used in the build.yml Github workflow
 POSTGRES_PASSWORD='temppassword'
 POSTGRES_USER='tempuser'
 POSTGRES_DB='tempdb'

docker/parabol-ubi/docker-build/environments/pipeline → docker/parabol-ubi/environments/pipeline

@@ -36,7 +36,7 @@ PGADMIN_DEFAULT_EMAIL=''
 PGADMIN_DEFAULT_PASSWORD=''
 PGSSLMODE=''
 PORT='3000'
-# Database configurations must be the same used in the docker-build.yml Github workflow
+# Database configurations must be the same used in the build.yml Github workflow
 POSTGRES_PASSWORD='temppassword'
 POSTGRES_USER='tempuser'
 POSTGRES_DB='tempdb'

packages/server/postgres/Dockerfile → docker/postgres/Dockerfile

@@ -9,7 +9,7 @@ RUN apt-get update && apt-get install -y \
     git
 
 RUN cd /extensions/postgres-json-schema && make install && make installcheck
-RUN git clone --branch v0.5.0 \
+RUN git clone --branch v0.5.1 \
        https://github.com/pgvector/pgvector.git /extensions/pgvector && \
        cd extensions/pgvector && make clean && make && make install
 

docs/alternative-licenses/us-department-of-defense/README.md

@@ -82,7 +82,7 @@ $ yarn && yarn build && yarn start
 - Click "Add New Server" and fill out the forms with your `.env` values
 
   - General.name = POSTGRES_DB
-  - Connection.host = 'postgres' (hardcoded from docker-compose dev.yml, not from .env!)
+  - Connection.host = 'postgres' (hardcoded from docker-compose `run-it/development/docker-compose.yml`, not from .env!)
   - Connection.username = POSTGRES_USER
   - Connection.password = POSTGRES_PASSWORD
   - Connection.maintenanceDatabase = POSTGRES_DB

package.json

@@ -21,7 +21,7 @@
     "packages/*"
   ],
   "scripts": {
-    "start": "pm2-runtime start pm2.config.js",
+    "start": "pm2-runtime start run-it/development/pm2.config.js",
     "heroku-postbuild": "echo 'Skipping build'",
     "build": "node --max-old-space-size=8192 scripts/prod.js",
     "clean": "git clean -fdx -e .env",
@@ -30,12 +30,12 @@
     "pg:migrate": "node-pg-migrate -f ./packages/server/postgres/pgmConfig.js",
     "pg:generate": "export $(grep ^POSTGRES_ .env | tr -d \"'\"); yarn kysely-codegen --exclude-pattern \"(PgMigrations|StripeQuantityMismatchLogging)\" --out-file ./packages/server/postgres/pg.d.ts --dialect postgres --url postgresql://$POSTGRES_USER:$POSTGRES_PASSWORD@$POSTGRES_HOST:$POSTGRES_PORT/$POSTGRES_DB && prettier --write ./packages/server/postgres/pg.d.ts",
     "pg:restore": "node ./scripts/toolbox/pgRestore.js",
-    "db:start": "docker compose -f docker/dev.yml up -d",
-    "db:stop": "docker compose -f docker/dev.yml down",
+    "db:start": "docker compose -f run-it/development/docker-compose.yml up -d",
+    "db:stop": "docker compose -f run-it/development/docker-compose.yml down",
     "db:migrate": "node scripts/migrate.js",
     "deduplicate": "yarn yarn-deduplicate yarn.lock",
     "predeploy": "node dist/preDeploy.js",
-    "dev": "PM2_SILENT=true pm2 start pm2.dev.config.js --no-daemon",
+    "dev": "PM2_SILENT=true pm2 start run-it/development/pm2.dev.config.js --no-daemon",
     "lint": "nx run-many --target=lint",
     "newMutation": "yarn sucrase-node scripts/codeshift/newMutation.ts",
     "typecheck": "nx run-many --target=typecheck",

packages/server/database/README.md

@@ -10,4 +10,3 @@ Since that time, RethinkDB has occassionally been unstable for us under high loa
 Since our data is very relational, it made sense to move PostgresQL, which we are actively doing.
 
 - Migrations are stored in [`packages/server/database/migrations`](./migrations/)
-- RethinkDB Dashboard is at [http://localhost:8080](http://localhost:8080)

packages/server/postgres/README.md

@@ -1,18 +1,5 @@
 # PostgreSQL
 
-## Setup
-
-- pgadmin is at [http://localhost:5050](http://localhost:5050)
-- Connect using the values of `PGADMIN_DEFAULT_EMAIL` and `PGADMIN_DEFAULT_PASSWORD` from your `.env`
-- Click "Add New Server" and fill out the forms with your `.env` values
-
-  - General.name = POSTGRES_DB
-  - Connection.host = 'postgres' (hardcoded from docker-compose dev.yml, not from .env!)
-  - Connection.username = POSTGRES_USER
-  - Connection.password = POSTGRES_PASSWORD
-  - Connection.maintenanceDatabase = POSTGRES_DB
-  - Connection.port = POSTGRES_PORT
-
 ## Migrations
 
 This folder contains all the postgres migrations that have been run on the database.
@@ -56,13 +43,3 @@ Parameters are capped at 16-bit, so if you're doing a bulk insert, you'll need t
 In other words, if `# rows * columns per row > 65,535` you need to do it in batches.
 `pg-protocol` shows this here: <https://github.com/brianc/node-postgres/blob/master/packages/pg-protocol/src/serializer.ts#L155>
 Issue here: <https://github.com/brianc/node-postgres/issues/581>
-
-#### Too many connections
-
-Sometimes pg pool will hit its connection limit. This should never happen in prod, but happens on occassion in dev.
-You'll know it's happening because PG will say there are too many connections.
-To fix, you can run the following SQL to remove all the connections except the one that is running the script
-
-```sql
-select pg_terminate_backend(pid) from pg_stat_activity where datname='parabol-saas' AND pid <> pg_backend_pid();
-```

run-it/development/README.md

@@ -0,0 +1,44 @@
+# Development stack
+
+> ⚠️ **Parabol does not provide any support on this stack**. Use it under your own resposibility.
+
+## General notes
+
+- **This stack is not meant for production use.** It is our development stack and can change at any moment, have errors and incorporate and remove components we are testing without any notice.
+- A .env file localed in the root of this repository is used as [Docker env_file](https://docs.docker.com/compose/environment-variables/set-environment-variables/#use-the-env_file-attribute). It is the same file used by our command `yarn dev` to run the application in development mode.
+- This stack is designed to be managed using `yarn db:start` and `yarn db:stop` to start the databases and `yarn dev` or `yarn start` to start and manage the application, depending on the use case.
+
+## Components
+
+- **Datadog agent:** additional configuration can be added in the folder `datadog/dd-conf.d`.
+- **RethinkDB:** ports 28015 and 8080 (console) exposed to communicate with the cluster. Data mounted in a volume rethinkdb-data.
+- **Postgres:** container built from a Dockerfile in [docker/postgres](docker/postgres), that incorporates some extra extensions used by the application. Exposed through port 5432 and with the data mounted in a volume postgres-data.
+- **PGAdmin:** available on 5050 with credentials on the `.env` file. Connect using the values of `PGADMIN_DEFAULT_EMAIL` and `PGADMIN_DEFAULT_PASSWORD` from the `.env`. Data mounted on a volume pgadmin-data.
+- **Redis:** available on 6379 with the data mounted on a volume redis-data.
+- **Redis Commander:** available on 8081.
+- **Text Embedding Inference:** toolkit to deploy and serve open source text embeddings and sequence classification models. Exposed on 3040. More information in [their Github](https://github.com/huggingface/text-embeddings-inference).
+
+### Configue PGAdmin
+
+- pgadmin is at [http://localhost:5050](http://localhost:5050)
+- Connect using the values of `PGADMIN_DEFAULT_EMAIL` and `PGADMIN_DEFAULT_PASSWORD` from your `.env`
+- Click **Add New Server** and fill out the forms with your `.env` values
+
+  - General.name = POSTGRES_DB
+  - Connection.host = 'postgres' (hardcoded from docker-compose dev.yml, not from .env!)
+  - Connection.username = POSTGRES_USER
+  - Connection.password = POSTGRES_PASSWORD
+  - Connection.maintenanceDatabase = POSTGRES_DB
+  - Connection.port = POSTGRES_PORT
+
+### Postgres
+
+#### Too many connections
+
+Sometimes pg pool will hit its connection limit. This should never happen in prod, but happens on occassion in dev.
+You'll know it's happening because PG will say there are too many connections.
+To fix, you can run the following SQL to remove all the connections except the one that is running the script
+
+```sql
+select pg_terminate_backend(pid) from pg_stat_activity where datname='parabol-saas' AND pid <> pg_backend_pid();
+```