Deployment changes

[ChangingTerry]

Thanks for you feedback on my last pull request. I had a look through as you suggested and have a few imporovements that may make things better for the end user.

1. I moved all the environment variables to their respective Dockerfiles. This means that if a user does not copy the .env.example file correctly the backend/frontend will still start with defaults and not error out straight away. It also indicates to other docker deployment managers (e.g. portainer) what the expected environment vars will be.
2. Moved the frontend to a single nginx instance. Proxy static files inside a 2nd nginx seemed a bit odd. This way you can reduce the number of services running which is easier to maintain.
3. Added package-json.lock, moved to a CI install in Dockerfile and dropped root at the build stage. This allows for faster more consitent builds (if you need to revert or anything). Also if any of your packages get hit with a supply chain attack they can't run root scripts on build.
4. Removed the generated certficates that were still in the repo - these were (I assume) just self-signed ones from the script that used to be present but these should not be hard coded to my understanding.
5. Adding a run time step for production frontend build to deploy self-signed certs if no TLS cert present. This will encourage users to be SSL first and a good look for a security product. I updated the README with details about this. This is done using the templates feature from the nginx image > 1.9 and simple entry point script.
6. I added many security headers to the final nginx output. I mostly stole these from MISP and is it NOT fully audited but should be a good starting point.
7. Now the defaults have all been moved I also updated the deploy to just need the docker-compose.yml file which less work than cloning a repo IMO
8. I added a dev version of the docker-compose for you guys when you're working locally. This will enable SSL and the proxy on vite so you are more closely mirroring a live environment when you're developing. This was a bit quick but works well in my testing but feel free to remove.
9. Moved all data to a single directory to make it easier to backup/manage

Couple of bit to be aware of I noticed:

Obviously having the self signed certs left in the repo, even though they are probably benign, is not a great look for a security product. You'll have a security review and keep having to explain about then. You might want to remove those files from all commits and save yourself the headache.

In your original docker compose you referred to the frontend image as socfortress/copilot-frontend:latest and the backend as ghcr.io/socfortress/copilot-backend:latest. With this method the client won't have to build thier own frontend. I wanted to make sure you intend for frontend to live next to backend so I put the image as ghcr.io/socfortress/copilot-frontend:latest and not the docker hub verson.

The deploy script references a specfic release version in the README.md, currently 0.0.2 in this link: wget https://raw.githubusercontent.com/socfortress/CoPilot/v0.0.2/docker-compose.yml - this you will need to change on each release but should be easy to script.

The dev version of the frontend build uses a cert created at build time. If you want me to make this more flexable let me know.

[taylorwalton]

hey @ChangingTerry thanks for the assistance....i appreciate your recommendations! Have you tried building the frontend image? I cloned your repo but got the below error when attempting to build the frontend:

 => [stage-2 5/7] RUN mkdir /etc/nginx/templates                                                                                                          1.0s 
 => [stage-2 6/7] COPY build/etc/nginx/sites-enabled/default.conf /etc/nginx/templates/default.conf.template                                              0.1s 
 => [builder 2/5] WORKDIR /app                                                                                                                            1.7s 
 => [builder 3/5] COPY build .                                                                                                                            0.1s 
 => ERROR [builder 4/5] RUN npm ci                                                                                                                        1.8s 
------
 > [builder 4/5] RUN npm ci:
1.708 npm ERR! code EUSAGE
1.724 npm ERR!
1.725 npm ERR! The `npm ci` command can only install with an existing package-lock.json or
1.725 npm ERR! npm-shrinkwrap.json with lockfileVersion >= 1. Run an install with npm@5 or
1.725 npm ERR! later to generate a package-lock.json file, then try again.
1.725 npm ERR!
1.726 npm ERR! Clean install a project
1.726 npm ERR!
1.726 npm ERR! Usage:
1.726 npm ERR! npm ci
1.746 npm ERR!
1.753 npm ERR! Options:
1.753 npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
1.753 npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
1.753 npm ERR! [--include <prod|dev|optional|peer> [--include <prod|dev|optional|peer> ...]]
1.753 npm ERR! [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
1.753 npm ERR! [--no-bin-links] [--no-fund] [--dry-run]
1.753 npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
1.753 npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
1.753 npm ERR!
1.753 npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
1.753 npm ERR!
1.753 npm ERR! Run "npm help ci" for more info
1.757
1.758 npm ERR! A complete log of this run can be found in: /home/node/.npm/_logs/2024-02-11T16_50_46_200Z-debug-0.log
------
Dockerfile:14
--------------------
  12 |
  13 |     # Install project dependencies
  14 | >>> RUN npm ci
  15 |
  16 |     # Run the Vue.js project build
--------------------
ERROR: failed to solve: process "/bin/sh -c npm ci" did not complete successfully: exit code: 1

[ChangingTerry]

Sorry, looks like my ide changed the path in the Dockerfile and I didn't recheck it. .

If you clone it now and do cd frontend && docker build . it'll work now. I tested building on linux and docker desktop to be sure.

[taylorwalton]

have you built a frontend image yourself? I've built the image based off your latest changes (i havent totally dug into them yet, but just wanted to quickly test), but its not finding the 90-copilot-ssl.sh at runtime:

copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
copilot_new-copilot-frontend-1  | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
copilot_new-copilot-frontend-1  | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
copilot_new-copilot-frontend-1  | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/90-copilot-ssl.sh
copilot_new-copilot-frontend-1  | /docker-entrypoint.sh: line 31: /docker-entrypoint.d/90-copilot-ssl.sh: not found
copilot_new-copilot-frontend-1 exited with code 127

i havent tried creating own certs yet

[ChangingTerry]

I'm very sorry - I didn't mean to make more work for you.

I have and I just cleared my cache and tried again.... all good.

It's a bizzare error as you can see /docker-entrypoint.sh: Launching /docker-entrypoint.d/90-copilot-ssl.sh that it found the script to attempt to run it.

Looking at here which seems to be the line it failed - is there a chance you checked this out with Windows line endings applied? It feels like it's not reading the bang at the top?

[taylorwalton]

hey @ChangingTerry , no probs at all, I appreciate the changes! and yup I hit that script with a dos2unix, rebuilt the image and all good now 👍

I do have a question regarding your change to the install docs, did you mean to remove the step to have the user create the .env file or are you just assuming that step will be known to be taken?

Thanks again!

[ChangingTerry]

No probs - i've had so much value out of your videos and medium guides, it felt like a tiny way to give back.

When you have a .env which has to be copied or the backend service will fail to start - by moving this to the Dockerfile it means that first the environment var will always be present ensure the container can start. Second it one less step in deployment which in my experience (I might be well off with what you envisioned) means more users will try it as they just run the old docker compose up -d and it will work even if environment file is missing.

When a user is ready to start adding in thier credentials for other services they can then start overiding the env vars either direct in the compose file, with an env file or using vault or another key manager (prod style). This gives them to option to manage how they want.

Also some docker orchestrators/manages read the environment vars out of the image (docker inspect) and this will then give autocomplete/suggestions to people using it.

So yes, on deploy it would be now imagined that the user does not need the .env file, they can just set these values how they want and it's less brittle.

[ChangingTerry]

Glad you liked it - I'm currently playing with it on my own stack and it's a good product.

If I can help with anything else let me know

[taylorwalton]

All great changes, thanks again :) ... will do and any features / improvements you recommend just let me know! still alot of ideas I have in mind that I plan to roll out but any other feedback is welcome

[compgeniuses]

thats for these wonderful changes, now i can begin working on a nethserve 8 module for the same, to make it easy to isntall for SME