From 5b5e6afd3f3557f618c71b9fea30ce07d8646122 Mon Sep 17 00:00:00 2001 From: EduardoFF Date: Sun, 15 Oct 2023 11:12:46 +0300 Subject: [PATCH] P3 instructions --- docs/projects/P3/deployment.md | 171 +++++++++++++++++---------------- docs/projects/P3/index.md | 141 ++++----------------------- 2 files changed, 104 insertions(+), 208 deletions(-) diff --git a/docs/projects/P3/deployment.md b/docs/projects/P3/deployment.md index a6306c1fd..c256565c8 100644 --- a/docs/projects/P3/deployment.md +++ b/docs/projects/P3/deployment.md @@ -1,101 +1,102 @@ -# Deploying NodeBB to Fly.io +# Serverless NodeBB Deployment on GCP -This document will provide instructions to deploy NodeBB on Fly.io. We recommend following this tutorial with a clean install of NodeBB to ensure it works. Once you successfully deploy, you can add your changes. +This document will provide instructions to create a serverless deployment of NodeBB on GCP. -The steps are as follows: +## Setup Redis Database on upstash -## Install Fly.io command line tools +We'll be using a database-as-a-service called upstash to host the Redis database associated with our NodeBB deployment. -First, you should install and log into Fly.io’s command line utility. Follow these guides! +1. Visit [upstash](upstash.com) and create an account. +2. Make sure Redis and selected, and click "Create Database" +3. Name your database, select an appropriate region and hit "Create" -- [https://fly.io/docs/hands-on/install-flyctl/](https://fly.io/docs/hands-on/install-flyctl/) -- [https://fly.io/docs/getting-started/log-in-to-fly/](https://fly.io/docs/getting-started/log-in-to-fly/) +You should see a page that contains the endpoint, port, password, and other details associated with the redis instance you just created. -## Edit database connection file +## Update your Dockerfile -In `src/database/redis/connection.js`. Add +To have all the packages we need to build NodeBB, we need to change our Dockerfile. Make the following change and push your changes to your repository. -```JS -family: 6, +``` +- RUN npm install --only=prod && \ ++ RUN npm install && \ ``` -to whenever a new Redis object is created. ~ lines 20, 28, 37. i.e. +## Redeem your GCP Credits -```JS hl_lines="4" -new Redis({ - sentinels: options.sentinels, - ...options.options, - family: 6, -}); -``` +First, you need to redeem your GCP credits using the following instructions. + +1. Fill out the [this form](https://gcp.secure.force.com/GCPEDU?cid=XufGAKC79oWNCFv7SSDIXbCrKy3MTPzQgZwC9%2FJ1AuF%2Fv89lUW0CmubO%2FsGm2Izw/) link with your first/last name, and Andrew ID +2. Go to your school email and click the link in an email from "Google Cloud Notifications" to verify your email address +3. Go back to your email, click on the link ("click here to redeem") within a second email from "Google Cloud Notifications," and copy the provided code into the field within the new link + +Once you submit, a $50 credit should be applied to your GCP billing account. + +!!! Warning "Do not misuse!" + We've been awarded enough credits such that each student in 17-313 can redeem one coupon. We'll be closely monitoring coupon redemption. Any and all misuse including sharing redemption instructions, redeeming multiple times, etc. will be punished. + + +## Deploy on GCP Cloud Run + +Make sure you're logged into the Google account you used to redeem your GCP credits. + +Once you're logged into the right credit-bearing Google account, use the following instructions to deploy on GCP Cloud Run. + +1. Create a project called "NodeBB" using the [GCP Cloud Console](https://console.cloud.google.com/projectcreate?previousPage=%2Fwelcome%3Fproject%3Dextreme-startup&organizationId=703967796528) (you can set the location to "Students") +2. Visit the [Cloud Run console](https://console.cloud.google.com/run) and select the project you just created using the project selector drop down(top-left) +3. Click on "Create Service" +4. Select "Continuously deploy new revisions from a source repository" and click "Set up with Cloud Build" +5. Set the Source repository to be your team's NodeBB repository - you may need to click on "Manage connected repositories" and authenticate with GitHub if you don't see the repository. +6. Set the Build Type to the "Dockerfile" option +7. In the "Autoscaling" section, set the minimum number of instances to 1 +8. In the "Authentication" section select "Allow unauthenticated invocations" +9. Click on the "Container, Networking, and Security" dropdown and set the "Container Port" to 4567 +10.Click "Create" +11. Once the deployment is complete, click on the URL of the form `*.run.app` at the top of the page to view the deployment. + +You should see a form that says "NodeBB Web Installer". Keep this URL handy because you'll need it later :) -The Redis database that Fly.io creates connects via IPv6, so we must change our connection config to account for this. - -### Initial Launch - -1. In your NodeBB directory, run `flyctl launch` -2. For the following prompts, answer as follows: - - **Name** -> Anything reasonable you’d like for your app name - - **Region** -> Ashburn, Virginia (or any reasonable US ones) - - **Setup a Postgres DB?** -> No - - **Setup a Redis DB?** -> Yes - - Choose the free 100MB data plan - - It’ll output something along these lines - ```console - Your Upstash Redis database nodebb-redis is ready. - Apps in the personal org can connect to at - redis://default:a2b2e64367fa4b7c8a14f8f8d558f3dc@fly-nodebb-redis.upstash.io - ``` - - Keep this URL handy! - - **Create a .dockerignore?** -> No - - **Deploy now?** -> No - - We still need to update some config files -3. Edit `fly.toml` - - Change internal_port to 4567. This is the port that NodeBB uses as default. - - !!! warning - Make sure to delete your local `config.json` file before proceeding to the next step This will be generated by NodeBB when it runs`./nodebb setup` remotely. - -4. Run `flyctl deploy` - - This will build the image. - - Click the monitoring link, which will open your app’s dashboard on Fly.io - -## Remote Setup - -1. On the monitoring page, click Overview on the left navbar, and click the purple hostname link. You should now see the NodeBB installer page -2. Enter admin username, email, and password, as usual -3. For database type, select Redis -4. Populate database form. Remember that URL from the initial launch? - - Host IP or address: the `fly-XXXX-redis.upstash.io` part of the link above - - Host port: 6379 - - Password: the string between the `:` and `@` of the link - - Example URL: redis://default:==a2b2e64367fa4b7c8a14f8f8d558f3dc==@^^fly-nodebb-redis.upstash.io^^ - - The underlined text is your host, and highlighted text is your password. -5. Click **Install NodeBB** -6. Monitor deployment back on Fly.io page -7. It will run, and eventually attempt to build the TypeScript files. **Here, it will fail. This is okay!** The free Fly.io instances don’t have enough memory to fully build NodeBB. To get around this, we’ll build NodeBB locally and deploy the compiled files. The steps we’ve completed thus far just served to set up our database correctly. - -## Local Setup - -1. Run `./nodebb setup` and set it up like usual. No need to run `./nodebb start` -2. The above command will build NodeBB and generate a `config.json` file. We’ll need to edit this so that it matches the `config.json` of our deployed app. -3. Edit `config.json` - a. Run `flyctl ssh console` to SSH into your app. - b. Run `cd usr/src/app` - c. Run `cat config.json` - d. Copy the output from the console into your local `config.json` file -4. Next, we’ll update the Dockerfile (which are the instructions Fly.io uses to deploy your app) - a. Open the `Dockerfile` file in your local NodeBB directory - b. Update line 25 as follows. We remove the build command from the deployment instructions. This way, we use the build artifacts from our local build. -```Dockerfile --CMD test -n "${SETUP}" && ./nodebb setup || node ./nodebb build; node ./nodebb start -+CMD test -n "${SETUP}" && ./nodebb setup || node ./nodebb start +## Create Config Script + +At this point, we could use the web installer to generate the `config.json` file in our container and setup NodeBB. However, since this a serverless deployment, we're not guaranteed any persistence of data generated at runtime. + +Therefore, we need to change our Dockerfile to generate the `config.json` file at build time of the container. + +To do so, first create a template file called `config_template.json` that looks exactly like the following, and push your changes to your repository. + +``` +{ + "url": "", + "secret": "c5502d62-84a5-41f1-87eb-ee33a76fb7bc", + "database": "redis", + "redis": { + "host": "", + "port": "", + "password": "", + "database": "0" + }, + "port": "4567" +} ``` -## Deploy -1. Run `flyctl deploy` -2. Click the monitoring link, which will open your app’s dashboard on Fly.io. Wait until the log output shows that NodeBB was initialized. Click the overview again and open your app! +!!! info "Why can't we just push a pre-populated `config.json` file?" + This would solve the persistence problem and deploy NodeBB correctly. However, as a result, we expose secrets like the upstash redis connection details on a public GitHub repository. Injecting these secrets as environment variables at runtime gives our deployment access to them, while ensuring that the secrets remain secret. -Congratulations, you have now successfully deployed NodeBB! +Configure the following environment variables by visiting the Cloud Run dashboard for your deployment, clicking on "Edit and deploy new revision" and then clicking on "Add Variable" in the "Environment Variables" section. + +* `DEPLOYMENT_URL`: URL of the form `*.run.app` that your Cloud Run deployment is live at. +* `REDIS_HOST`: Endpoint value from your upstash redis database dashboard. +* `REDIS_PORT`: Port value from your upstash redis database dashboard. +* `REDIS_PASSWORD`: Password value from your upstash redis database dashboard. + +Click on "Deploy" at the bottom of the page to save your changes. + +We'll now add a bash script that will populate this template with environment variables defined at build time of our Docker container. Create a file called `create_config.sh` in your NodeBB repo, and populate the file with the following bash script. + +``` +#!/bin/bash -Be sure to share the application link with your teammates to ensure they can also access the application and test that all your added features from Project 2 have been successfully applied. +# Check that environment variables have been defined +if [[ -z "${REDIS_HOST+x}" ]]; then + # var is not defined + echo "Error: REDIS_HOST is not defined!" + exit 1 diff --git a/docs/projects/P3/index.md b/docs/projects/P3/index.md index cd180932a..0e156d797 100644 --- a/docs/projects/P3/index.md +++ b/docs/projects/P3/index.md @@ -12,7 +12,7 @@ Adding continuous integration for quality assurance is a critical part of software development. Although you have been testing your new system this whole time, you are now setting out to establish sustained practices that can be used moving forward as you iterate over and continue to improve your system. -Your manager has assigned you two major tasks. Firstly, establishing a deployment pipeline to create a test version of the website that can be sent to beta testers who have little to no experience with code development (and therefore can not set up the developer environment by themselves). +Your manager has assigned you two major tasks. Firstly, establishing a deployment pipeline to create a test version of the website that can be sent to beta testers who have little to no experience with code development (and therefore can not set up the developer environment by themselves). Secondly, evaluating existing tools and practices beyond simple linting or unit testing, then producing a report on the cost/benefit tradeoffs and risks of them. You will also select and integrate one (or more!) of these tools into your development process. @@ -20,37 +20,36 @@ Secondly, evaluating existing tools and practices beyond simple linting or unit ## Deliverables and Deadlines There are two (2) deadlines for this project. This project is worth a total of 120 points. -**Checkpoint Deliverables** – 35 points – due ~~Thursday, March 16th~~ Friday, March 17th, 11:59pm +**Checkpoint Deliverables** – 35 points – due Thursday, October 26th, 11:59pm - [Deployed Application (25 pts)](#deployed-application-25-pts) - [Tools Checkpoint (10 pts)](#tools-checkpoint-10-pts) -**Final Deliverables** – 85 points – due Thursday, March 23rd, 11:59pm +**Final Deliverables** – 65 points – due Thursday, November 2, 11:59pm -- [CD Implementation (10 pts)](#cd-implementation-10-pts) -- [Tool Analysis Design Doc (60 pts)](#tool-analysis-design-doc-60-pts) +- [Tool Analysis Design Doc (50 pts)](#tool-analysis-design-doc-60-pts) - [Tool Integration (15 pts)](#tool-integration-15-pts) -**Extra Credit (Individual) ** - 6 points - due Thursday, March 23rd, 11:59pm +**Extra Credit (Individual)** - 6 points - due Thursday, November 2, 11:59pm - [Feature Review (6 pts)](#feature-review-6-pts) !!! info "Work Distribution" - There are two main focuses in this project: deployment and static/dynamic analysis. For the purposes of equitable distribution of labor, we recommend that you nominate one of your members to act as the SRE for this assignment who will be primarily responsible for deployment, and have all other teammates focus on tool research and integration. + There are two main focuses in this project: deployment and static/dynamic analysis. For the purposes of equitable distribution of labor, we recommend that you nominate one of your members to act as the SRE for this assignment who will be primarily responsible for deployment, and have all other teammates focus on tool research and integration. ## Checkpoint Deliverables ### Deployed Application (25 pts) -Your team will be using the website fly.io for the deployment of the NodeBB application. Further instructions on how to deploy can be found [here](/projects/P3/deployment). +Your team will be using Google Cloud Platform for the deployment of the NodeBB application. Further instructions on how to deploy can be found [here](/projects/P3/deployment). -Once you have successfully deployed your website, make sure to test within your team to ensure that your added feature(s) from Project 2 are properly integrated. +Once you have successfully deployed your website, make sure to test within your team to ensure that your added feature(s) from Project 2 are properly integrated. By the checkpoint deadline you should - Submit a link to the deployed site onto Gradescope -- Add your deployed site to this [public spreadsheet](https://docs.google.com/spreadsheets/d/1MjR3MC6kRoXZCfIKmrR-SAiZCXOYRsug6NzAoIhFm8Q/), alongside your team name & UserGuide.md that your team submitted for Project 2. This will be used in [Feature Review](#feature-review-6-pts) for extra credit. +- Add your deployed site to this [public spreadsheet](https://docs.google.com/spreadsheets/d/155eTtRrJGFE0QtTapAca3MhTSZ9qh9NlhENS4MRDN80/edit?usp=sharing), alongside your team name & UserGuide.md that your team submitted for Project 2. This will be used in [Feature Review](#feature-review-6-pts) for extra credit. ### Tools Checkpoint (10 pts) @@ -69,19 +68,19 @@ For each tool that you assess 1. Create a separate testing branch in your repository (named appropriately for the tool you’re testing) to integrate the tool into your project and test out its capabilities 2. Create a pull request to the main branch from each of these testing branches. The PR should have - + - **Concrete evidence that you had successfully installed the tool** through trackable file changes demonstrating that extra files/NPM packages were installed. - **Artifacts that demonstrate that you have successfully run the tool on your repository.** Acceptable artifacts include output files generated by the tool, or a text file containing the terminal output from the tool; you may also attach screenshots as additional pieces of evidence. They can be attached to the Pull Request in either the description or follow-up comments. !!! note "Grading Note" We will not be grading the quality or quantity of any code you put into these testing branches/PRs, just the evidence that you have successfully installed and run the tool. - + In your evaluations, consider & experiment with the types of customization that are appropriate or necessary for this tool, both a priori (before they can be used in your project) and over time. Assess the strengths and weaknesses of each tool/technique, both quantitatively and qualitatively. -Consider some of the following questions: +Consider some of the following questions: -- What types of problems are you hoping your tooling will catch? What types of problems does this particular tool catch? -- What types of customization are possible or necessary? +- What types of problems are you hoping your tooling will catch? What types of problems does this particular tool catch? +- What types of customization are possible or necessary? - How can/should this tool be integrated into a development process? - Are there many false positives? False negatives? True positive reports about things you don’t care about? @@ -91,23 +90,14 @@ Consider some of the following questions: !!! note "Time Management" Don’t spend too long for this checkpoint. Set deadlines within your team to ensure that you have enough time for both the design document and integration deliverables described below for the final deadline. -By the checkpoint deadline, your team will submit +By the checkpoint deadline, your team will submit -- your **initial list of the N-1 tools** that your team plans on exploring, and +- your **initial list of the N-1 tools** that your team plans on exploring, and - links to the **PRs that demonstrate that you have successfully installed and run each of these tools** ## Final Deliverables -### CD Implementation (10 pts) -Now that you have successfully deployed your application manually for the checkpoint deadline, add a workflow to your team repository that will automatically deploy your application. In doing so, consider how often you would want to deploy, and if you would need to make changes to your repository to ensure best development practices. - -!!! warning "GitHub Secrets" - You may have sensitive or secret values that will be required by your workflow in order to successfully deploy your application. As your team repository is public, be sure to follow best GitHub practice in [creating secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets) to prevent such values from being leaked. - -Submit a link to a successful deployment GitHub Action run along with a brief description/justification of how you set up the workflow to Gradescope. - - -### Tool Analysis Design Doc (60 pts) +### Tool Analysis Design Doc (50 pts) Create a Design Document/RFC that includes: @@ -142,7 +132,7 @@ This section should address the different factors to take into consideration whe - **Technical Questions** - How are you integrating the tool (high-level)? At what point in the development/deployment process shall it be integrated? What sorts of customization or configuration will you be using? - If you added any specific configuration to allow the main branch of your repository to pass its status checks, add the justification for those decisions in this section. -- **Social Integration Questions** +- **Social Integration Questions** - How do you foresee the team using the tool during their development process? Consider the incentives & deterrents to the developers when it comes to using the tool, and their personal motivation to use it. Your answers should be based on your experiences running the tools on your team repository and be grounded in data from your research on different factors such as tool usability, output, and customizability. @@ -155,98 +145,3 @@ In this section, provide a brief summary of your findings along with items that - Are there any open questions? - Are there any issues you consider to be out of scope? -- What drawbacks of the proposed process/tooling are you accepting for some (good) reason? - -This section should be used to wrap everything up and ensure you have a good/complete design document! - -Submit the Design Document as a single PDF to Gradescope. - -### Tool Integration (15 pts) - -Once you have a tool selected along with a general integration plan, you should fully integrate one of the tools into your project’s workflow. For your checkpoint research, you should have successfully run this tool locally; you should then create a new workflow within the project to run it as part of the development cycle. - -Your team should discuss: - -- How often should this new integration be run (on each pull request? on commits to main?) -- What level of customization is needed for this tool? -- How should the integration of this tool be enforced? - -This configuration must have been justified in your design document under the Integration section. - -To be considered successfully integrated, the tool must: - -- Be **merged into your main branch** -- **Have been run at least once in the Git flow cycle** (i.e. either during the pull request, merge, or commit stage). -- **Pass** when run on your codebase. This is indicated by having a green checkmark. - -!!! note "Ensuring Passing Checks" - In order to ensure your checks pass successfully, you may have to make additional changes to your repository, such as fixing reported issues or tweaking tool configuration. - - These changes should be documented and addressed in your design document. Continually failing builds show you have *not* completely integrated the tool into your workflow. - -On Gradescope, submit a link to your repository and a link to one of the successful GitHub Action runs. - -## Extra Credit - -Now that you and your classmates have deployed your applications, you will be able to test out each other’s features and provide constructive feedback on your experience and how to improve them! Take this as an opportunity to learn about what your classmates have been working on for the past few weeks. - -Note that this is an **individual** task, unlike the rest of project. - -### Feature Review (6 pts) - -For extra credit, you will conduct reviews of features developed by three other teams' project. Pick **three** teams's deployment from the [public spreadsheet](https://docs.google.com/spreadsheets/d/1MjR3MC6kRoXZCfIKmrR-SAiZCXOYRsug6NzAoIhFm8Q/) to review, 1 from your own section, and 2 from other sections. - -For each team, you will submit a review of their feature(s). You will need to test the feature(s) as described in their UserGuide and provide feedback on the following: - -1. How was the experience of using the feature(s), would this be something you think would help enable better communication between faculty and students and why? -2. How do you think the feature can be improved? and/or What do you think the feature did well in? -3. Did you discover any bugs using the feature(s)? - -To qualify for extra credit, you will have to submit your review: - -- on Gradescope -- in the appropriate sheet in the [public spreadsheet](https://docs.google.com/spreadsheets/d/1MjR3MC6kRoXZCfIKmrR-SAiZCXOYRsug6NzAoIhFm8Q/). There should be one sheet per team, and you should add your review to the sheet for the team you are reviewing. - -## Grading -To receive full credit for the checkpoint, we expect: - -- [ ] A link to your successfully deployed web application for your team repository -- [ ] A list of N-1 different static and dynamic analysis tools, where N is the number of members on your team. This list must satisfy all the following criteria: - - Contain at least one static analysis tool - - Contain at least one dynamic analysis tool - - Contain at least one tool not on our starter list of tools -- [ ] Links to N-1 pull requests for each of the selected tools containing evidence of the tool being run at least once on your repository - -To receive full credit for the final deadline, we expect: - -- [ ] A link to your successfully run CD GitHub action that deploys the website while following proper GitHub practices in handling deployment secrets -- [ ] A design document describing your research into each of the potential tools, justification for your selection of integrated tool(s), and your final integration plan -- [ ] A link to a succuessful run of a GitHub Action that demonstrates your integration of your selected tool(s) into your team workflow - -To receive full credit for the extra credit, we expect: - -- [ ] Your review of three different teams' features on Gradescope and on the [public spreadsheet](https://docs.google.com/spreadsheets/d/1MjR3MC6kRoXZCfIKmrR-SAiZCXOYRsug6NzAoIhFm8Q/), addressing the three questions described. -## Resources & Documentation - -### Starter List of Tools - -NodeBB is built in Javascript/Typescript using Node.js and uses Benchpress for its frontend templating. Below are non-exhaustive lists of analysis tools that are available. - -For other resources, [Awesome Static Analysis page](https://github.com/david-a-wheeler/awesome-static-analysis) and [Awesome Dynamic Analysis page](https://github.com/analysis-tools-dev/dynamic-analysis) have extensive listings of available static and dynamic analysis tools for a pretty hefty list of programming languages. - -Some of the tools already have existing GitHub Actions workflows on GitHub Marketplace; use your Googling skills, and see what you find! - -#### Static Tools - -- [flow](https://flow.org/): Static type checker for JavaScript -- [JScent](https://github.com/moskirathe/JScent): Program analyzer for detecting “code smells” -- [JSHint](https://jshint.com/docs/): Used to flag suspicious usage in JavaScript programs -- [StandardJS](https://standardjs.com/)/[ts-standard](https://github.com/standard/ts-standard): Static analysis tool for code quality within JavaScript/TypeScript projects -- [Retire.js](https://retirejs.github.io/retire.js/): Finds library/node module vulnerabilities within your project - -#### Dynamic Tools - -- [Iroh](https://maierfelix.github.io/Iroh/): Runtime code tracking and visualization -- [Jalangi](https://github.com/Samsung/jalangi2): Framework for dynamic analyses in JavaScript -- [Fast-Fuzz](https://www.npmjs.com/package/fast-fuzz): Fuzzing framework for TypeScript -- [Stryker Mutator](https://stryker-mutator.io/): Mutation testing tool for JavaScript