Skip to content

Commit

Permalink
feat(identity): verified merged (#364)
Browse files Browse the repository at this point in the history
* feat: add sequelize and pg libraries

* feat: add sequelize migrations configuration

* feat: add sequelize models for identity

* feat: add database migrations to create tables

* feat: init database connection before starting server

* refactor: capitalize enum values and change to admin or user

* fix: use correct key in db config for username

* feat: add local dev postgres through docker compose

* refactor: remove repo_name column

* feat: add unique index on email

* feat: add SiteService

* feat: add AuthService

* feat: add middleware to inject site access token

* refactor: change variable casing of imported services

* feat: add scaffold for AuthService otp methods

* feat: add UserService

* feat: extend auth middleware to check for isomer user id

* feat: add routes for send and verify otp

* feat: implement actual totp methods

* feat: implement check for whether otp can be sent

* feat: add validator for email format

* feat: implement mail client using postman transactional api

* feat: implement TokenStore

* refactor: initialize services from module index

* fix: use correct expiry value

* feat: add github_id column to user model

* feat: allow email to be null

* feat: create user with github id

* feat: apply verifyJwt for otp routes

* feat: add methods for update and find by github id

* feat: whitelist only by domain

* feat: return email in whoami response

* feat: add verifyOtp endpoint

* refactor: rename to email otp methods

* refactor: move otp routes to /user

* refactor: move identity services to module

* refactor: move email verification methods to user service

* feat: add verification for contact number

* feat: use bad request error

* chore: upgrade sequelize

* refactor: use router pattern for users routes

* fix: undefined reference to usersService

* chore: add sms cred name to example env file

* feat: track last logged in date time

* feat: update err message for non-govt emails

* build(pcakage-lock): update build deps

this commit is from running npm i - there should be no breakage and should be safe to rollback.

* feat(identity): add types (#367)

* build(src): add tsconfig and deletes jsconfig

* build(package): install ts in dev deps

* feat(eslint): adds config for ts

* feat(eslint): adds config for ts

* refactor(authservice): add types to auth service

* refactor(mailclient): add types

* chore(package-lock): remove trailing comma

* build(package): installed ts import resolver

* chore(eslint): update eslint to recognise imports properly

* refactor(smsclient): add types for smsclient

* refactor(tokenstore): add ts

* refactor(totpgenerator): add ts

* chore(eslint/tsconfig): update to recognise @database paths

* chore(services/identity): changed to new imports for TS

* build(tsconfig): changed module to cjs and changed to import statements

* build(package): removed sequelize cli

* feat(identity): unit tests for services (#369)

* build(package/): installed jest-mock-axios

* build(package): installed ts-jest and removed jest config in package.json

* test(jest.config,.js): added jest config

this resolves ts files so that jest can handle them and ts-jest can provide proper typings. the
module name mapping is relocated here

* test(authservice): add spec

* refactor(mailclient): changed mailclient so that initialization fails if api key is empty

* test(mailclient): add tests and axios mock

* refactor(constants): add constants for tests

* refactor(smsclient): change api key retrieval to be done at constructor to allow for tesing

* test(smsclient): add test

* test(tokenstore): added tests

* build(paths): add mocks to path

* chore(tokenstore): update naming for clarity

* chore(totpgenerator): changed access values

* test(totpgenerator): add tests

* refactor(mailclient): changed mailclient initialization for better readability

* refactor: use sequelize-typescript (#374)

* build(deps): add sequelize-typescript and required deps

As described in https://www.npmjs.com/package/sequelize-typescript

* chore(tsconfig): add required flags

Also specified in sequelize-typescript docs

* refactor: convert config file to ts

* refactor: convert db model files into ts

* refactor: use sequelize-ts in server.js

* build(package): installed ts-node-dev

* build(tsconfig): updates tsconfig with ts-node options

we forc ets node to require tsconfig/paths to allow for path aliasing. this is because typescript
requires that the runtime (node/ts-node) does the path aliasing, rather than ts. hence, requiring
tsconfig/paths allows us to use path alias at runtime.

additionally, we add a transpileOnly option. This prevents ts-node from typechecking and transpiles to js only, which increases its transpilation speed.

another option, swc, was also added. this option uses the rust swc compiler for ts, which is significantly faster than the other compilers. this should speed up transpilation speed.

* fix(server): fixed import of logger and sequelize

sequelize (the class) is not a default export and has been changed accordingly. also, logger has
been changed from morgan to winston (the one in our package) as morgan is less powerful than winston

* chore(loaders/www): removed extra loader in www as we now perform it in server

* chore(logger): fixed exports

* fix(pcakage): update build scripts to use ts-node for start

* build(package): install swc transpiler

* revert(database/config): rollback to .js to avoid migration having issues

* feat(identity): add types to last 2 services (#376)

* refactor(server): separated out sequelize from server.js

* chore(services/identity): update to use esm export

chore(tokenstore): use esm export

* refactor(userservice): convert to ts

refactor(userservice): update to ts

* refactor(sitesservice): use ts

* refactor(identity): migrate index to ts

* refactor(server): use new export from identity

* chore(totpgenerator): relaxed typing

* refactor(identity): add comments and rename function

* fix(databse/index): add typecast

* fix(database/config): updated exports

* fix(totpgenerator): made expiry required

* test(identity): add unit tests for user/sites service (#378)

* refactor(server): separated out sequelize from server.js

* build(package): removed sequelize cli

* build(package/): installed jest-mock-axios

* build(package): installed ts-jest and removed jest config in package.json

* test(jest.config,.js): added jest config

this resolves ts files so that jest can handle them and ts-jest can provide proper typings. the
module name mapping is relocated here

* refactor(usersservice): refactor to take in instance of sequelize for ease of testing

* refactor(database): defer sequelization initialization to caller

this allows us to pass in base models at run time rather than statically deciding at compile time.
this is useful when we want to test services that require a db but we wish to mock the db

* test(usersservice): add test

* refactor(sitesservice): change types; add non-null assertion (TO BE REMOVED LATER)

* refactor(sitesservice): propagate null value upwards

* fix(auth): set accessToken only if siteAccessToken exists

* test(sitesservice): add tests

* build(package): installed missing ts-node-dev dependency

* chore(database): allow additional dbconfig for testing

* chore(sequelizerc): removed extra .js

* fix(usersservice): removed extra dot in default whitelist

* feat(identity): add types to user router (#380)

* build(types): installed express and jest typings

* feat(types): add more types

* refactor(req): changed setting userId on req to res.locals

* build(package): installed type-fest

* fix(usersservice): update typing for updateUserByGitHubId

* refactor(users): shift to ts

* refactor(users): fixed typing

Co-authored-by: seaerchin <[email protected]>
Co-authored-by: seaerchin <[email protected]>

Co-authored-by: Preston Lim <[email protected]>

* test(identity): integration tests for user routes (#382)

* build(package): installed sqlite3 and types

* chore(users): removed extra await

* build(build deps): installed umzug (query interface for sequelize)

* chore(package): update test command to use test env

* feat(docker-compose): update to have another test db

* test(jest.config.js): setup global setup/teardown files for integraiton

* chore(config): add path aliases

* build(tests): add setup/teardown scripts for integration

* chore(tests): add logs for setup/teardown

* test(users): add integration tests!!!!!!

* test(usersservice): update test to fit new api

* chore(.env.test): add test env file

* fix(services/identity/index): fixed typing of mockMailer

* chore(constants): shift test constants to tests folder

* chore(constants): add new bearertokenheader

* chore(testss): miinor test fixes

* fix(users.spec): remove explicit mocking for otplib in integration testing

* chore(users): remove extra console log

* chore(users.spec): shift into integration

* Refactor/move identity to v2 endpoints (#368)

* Fix/retrieve last updated time (#354)

* Fix: use pushed_at instead of updated_at

* Fix: getLastUpdated endpoint

* Fix: prepend unique prefix for release script (#370)

* 0.1.0

* Refactor/auth (#328)

* Feat: Add Auth service

* Feat: add auth router

* Feat: add auth endpoints

* Feat: convert auth middleware to use dependency injection

* feat: add tests for new auth router and service

* Fix: add updated auth router endpoints to auth middleware

* Refactor: verifyJwt into separate helper method

* Nit: add exports for constants

* Nit: add helper function for secure and comments for time

* Style: early return

* Style: test styling

* Nit: adjust comments for importing error

* Nit: adjust unknown route comment

* Nit: return early

* Nit: link issue

* Refactor: move isSecure to utils

* Style: early return

* Nit: remove irrelevant comment

* Refactor/separate auth initialisation (#377)

* Chore: remove unused index router

* Refactor: initialise middleware outside of server.js

* Refactor: move unknown route check to server.js

* Refactor: import auth middleware directly in routers

* Chore: remove noverify method

* Refactor/sites (#341)

* Feat: add sitesService

* Feat: add sites router

* Feat: add site endpoints

* Tests: add tests

* Fix: use pushed_at instead of updated_at

* Style: add spacing to tests

* Style: object destructuring

* Style: return type and map

* Fix: return last updated time instead of string representing time and update tests

* Feat: throw 404 error if no staging url found

* Style: destructuring response

* Feat: add access check in github service

* Fix: tests

* Fix: test comment

* Rebase: use auth verify middleware in sites router

* Fix: test for config priority

* Fix: use originalUrl instead of url (#388)

* fix: order of ping (#392)

* 0.2.0

* chore(build): add mergify workflow (#385)

* chore(add mergify): add mergify workflow

* chore(mergify): remove extra build step

* Fix: use existing githubservice instead of axiosinstance to check for user access

* Refactor: move new logic from auth route into authService

* refactor: shift auth middleware methods to authMiddlewareService

* Fix: await method

* Rebase: inject site token middleware into routers

This commit also adds a path prefix for each router when using the middleware - I realised that we were previously hitting verifyJwt multiple times because a request would sequentially hit all routers until it reached the correct one - given that there was no check on the verifyJwt use in each router, it was called repeatedly

* Fix: add auth middleware to users router

* Fix: auth route and service tests

* Feat: use dependency injection for middleware and identity services

* Refactor: move v2 endpoints into subrouters

* Refactor: use subrouters for v1 endpoints

* Fix: tests

* Fix: move identity constants to fixtures

* Fix: test folder name

* Nit: variable and function name

* Nit: add comments

* Chore: store repeated params in variable

* Nit: test name

* Nit: spacing

* Nit: separate condition for readability

* Chore: remove unnecessary error throwing

* Chore: add comment on error transformation

* Chore: add log statements for errors

* Fix: set userId in res.locals instead of in req

* Nit: add log statement for error

* Nit: add proper error message for not logged in error

* Fix: rebase errors

* Chore: update comment

* Chore: update error message

* Fix: router path

* Chore: update comment

* Chore: update error messages

* Fix: log error instead of providing information in error

Co-authored-by: seaerchin <[email protected]>

* Fix: merge errors

* Fix: modify sequelizerc file to use config.js (#406)

* feat(identity): infrastructure changes (#405)

* chore(www.js): rename www -> www.js for ts to transpile it

* build(package): changed start commands and added a build script

the start command is retrieved from:
https://stackoverflow.com/questions/61342753/paths-from-tsconfig-json-doesnt-work-after-tsc

this is required because ts doesn't destructure the path aliases into the actual imports.

* build(tsocnfig): updated tsconfig to emit

this won't affect npm run dev as we are using ts-node and it will transpile on the fly

* ci(ci): add step to build

* chore(gitignore): ignore build drivre

* fix(tsconfig): remove swc :(

swc transpiles experimental flags (metadata/decorators) wrongly; this results in
sequelize-typescript being transpiled wrongly

* chore(authmiddlewareservice): remove logging of cookies

* Fix: set reply_to param for OTP email (#408)

* Fix: set reply_to param for OTP email

* Chore: use support@isomer instead

* Chore: update email to noreply

* Fix: ping endpoint order (#407)

Co-authored-by: Lam Kee Wei <[email protected]>
Co-authored-by: Preston Lim <[email protected]>
Co-authored-by: Alexander Lee <[email protected]>
  • Loading branch information
4 people authored Mar 30, 2022
1 parent ebbeb8b commit ae602b9
Show file tree
Hide file tree
Showing 105 changed files with 7,281 additions and 13,800 deletions.
21 changes: 17 additions & 4 deletions .env-example
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,6 @@ export GITHUB_ORG_NAME="isomerpages"
export GITHUB_BUILD_ORG_NAME="opengovsg"
export GITHUB_BUILD_REPO_NAME="isomer-build"
export MUTEX_TABLE_NAME=""
export E2E_TEST_REPO="e2e-test-repo"
export E2E_TEST_SECRET="blahblahblah"
export E2E_TEST_GH_TOKEN=""

# Required to connect to DynamoDB
export AWS_ACCESS_KEY_ID=""
Expand All @@ -22,4 +19,20 @@ export AWS_SECRET_ACCESS_KEY=""
# Required to run end-to-end tests
export E2E_TEST_REPO="e2e-test-repo"
export E2E_TEST_SECRET=""
export E2E_TEST_GH_TOKEN=""
export E2E_TEST_GH_TOKEN=""

# Database
export DB_URI="postgres://isomer:password@localhost:15432/isomercms_dev"
export DB_MIN_POOL="1"
export DB_MAX_POOL="10"
export DB_ENABLE_LOGGING=""

export LOCAL_SITE_ACCESS_TOKEN=""
export OTP_SECRET="dummysecret"
export DOMAIN_WHITELIST=".gov.sg"

# Email
export POSTMAN_API_KEY=""

# SMS
export POSTMAN_SMS_CRED_NAME=""
41 changes: 32 additions & 9 deletions .eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,18 @@
},
"extends": [
"airbnb-base",
"prettier"
"prettier",
"plugin:import/typescript"
],
"plugins": ["only-warn", "import"],
"parserOptions": {
"ecmaVersion": 12
},
"rules": {
"no-underscore-dangle": "off",
"class-methods-use-this": "off",
"import/no-unresolved": "error",
"import/extensions": ["warn", "never"],
"import/order": [
"error",
{
Expand All @@ -24,8 +28,7 @@
},
"newlines-between": "always",
"groups": ["builtin", "external", "internal", "parent", "sibling", "index", "object"],
"pathGroups": [
{
"pathGroups": [{
"pattern": "@logger/**",
"group": "internal",
"position": "before"
Expand Down Expand Up @@ -56,31 +59,51 @@
"position": "before"
},
{
"pattern": "@validators/**",
"group": "internal",
"position": "before"
"pattern": "@validators/**",
"group": "internal",
"position": "before"
}
]
}
]
},
"overrides": [
{
"files": ["*.ts", "*.tsx"],
"parser": "@typescript-eslint/parser",
"extends": ["plugin:@typescript-eslint/recommended"]
}
],
"settings": {
"import/parsers": {
"@typescript-eslint/parser": [".ts", ".tsx"]
},
"import/resolver": {
"alias": [
"typescript": {
// always try to resolve types under `<root>@types` directory even it doesn't contain any source code, like `@types/unist`
"alwaysTryTypes": true
},
"node": {
"extensions": [".js", ".jsx", ".ts", ".tsx"]
},
"alias": [
["@root", "./"],
["@classes", "./classes"],
["@errors", "./errors"],
["@logger", "./logger"],
["@middleware", "./middleware"],
["@routes", "./routes"],
["@utils", "./utils"],
["@loaders", "./loaders"],
["@database", "./database"],
["@services", "./services"],
["@validators", "./validators"],
["@fixtures", "./fixtures"]
["@fixtures", "./fixtures"],
["@database", "./database"]
]
}
},
"globals": {
"Base64": true
}
}
}
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ jobs:
${{ runner.OS }}-
- name: Install NPM
run: npm ci
- name: Build application
run: npm run build
- name: Zip application
run: zip -r "deploy.zip" * .platform -x .env-example .gitignore package-lock.json
- name: Get timestamp
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.env
node_modules/
.vscode/
.vscode/
build/
8 changes: 8 additions & 0 deletions .sequelizerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
const path = require("path")

module.exports = {
config: path.resolve("./database", "config.js"),
"models-path": path.resolve("./database", "models"),
"seeders-path": path.resolve("./database", "seeders"),
"migrations-path": path.resolve("./database", "migrations"),
}
15 changes: 15 additions & 0 deletions __mocks__/@aws-sdk/client-secrets-manager.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// This is a manual module mock for the AWS client.
// This is done to prevent our internal services from pinging the actual AWS ones
// and to ensure our tests are 1. consistent 2. reliable.

export const mockSend = jest.fn()

export const secretsManagerClient = {
send: mockSend,
}

export const SecretsManagerClient: jest.Mock<
typeof secretsManagerClient
> = jest.fn().mockImplementation(() => secretsManagerClient)

export const GetSecretValueCommand = jest.fn((secret) => ({ SecretId: secret }))
3 changes: 3 additions & 0 deletions __mocks__/axios.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
import mockAxios from "jest-mock-axios"

export default mockAxios
6 changes: 6 additions & 0 deletions __mocks__/otplib.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// eslint-disable-next-line import/prefer-default-export
export const totp = {
clone: jest.fn().mockReturnThis(),
generate: jest.fn(),
verify: jest.fn(),
}
94 changes: 0 additions & 94 deletions bin/www

This file was deleted.

91 changes: 91 additions & 0 deletions bin/www.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/usr/bin/env node

/**
* Module dependencies.
*/
require("module-alias/register")

const http = require("http")

const app = require("@root/server")

const debug = require("debug")("isomercms:server")

// Import logger
const logger = require("@logger/logger")

/**
* Normalize a port into a number, string, or false.
*/

function normalizePort(val) {
const port = parseInt(val, 10)

if (isNaN(port)) {
// named pipe
return val
}

if (port >= 0) {
// port number
return port
}

return false
}

/**
* Event listener for HTTP server "error" event.
*/

function onError(error) {
if (error.syscall !== "listen") {
throw error
}

const bind = typeof port === "string" ? `Pipe ${port}` : `Port ${port}`

// handle specific listen errors with friendly messages
switch (error.code) {
case "EACCES":
logger.error(`${bind} requires elevated privileges`)
process.exit(1)
case "EADDRINUSE":
logger.error(`${bind} is already in use`)
process.exit(1)
default:
throw error
}
}

/**
* Event listener for HTTP server "listening" event.
*/

function onListening() {
const addr = server.address()
const bind = typeof addr === "string" ? `pipe ${addr}` : `port ${addr.port}`
debug(`Listening on ${bind}`)
logger.info(`isomerCMS app listening on port ${port}`)
}

/**
* Get port from environment and store in Express.
*/

const port = normalizePort(process.env.PORT || "8081")
app.set("port", port)

/**
* Create HTTP server.
*/

const server = http.createServer(app)

/**
* Listen on provided port, on all network interfaces.
*/

server.listen(port)
server.on("error", onError)
server.on("listening", onListening)
Loading

0 comments on commit ae602b9

Please sign in to comment.