PRMT-4568 - ehr-repository - Remove PostgreSQL Implementation and Seq…

…uelize Dependencies (#73)

* [PRMT-4565] store a working version of dynamodb localstack setup.

* [PRMT-4565] add basic test for dynamodb client

* [PRMT-4565] minor change in config, small change to generate_secure_string to address a mac specific problem

* [PRMT-4565] Read dynamodb table name from SSM, change the table name for local testing

* [PRMDR-4565] Replace localstack with dynamodb-local for running dynamodb integration test

* [PRMT-4565] Change timestamp format to include timezone offset

* [PRMT-4566] add new version of updateFragmentAndCreateItsParts, start separating model and repository layer

* [PRMT-4566] Config test setup and teardown

* [PRMT-4566] Add option to query specific record type

* [PRMT-4566] Add state enums

* [PRMT-4566] separate client and repository layer

* [PRMT-4566] Reorganise new codes

* [PRMT-4566] Add terraform iam policy for accessing the ehr-transfer-tracker table

* [PRMT-4566] Reorganise code, use CreatedAt instead of UpdatedAt to determine latest record

* [PRMT-4566] Migrate all tests in message-repository.integration.test.js to new dynamodb-based implementation

* [PRMT-4566] Rename test files

* [PRMT-4566] Continue migrating database methods and integration tests. Store current progress

* [PRMT-4566] Migrating integrated test for more methods

* PRMT-4566 reorganise code

* [PRMT-4566] Migrated all database-related integration tests. Store current progress

* [PRMT-4566] Rename tests files

* [PRMT-4566] Rename methods to avoid import conflict, add deprecated mark and ticket number to old postgres related codes

* [PRMT-4566] Rename methods

* [PRMT-4566] minor change in wordings

* [PRMT-4566] Correct dynamodb related env vars

* [PRMT-4566] Change DeletedAt field to store ttl time of 8 weeks later

* [PRMT-4567] Migrated store-message-controller.js and its tests

* [PRMT-4567] migrated message-location-controller.js and its tests

* [PRMT-4567] migrated app integration tests for POST /messages endpoint

* [PRMT-4567] rename test files

* [PRMT-4567] Migrate get-fragment-controller.js and integration tests for /fragments/${conversationId}/${fragmentMessageId} endpoint

* [PRMT-4567] migrated endpoint `GET /patients/:nhsNumber` and controller at `src/api/patients/patient-details-controller.js`

* [PRMT-4567] Migrate controller `src/api/patients/health-record-controller.js` and endpoint `GET /patients/:nhsNumber/health-records/:conversationId`

* [PRMT-4567] minor clean up

* [PRMT-4567] migrated controller delete-ehr-controller.js

* [PRMT-4567] Add integration test for endpoint DELETE /patients/:nhsNumber

* [PRMT-4567] Remove postgres db checking from health check endpoint

* [PRMR-4567] Unplug postgres db from application, comment out redundant tests

* [PRMT-4567] mark redundant tests as deprecated, fix unit tests

* [PRMT-4567] Update dtest.yml, remove postgresdb healthcheck from docker test

* [PRMT-4567] comment out postgresdb migration command in ./tasks

* [PRMT-4567] minor bugfix

* [PRMT-4567] Update README for the change related to database and integration tests

* [PRMR-4567] improve configs about running integration test with dojo

* [PRMT-4567] Fix test related to DeletedAt timestamp

* [PRMT-4567] Amend Layer sort key prefix to be all capital case

* [PRMT-4567] use the string "STORED_IN_REPOSITORY" for complete status of CORE and FRAGMENT

* [PRMT-4567] Fixed an issue related to DeletedAt time and DST

* [PRMT-4568] Remove postgres-db related code

* [PRMT-4568] Continue removing postgres-db related code

* [PRMT-4568] Remove redundant comments

* [PRMT-4568] Fix broken import of deprecated code

* [PRMT-4568] Remove postgres related packages from dependencies

* [PRMT-4568] Remove postgresdb from docker config, remove .sequelizerc

* [PRMT-4568] Continue removing postgres-db related stuffs

* [PRMT-4568] Remove postgres db related content from ./tasks (except terraform things)

* [PRMT-4568] Run linter and formatter

* [PRMT-4568] Remove commented out code

* [PRMT-4568] remove sequelize from package.json

* [PRMT-4568] Fix Dockerfile (remove references to deleted file)

* [PRMT-4568] remove db:migrate from package.json

* [PRMT-4568] Use underscore instead of hyphen in terraform code

* [PRMT-4568] Remove trailing comma

* [PRMT-4568] Fix typo

* [PRMT-4568] fix typo (Outbound --> OUTBOUND) at TransferStatus

* [PRMT-4568] Adapt to database schema change: sort key for core change from `CORE#{messageId}` to just `CORE`

* [PRMT-4568] address PR comments

.prettierrc

@@ -1,4 +1,5 @@
 {
   "singleQuote": true,
-  "printWidth": 100
+  "printWidth": 100,
+  "trailingComma": "none"
 }

Dockerfile

@@ -1,8 +1,8 @@
 FROM node:16.19.0-alpine AS builder
 
-# install python and postgres native requirements
+# install python native requirements
 RUN apk update && \
-    apk add --no-cache bash tini postgresql-client && \
+    apk add --no-cache bash tini && \
     rm -rf /var/cache/apk/*
 
 RUN apk add --no-cache \
@@ -14,7 +14,7 @@ RUN apk add --no-cache \
     && rm -rf /var/cache/apk/*
 
 # Install sequelize postgress native dependencies
-RUN apk add --no-cache postgresql-dev g++ make
+RUN apk add --no-cache g++ make
 
 COPY package*.json /app/
 
@@ -31,10 +31,10 @@ COPY --from=builder /usr/local/bin/node /usr/local/bin
 # take native-install node modules
 COPY --from=builder /app /app
 
-# install python and postgres native requirements (again, as per builder)
+# install python native requirements (again, as per builder)
 # add root CA from deductions team to trusted certificates
 RUN apk update && \
-    apk add --no-cache openssl ca-certificates bash tini postgresql-client && \
+    apk add --no-cache openssl ca-certificates bash tini && \
     rm -rf /var/cache/apk/*
 
 RUN apk add --no-cache \
@@ -46,9 +46,6 @@ RUN apk add --no-cache \
     && rm -rf /var/cache/apk/*
 
 COPY build/                   /app/build
-COPY database/                /app/database
-COPY build/config/database.js /app/src/config/
-COPY .sequelizerc             /app/
 
 COPY scripts/load-api-keys.sh      /app/scripts/load-api-keys.sh
 COPY scripts/run-server-with-db.sh /usr/bin/run-ehr-server

README.md

@@ -10,11 +10,6 @@ The stored the EHR is composed of the messages sent from its previous holder in
 - [Docker](https://docs.docker.com/install/)
 - [kudulab/dojo](https://github.com/kudulab/dojo#installation)
 
-In order to run npm install locally on your host (outside of dojo), you'll need to install postgresql:
-```
-brew install postgresql
-```
-
 ### AWS helpers
 
 This repository imports shared AWS helpers from [prm-deductions-support-infra](https://github.com/nhsconnect/prm-deductions-support-infra/).
@@ -31,45 +26,37 @@ If you would like to run the app locally outside `dojo`, you need to:
 ```
 - `NHS_ENVIRONMENT` - should be set to current environment in which the container is deployed. The name must also exist in the `database.json` file.
 - `S3_BUCKET_NAME` - the name of S3 bucket to store the EHR fragments in.
-- `DATABASE_USER` - username for the database
-- `DATABASE_PASSWORD` - password to the database
-- `DATABASE_NAME` - name of the database on server.
-- `DATABASE_HOST` - database server hostname to connect with.
-- `LOCALSTACK_URL` - (Test) the location of localstack, only used for tests
+- `LOCALSTACK_URL` - (Test) the location of localstack, only used for s3 related tests
+- `DYNAMODB_NAME` - The table name of the dynamodb table (ehr-transfer-tracker) used
 ```
 
 ## Running the tests
 
-Run the unit tests with
-
-by entering the `dojo` container and running `./tasks _test_unit`
+Run the unit tests by entering the `dojo` container and running `./tasks _test_unit`
 or on your machine with `npm run test:unit`
 
 Run the integration tests within a Dojo container
 
-1. Run `dojo -c Dojofile-itest` which will spin up the testing container
+1. Run `./tasks test_integration_shell` which will spin up the testing container
 2. Run `./tasks _test_integration`
 
-You can also run them with `npm run test:integration` but that will require some additional manual set-up:
+You can also run them with `./tasks test_integration` from out of dojo.
 
-```bash
-# Brings up the local test environment
-docker-compose up &
-
-# Alternative with node-dojo (interactive)
-# Requires changes to Environment Variables:
-#   DATABASE_HOST=db
-#   LOCALSTACK_URL=http://localstack:4572
-dojo -c Dojofile-itest
+You can also run each individual integration test separately in an IDE (assuming IntelliJ),
+but that will require some additional manual set-up:
 
-npm run test-local
-
-# This is equivalent of:
-sequelize-cli db:migrate    # Runs the migration
+```bash
+# Config env var, spin up docker containers and enter interactive dojo environment
+./tasks test_integration_shell
 
-npm test
+# If things work as expected your prompts should looks like `dojo@xxxx(node-dojo):/dojo/work$`
+# inside dojo, run the below script to create a dynamodb table for integration test
+scripts/create-dynamodb-table.sh
 
-sequelize-cli db:migrate:undo:all # Undoes the migration to leave clean env
+# The above script will create a test table in dynamodb-local docker image. 
+# The dynamodb-local is accessible at endpoint http://dynamodb-local:8000 within docker, 
+# or at endpoint http://localhost:4573 from out of docker.
+# This should allow you to run or debug db-related integration tests from Intellij's play button. 
 ```
 
 ## Run the coverage tests (unit test and integration test)

docker-compose-dtest.yml

@@ -6,17 +6,10 @@ services:
   ehr-repo:
     image: ${REPOSITORY_URI}:${IMAGE_TAG}
     links:
-      - db:db
       - localstack:localstack
+      - dynamodb-local:dynamodb-local
     ports:
     - 3000:3000
-  db:
-    image: postgres:12.1
-    environment:
-      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
-      POSTGRES_USER: ${DATABASE_USER}
-      POSTGRES_DB: ${DATABASE_NAME}
-      LC_ALL: C
   localstack:
     image: localstack/localstack:0.10.9
     environment:
@@ -31,3 +24,8 @@ services:
     volumes:
       - ./scripts/create-bucket.sh:/docker-entrypoint-initaws.d/create-bucket.sh
       - /var/run/docker.sock:/var/run/docker.sock
+  dynamodb-local:
+    image: amazon/dynamodb-local
+    command: "-jar DynamoDBLocal.jar -sharedDb -inMemory"
+    ports:
+      - "4573:8000"

docker-compose-itest.yml

@@ -2,19 +2,10 @@ version: '2'
 services:
   default:
     links:
-      - db:db
       - localstack:localstack
+      - dynamodb-local:dynamodb-local
     ports:
       - "3000:3000"
-  db:
-    image: postgres:12.1
-    environment:
-      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
-      POSTGRES_USER: ${DATABASE_USER}
-      POSTGRES_DB: ${DATABASE_NAME}
-      LC_ALL: C
-    ports:
-      - "5432:5432"
   localstack:
     image: localstack/localstack:0.10.9
     environment:
@@ -27,4 +18,10 @@ services:
       - ./scripts/create-bucket.sh:/docker-entrypoint-initaws.d/create-bucket.sh
       - /var/run/docker.sock:/var/run/docker.sock
     logging:
-      driver: none
+      driver: none
+  dynamodb-local:
+    image: amazon/dynamodb-local
+    command: "-jar DynamoDBLocal.jar -sharedDb -inMemory"
+    ports:
+     - "4573:8000"
+

docker-compose.yml

@@ -1,16 +1,5 @@
 version: '3.4'
 services:
-  postgres:
-    image: postgres:12.1
-    ports:
-      - 5432:5432
-    expose:
-      - 5432
-    environment:
-      POSTGRES_USER: ${DATABASE_USER}
-      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
-      POSTGRES_DB: ${DATABASE_NAME}
-      LC_ALL: C
   localstack:
     image: localstack/localstack
     ports: