Wire up a database #8

david-mears-2 · 2024-07-25T17:21:24Z

Adds scripts for running a postgres db in a docker container, and Prisma for managing migrations and ORM.

Go to localhost:3000/scenarios to check if the app can talk to the database. As you can see below, the nuxt devtools can also show you what data was sent along with the rendered page - proving that the call to the api endpoint '/api/scenarios' happened server-side.

I'll do tests when we have something real to test - the api endpoint and page in this PR are only dummy ones to prove that the wiring up worked.

TODO:

Figure out how migrations get run on CI envs and applying new migrations to existing envs (prod, dev, etc)

codecov · 2024-07-26T10:16:55Z

Codecov Report

Attention: Patch coverage is 15.00000% with 17 lines in your changes missing coverage. Please review.

Project coverage is 50.62%. Comparing base (35eac62) to head (ec4dfd5).

Files	Patch %	Lines
lib/prisma.ts	0.00%	9 Missing and 1 partial ⚠️
server/api/scenarios.ts	0.00%	6 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #8      +/-   ##
==========================================
- Coverage   58.93%   50.62%   -8.31%     
==========================================
  Files          13       16       +3     
  Lines         599      401     -198     
  Branches       23       25       +2     
==========================================
- Hits          353      203     -150     
+ Misses        242      192      -50     
- Partials        4        6       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

EmmaLRussell

Works for me, but I think we really need a single-liner script to run and migrate the db and spin up other dependencies - the README is a little unclear about what you need to do just to run the current branch (and in what order) vs upating and generating new migrations.

Other than that, mostly questions!

I'm not really sure where Anmol got all his pg config from for packit - maybe chat to him about it when he gets back!

I was expecting to see a model Typescript type scenario generated by the ORM as part of this branch, in addition to the prisma schema and the migrations. How are the queries to prisma made type safe? Is it all wrapped up in the prisma client that ends up in node_modules somehow?

EmmaLRussell · 2024-07-26T16:32:55Z

README.md

+Copy `.env.example` to `.env`.
+Build and run the database container:
+
+```bash
+./db/scripts/build
+./db/scripts/run
+```


Since you're going to have to do this a lot during development, I think it would be better to make this a single script call - you might as well put that in a run-dependencies now since you know you're going to have to include the api soon too. It should apply the db migrations you need to.

Do we need to distinguish .env from .env.example here? Can't we just make .env the settings that will work for local development and then in the deployment tool we'll make sure we overwrite with whatever settings we need for the environments we're deploying?

To streamline setting up .env, in the next PR I'm updating the README to make copying and npm installing a single action:

Copy .env.example to .env and install the JS dependencies:

cp .env.example .env npm install

EmmaLRussell · 2024-07-28T09:32:03Z

README.md


 ```bash
 npm install
 ```

-## Local development
+Prisma ORM can only query the database once you 'generate' the Prisma Client, which generates into `node_modules/.prisma/client`. This should happen when you install the JS dependencies and whenever you run a migration, but if the Prisma client gets out of sync or doesn't generate, you can manually generate it:


..and this generates the client based on schema.prisma?

Updating the README to make this clear

EmmaLRussell · 2024-07-28T09:35:51Z

README.md

+### DB
+
+To create migrations to the database, first update the Prisma schema at ./prisma/schema.prisma as required, then run the below to generate the corresponding SQL migration and to apply it to the database:
+
+```bash
+npx prisma migrate dev
+```
+
+The same command is also used to apply migrations that already exist in ./prisma/migrations but which have not been applied to the database.


OK, so we're going to keep this as a manual development process? When you need to update the schema, you'll update both the .schema file, and generate the SQL migrations, and commit both to git?

..and as well as generating the migrations this command also applies them to the running database (defined in .env) - running in the container. So we'll run ths same thing as part of deployment (?) to update the db?

Updating README to add "You should commit both the Prisma schema and the migration file to Git."

Will look into the precise process for how migrations are done in deployment, but probably it's a GitHub Action that runs the command prisma migrate deploy (as opposed to the dev-specific command above).

Yeah, we just need to run prisma migrate deploy in a github action, and remember to set the DATABASE_URL env var (which we can do in github per-environment).

https://www.prisma.io/docs/orm/prisma-client/deployment/deploy-database-changes-with-prisma-migrate

I think it makes most sense to try that out when we have some environments up, with databases running, which we can see whether or not they correctly get migrated.

Note to self for when I'm working on this:

It might be the case that we need to explicitly re-generate the prisma client (using prisma generate) after running prisma migrate deploy, since that command "does not reset the database or generate artifacts (such as Prisma Client)"

I now have a 'Add prisma migrations to deploy workflow' ticket.

EmmaLRussell · 2024-07-28T09:42:22Z

db/README.md

+Run an SQL query:
+
+```
+./db/scripts/runQuery "SELECT id FROM scenario"


EmmaLRussell · 2024-07-28T09:45:51Z

db/scripts/run

+
+docker run --rm -d \
+ --network=bridge \
+ --name daedalus-web-app-db \


daedalus-web-db might save a few characters of pain every time you need to tye this! Or even daedalus-db if there isn't any danger of an R api db container (not sure what the plan is there though)...

Personally I'll find remembering to omit 'app' more confusing, since I'm generically calling this app 'daedalus-web-app' everywhere, e.g. the github repo, package name. (Didn't want to fully commit to the provisional name 'Daedalus Explore'!)

There will be at least this many Daedalusy 'packages' (broadly defined) to keep track of, so explicit naming should help us steer clear of conflation:

web app

nginx for web app

r api

model

EmmaLRussell · 2024-07-28T20:38:07Z

lib/prisma.ts

+if (process.env.NODE_ENV !== "production")
+  globalThis.prismaGlobal = prisma;


So the Prisma Client is the thing that will be used to query the db? I'm not clear on why a distinction is made for production mode? It looks like in this case the client is expected to already exist?

I can't explain what the production condition is for. The origin of the code is (as commented at top of file) (a linted version of) https://www.prisma.io/docs/orm/more/help-and-troubleshooting/help-articles/prisma-nuxt-module#option-b-libprismats

david-mears-2 · 2024-07-30T11:06:31Z

Works for me, but I think we really need a single-liner script to run and migrate the db and spin up other dependencies - the README is a little unclear about what you need to do just to run the current branch (and in what order) vs upating and generating new migrations.

Yeah, let's address this in a follow-up PR.

Edit: #10 adds this

david-mears-2 · 2024-07-30T11:11:12Z

I was expecting to see a model Typescript type scenario generated by the ORM as part of this branch, in addition to the prisma schema and the migrations. How are the queries to prisma made type safe? Is it all wrapped up in the prisma client that ends up in node_modules somehow?

Yes, prisma client generates the typescript stuff! Here is an example that appeared as a tooltip in my IDE:

const scenario: {
    id: string;
    model_version: string;
    parameters: Prisma.JsonValue;
    parameters_hash: string;
    result_data: Prisma.JsonValue;
    result_data_completed_at: Date | null;
    created_at: Date;
    updated_at: Date;
} | null

I got a TS warning when trying to use a wrong column: "Object literal may only specify known properties, and 'email' does not exist in type"

Relevant docs: https://www.prisma.io/docs/orm/prisma-client/type-safety

david-mears-2 · 2024-08-06T14:05:28Z

I'm not really sure where Anmol got all his pg config from for packit - maybe chat to him about it when he gets back!

The git blame for the files I copied into /db/conf shows that it's exclusively Alex Hill's work I've copied, as far as pg config!

david-mears-2 · 2024-08-06T15:07:16Z

Ready for review again @EmmaLRussell - as noted in the PR description I won't add a test for the api endpoint since it's a dummy endpoint that we'll replace in the future.

EmmaLRussell

👍

david-mears-2 added 3 commits July 25, 2024 15:27

Update github org name

3bd0dc1

Update some dependencies

4d4b491

Add database

75e799c

david-mears-2 requested a review from EmmaLRussell July 25, 2024 17:24

Remove unused script

07c0a7a

Update npm packages

76cf4b2

EmmaLRussell reviewed Jul 28, 2024

View reviewed changes

david-mears-2 added 2 commits August 1, 2024 14:10

Clarify README

43d637e

Add commit hook that formats the schema.prisma file

ec4dfd5

david-mears-2 requested a review from EmmaLRussell August 6, 2024 15:07

EmmaLRussell approved these changes Aug 7, 2024

View reviewed changes

david-mears-2 merged commit 7159823 into main Aug 7, 2024
3 of 5 checks passed

david-mears-2 deleted the jidea-38-containerisation branch August 7, 2024 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wire up a database #8

Wire up a database #8

david-mears-2 commented Jul 25, 2024 •

edited

Loading

codecov bot commented Jul 26, 2024 •

edited

Loading

EmmaLRussell left a comment

EmmaLRussell Jul 26, 2024

david-mears-2 Jul 30, 2024 •

edited

Loading

EmmaLRussell Jul 28, 2024

david-mears-2 Jul 30, 2024

david-mears-2 Jul 30, 2024

EmmaLRussell Jul 28, 2024

david-mears-2 Jul 30, 2024

david-mears-2 Aug 1, 2024

david-mears-2 Aug 1, 2024 •

edited

Loading

david-mears-2 Aug 1, 2024

david-mears-2 Aug 1, 2024

EmmaLRussell Jul 28, 2024

EmmaLRussell Jul 28, 2024

david-mears-2 Jul 30, 2024 •

edited

Loading

EmmaLRussell Jul 28, 2024

david-mears-2 Jul 30, 2024 •

edited

Loading

david-mears-2 commented Jul 30, 2024 •

edited

Loading

david-mears-2 commented Jul 30, 2024 •

edited

Loading

david-mears-2 commented Aug 6, 2024

david-mears-2 commented Aug 6, 2024

EmmaLRussell left a comment

		if (process.env.NODE_ENV !== "production")
		globalThis.prismaGlobal = prisma;

Wire up a database #8

Wire up a database #8

Conversation

david-mears-2 commented Jul 25, 2024 • edited Loading

codecov bot commented Jul 26, 2024 • edited Loading

Codecov Report

EmmaLRussell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-mears-2 Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-mears-2 Aug 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-mears-2 Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

david-mears-2 Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

david-mears-2 commented Jul 30, 2024 • edited Loading

david-mears-2 commented Jul 30, 2024 • edited Loading

david-mears-2 commented Aug 6, 2024

david-mears-2 commented Aug 6, 2024

EmmaLRussell left a comment

Choose a reason for hiding this comment

david-mears-2 commented Jul 25, 2024 •

edited

Loading

codecov bot commented Jul 26, 2024 •

edited

Loading

david-mears-2 Jul 30, 2024 •

edited

Loading

david-mears-2 Aug 1, 2024 •

edited

Loading

david-mears-2 Jul 30, 2024 •

edited

Loading

david-mears-2 Jul 30, 2024 •

edited

Loading

david-mears-2 commented Jul 30, 2024 •

edited

Loading

david-mears-2 commented Jul 30, 2024 •

edited

Loading