Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Seed database with data from the downloaded archives #159

Merged
merged 12 commits into from
Nov 12, 2024
Merged

Seed database with data from the downloaded archives #159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
467dd77
changes in script to populate data on local db using the files in the…
ri72miieop Oct 30, 2024
58d1ab0
docs a bit more clear
ri72miieop Oct 31, 2024
867f724
removed from the original, keeping it the same
ri72miieop Oct 31, 2024
5a4af3f
add missing data on account table and archive_upload table
ri72miieop Nov 1, 2024
6c666c1
code cleanup
ri72miieop Nov 1, 2024
3e3c05e
Merge branch 'main' into update_import_script
TheExGenesis Nov 4, 2024
bdbf66d
granted necessary permissions to service_role
ri72miieop Nov 6, 2024
2e684b5
improved message on error
ri72miieop Nov 6, 2024
3da38b0
Merge branch 'update_import_script' of https://github.com/ri72miieop/…
ri72miieop Nov 6, 2024
2148ca6
removed commented code that will not be necessary
ri72miieop Nov 6, 2024
8bb86da
Merge branch 'main' into update_import_script
TheExGenesis Nov 11, 2024
5b82f57
fix pnpm build errors
ri72miieop Nov 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions docs/local-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,15 @@ For each change you make to the db:

### Seeding the local database

- Download the last export from [the Github Releases](https://github.com/TheExGenesis/community-archive/releases)
- make sure your `.env` file has `NEXT_PUBLIC_SUPABASE_URL` and `NEXT_PUBLIC_SUPABASE_ANON_KEY` setup to the Community Archive db
- run the script `pnpm dev:downloadarchive`
- make sure your `.env` file has `*_LOCAL_*` env vars setup to your local db
- set the `ARCHIVE_PATH` environment variable to the path of the archive folder
- run the script `pnpm dev:importdata`
- run the script `pnpm dev:importfiles`
- wait a bit and then you should be able to see the data in the local supabase studio at http://localhost:54323/project/default/editor
- (OPTIONAL)run the script `pnpm dev:validateimport` to validate the data. afaict there's a few edge cases where the data might not import correctly. For now fixing this is not a priority, PR is welcome!

Note: this process takes a bit of time because there's 100+ archives to import. Feel free to delete some of the archives from the `ARCHIVE_PATH` folder if you want to speed up the process. Another option is finding the optimal `BATCH_SIZE` value in `scripts/importFromFilesToDb.ts` for your machine.

### Sign-in in dev mode

Expand Down
4 changes: 3 additions & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
"dev": "set NODE_ENV=development & set NEXT_PUBLIC_USE_REMOTE_DEV_DB=false & next dev",
"dev-as-prod": "set NODE_ENV=production & next dev",
"dev-remote-db": "set NEXT_PUBLIC_USE_REMOTE_DEV_DB=true & next dev",
"dev:importdata": "tsx --env-file=.env scripts/importFromArchiveDataToDb.ts",
"dev:importfiles": "tsx --env-file=.env scripts/importFromFilesToDb.ts",
"dev:validateimport": "tsx --env-file=.env scripts/validateDbImport.ts",
"dev:downloadarchive": "tsx --env-file=.env scripts/download_storage_anon.mts",
"dev:testconversationids": "tsx --env-file=.env scripts/testConversationIds.ts",
"build": "set NODE_ENV=production & next build",
"start": "next start",
Expand Down
Loading