Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2 #37

Merged
merged 51 commits into from
Dec 21, 2024
Merged

v2 #37

Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
d4a1ba7
Move python code into separate folder
ruchernchong Dec 14, 2024
42512e2
Initial migration
ruchernchong Dec 14, 2024
e1f4fa6
Update project name in SST config
ruchernchong Dec 14, 2024
af4f483
Update details in package.json
ruchernchong Dec 14, 2024
22a6896
Clean up updater and use SST Secrets
ruchernchong Dec 14, 2024
da2c02c
Clean up imports
ruchernchong Dec 14, 2024
87614e9
Temporary disable caching
ruchernchong Dec 14, 2024
40864b1
Format with biome
ruchernchong Dec 14, 2024
666f11a
Unable to use SST Secret for drizzle
ruchernchong Dec 14, 2024
76965f2
Update GHA workflows
ruchernchong Dec 15, 2024
52bb2af
Update path alias in tests
ruchernchong Dec 15, 2024
5a22a7e
Update GHA workflow env secrets
ruchernchong Dec 15, 2024
f683db1
Add drizzle migrate
ruchernchong Dec 15, 2024
ed5e067
Add README
ruchernchong Dec 15, 2024
6f76cc1
Create LICENSE
ruchernchong Dec 15, 2024
d9fb27d
Update license
ruchernchong Dec 15, 2024
a1f849f
Update description
ruchernchong Dec 15, 2024
806f198
Refactor and clean up GHA workflows
ruchernchong Dec 15, 2024
649a6cf
Grant permissions for workflow callers
ruchernchong Dec 15, 2024
3909f1e
Refactor job name to lowercase in GHA workflow
ruchernchong Dec 15, 2024
8617dce
Add checksum caching back
ruchernchong Dec 16, 2024
553d789
Fix processing of CSV
ruchernchong Dec 16, 2024
34d7bf5
Update transformer to handle values like Coupe/ Convertible
ruchernchong Dec 21, 2024
b269c29
Update formatting to 2 spaces with biome
ruchernchong Dec 21, 2024
bbf3512
Setup updating data using trigger.dev
ruchernchong Dec 21, 2024
5efbc39
Fix table name
ruchernchong Dec 21, 2024
c958979
Fix table name
ruchernchong Dec 21, 2024
746959d
Add scheduler to trigger updater
ruchernchong Dec 21, 2024
19d2f64
Merge pull request #36 from sgcarstrends/35-setup-triggerdev
ruchernchong Dec 21, 2024
585aba2
Add custom CSV transform for different data
ruchernchong Dec 21, 2024
e292dc5
Fix types
ruchernchong Dec 21, 2024
47d8efd
Ugrade drizzle
ruchernchong Dec 21, 2024
40be00c
Refactor drizzle index deprecated API
ruchernchong Dec 21, 2024
b643507
Refactor table names
ruchernchong Dec 21, 2024
8e0cf6f
Temporary remove caching of checksum
ruchernchong Dec 21, 2024
eb647d8
Rename GHA workflow job names
ruchernchong Dec 21, 2024
a57e409
2.0.0-beta.1
ruchernchong Dec 21, 2024
d521040
Add missing pnpm setup in GHA workflow
ruchernchong Dec 21, 2024
d83a126
2.0.0-beta.2
ruchernchong Dec 21, 2024
2045b7b
Update npm scripts
ruchernchong Dec 21, 2024
b80e454
Temporary remove unused Hono API
ruchernchong Dec 21, 2024
9c48872
2.0.0-beta.3
ruchernchong Dec 21, 2024
ed29046
Ignore GHA workflow from triggering on tags
ruchernchong Dec 21, 2024
b6a5f50
Refactor redis
ruchernchong Dec 21, 2024
2dc0c2a
Add lastUpdated timestamp onsuccess
ruchernchong Dec 21, 2024
0546a40
Refactor and clean up repeated code
ruchernchong Dec 21, 2024
581b387
Update regex pattern due to security hotspot
ruchernchong Dec 21, 2024
4b0a8f9
Add error handling for updater
ruchernchong Dec 21, 2024
775886d
Refactor and clean up
ruchernchong Dec 21, 2024
577a00b
Reduce timeout in dev
ruchernchong Dec 21, 2024
f9c7fcc
Refactor LTA Datamall base url
ruchernchong Dec 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor table names
ruchernchong committed Dec 21, 2024
commit b643507bb6e634d6c3a9e4bc82c96cbed2948b76
4 changes: 2 additions & 2 deletions src/lib/updateCOE.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { coeTable } from "@/schema";
import { coe } from "@/schema";
import type { COE } from "@/types";
import { updater } from "./updater";

@@ -22,7 +22,7 @@ export const updateCOE = async () => {
];

const response = await updater<COE>({
table: coeTable,
table: coe,
zipFileName,
zipUrl,
keyFields,
4 changes: 2 additions & 2 deletions src/lib/updateCars.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { carsTable } from "@/schema";
import { cars } from "@/schema";
import type { Car } from "@/types";
import { updater } from "./updater";

@@ -8,7 +8,7 @@ export const updateCars = async () => {
const keyFields: Array<keyof Car> = ["month"];

const response = await updater<Car>({
table: carsTable,
table: cars,
zipFileName,
zipUrl,
keyFields,
5 changes: 2 additions & 3 deletions src/lib/updater.ts
Original file line number Diff line number Diff line change
@@ -6,11 +6,10 @@ import { createUniqueKey } from "@/utils/createUniqueKey";
import { downloadFile } from "@/utils/downloadFile";
import { type CSVTransformOptions, processCSV } from "@/utils/processCSV";
import { cacheChecksum, getCachedChecksum } from "@/utils/redisCache";
import { getTableName } from "drizzle-orm";
import type { PgTable } from "drizzle-orm/pg-core";
import { type Table, getTableName } from "drizzle-orm";

export interface UpdaterConfig<T> {
table: PgTable;
table: Table;
zipFileName: string;
zipUrl: string;
keyFields: string[];
12 changes: 6 additions & 6 deletions src/schema.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { index, integer, pgTable, text, uuid } from "drizzle-orm/pg-core";

export const carsTable = pgTable(
export const cars = pgTable(
"cars",
{
id: uuid("id").defaultRandom().primaryKey(),
@@ -21,7 +21,7 @@ export const carsTable = pgTable(
],
);

export const coeTable = pgTable(
export const coe = pgTable(
"coe",
{
id: uuid("id").defaultRandom().primaryKey(),
@@ -47,8 +47,8 @@ export const coeTable = pgTable(
],
);

export type InsertCar = typeof carsTable.$inferInsert;
export type SelectCar = typeof carsTable.$inferSelect;
export type InsertCar = typeof cars.$inferInsert;
export type SelectCar = typeof cars.$inferSelect;

export type InsertCOE = typeof coeTable.$inferInsert;
export type SelectCOE = typeof coeTable.$inferSelect;
export type InsertCOE = typeof coe.$inferInsert;
export type SelectCOE = typeof coe.$inferSelect;