Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Testing] Add Informal Test Suite #82

Open
asaprahul opened this issue Mar 13, 2023 · 0 comments
Open

[Testing] Add Informal Test Suite #82

asaprahul opened this issue Mar 13, 2023 · 0 comments

Comments

@asaprahul
Copy link
Collaborator

With the speed of iteration, it seems like the right time to introduce a very basic test suite to ensure there's no regression in accuracy.

Today: We have 8 queries that we run against every PR to get a vibe-check if stuff is breaking or accuracy has regressed.

- What are the three highest income zipcodes in {City}
- Give me 5 zipcodes in {State} with the highest income and hispanic population of at least 2,000
- Which 3 zipcodes in {City} have the highest female to male ratio?
- Which zipcode in {City} has the most racial diversity and what is the racial distribution?
- 10 highest crime cities in {State}
- Which 20 zipcodes in {State} have median income that's closes to the national median income?
- Which zipcode has a median income that is closest to the national median income?
- What is the distribution of all crime in {City}

Vibe-check:

  • Does the output make sense?
  • Is there a degradation in speed? i.e. is it fast (~2s for most e2e requests, sometimes 4s or 5s when the API has to retry)

For the API, we could make these unit tests (heads up - they could be very flaky because LLMs are non-deterministic). Verifying if the generated SQL matches the expected output may not work, but checking the table values might work! We could also just have the table values outputted and compare manually.

For the front-end, can we do something simple here? If not, we can just manually try the 8 queries in the Vercel preview deployments, that's pretty clutch already

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant