Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add uploadthing driver #390

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

juliusmarminge
Copy link
Contributor

@juliusmarminge juliusmarminge commented Jan 25, 2024

πŸ”— Linked issue

#389

❓ Type of change

  • πŸ“– Documentation (updates to the documentation, readme, or JSdoc annotations)
  • 🐞 Bug fix (a non-breaking change that fixes an issue)
  • πŸ‘Œ Enhancement (improving an existing functionality like performance)
  • ✨ New feature (a non-breaking change that adds functionality)
  • 🧹 Chore (updates to the build process or auxiliary tools and libraries)
  • ⚠️ Breaking change (fix or feature that would cause existing functionality to change)

πŸ“š Description

πŸ“ Checklist

  • I have linked an issue or discussion.
  • I have updated the documentation accordingly.


const getUTFetch = () => {
return (utFetch ??= ofetch.create({
baseURL: "https://uploadthing.com/api",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it makes sense to support an optional option for baseURL? (mainly to allow unit testing)

}).then((res) => res.ok);
},
getItem(key) {
return ofetch(`https://utfs.io/f/${key}`);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, somehow make it configurable too? (not sure about /api and this API difference but just a taught to again make testing easier)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utfs is our cloudflare rewriter that handles caching and mapping keys to different regions.

uploadthing.com/api is where everything else goes.

can make it configurable for testing though!

src/drivers/uploadthing.ts Outdated Show resolved Hide resolved
src/drivers/uploadthing.ts Outdated Show resolved Hide resolved
async clear() {
const client = getClient();
const keys = await client.listFiles({}).then((r) => r.map((f) => f.key));
return client.deleteFiles(keys).then(() => {});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clear(base) is suported too. . I guess we might need to support it (client-side or server side) otherwise one call drops entire collection of uploads πŸ—‘οΈ

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea πŸ˜…

not sure how though as currently we just have delete by key and all keys are random (no directories (yet))

@juliusmarminge
Copy link
Contributor Author

juliusmarminge commented Jan 25, 2024

@pi0 just tried to test this out and we stop usage of the utapi from client (mostly cause they shouldn't leak their api keys there)
CleanShot 2024-01-25 at 18 17 14

got any other good way to test this?


oh right and our api also doesn't enable cors so it cannot be used from browser anyways :)

CleanShot 2024-01-25 at 18 19 52

@pi0
Copy link
Member

pi0 commented Jan 25, 2024

We have unit tests you can run with pnpm run vitest (unstoage also targets server). We mainly need to mock endpoints (responding from mobile if couldn’t find it please tell me to push commit)

@juliusmarminge
Copy link
Contributor Author

I just realized that in order to support this we'd need to allow setting the key manually. right now uploadthing generates the key for you

* Primarily used for testing
* @default "https://uploadthing.com/api"
*/
uploadthignApiUrl?: string;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

@juliusmarminge
Copy link
Contributor Author

I just realized that in order to support this we'd need to allow setting the key manually. right now uploadthing generates the key for you

@pi0 is there a concept of mapping keys inside unstorage? For example when we setItem we get the key from UT, that we could then map to the unstorage key, i.e:

storage.setItem("foo")
// internally map foo -> KEY_FROM_UT
storage.getItem("foo")
// key = map["foo"], fetch(utfs.io/f/key)

@pi0
Copy link
Member

pi0 commented Jan 25, 2024

I think we mainly need a storage for keys which is usually same storage layer.

If your API is designed to always return random names, I am up to consider a new change that setItemRaw (setItem is mainly for text values) returns the random name. It is a non breaking change and i hope to unblock this presets also any other preset that need this.

(in the meantime, i guess it would be also nice if you consider supporting custom names. let's see which comes first haha)

@markflorkowski
Copy link

(in the meantime, I guess it would be also nice if you consider supporting custom names. let's see which comes first haha)

The reason we don't is because names must be URL-safe, and have a length limit.

@pi0
Copy link
Member

pi0 commented Jan 25, 2024

Hmm, I see. What if we make the driver make sure that input is url-safe and respects the limit like any fs does (so it throws an error if it doesn't comply) I think users of the driver, would understandably follow this and probably can benefit more from such control vs random ids. Thoughts?

@juliusmarminge
Copy link
Contributor Author

So I did a thing to track internal vs uploadthing keys - not sure if it aligns with what this project aims to do though - I'll let you be the judge.

Currently passing all tests except clear since our API throws a 500 when you try to delete 0 files currently (will fix that momentarily)

@markflorkowski
Copy link

Currently passing all tests except clear since our API throws a 500 when you try to delete 0 files currently (will fix that momentarily)

Merged fix for this on the API side

Comment on lines 7 to 16
/**
* Primarily used for testing
* @default "https://uploadthing.com/api"
*/
uploadthignApiUrl?: string;
/**
* Primarily used for testing
* @default "https://utfs.io/f"
*/
uploadthingFileUrl?: string;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these makes no sense as we don't allow this override to the UTApi anyways

Copy link
Member

@pi0 pi0 Jan 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option provides two benefits:

  • Unit testing the driver internally without actually calling UTAPi but mocking it in unstorage end-to-end tests (we use msw but it is not that reliable...)
  • Allow users a way to Proxy your API if their server deployment requires this for any reason

I think both are internal and advanced cases. But if you really want to discourage users more to change them, we can definitely make it better with naming and JSDocs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes but that would only override the URL for the fetch calls that doesnt use utapi. All fetches made within utapi will still use uploadthing.com (although we have an internal env variable that overrides it)

@juliusmarminge
Copy link
Contributor Author

current approach only works for a long-running server since it depends on that in-memory map. we're thinking about how to support user-provided keys

@juliusmarminge
Copy link
Contributor Author

CleanShot 2024-01-25 at 23 03 48

Now just gonna have to get the mocking right πŸ˜…

Copy link

codecov bot commented Jan 25, 2024

Codecov Report

All modified and coverable lines are covered by tests βœ…

Project coverage is 66.05%. Comparing base (f6841df) to head (6444a42).
Report is 39 commits behind head on main.

❗ Current head 6444a42 differs from pull request most recent head a54c648. Consider uploading reports for the commit a54c648 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
+ Coverage   64.79%   66.05%   +1.25%     
==========================================
  Files          41       41              
  Lines        4071     4186     +115     
  Branches      489      520      +31     
==========================================
+ Hits         2638     2765     +127     
+ Misses       1422     1411      -11     
+ Partials       11       10       -1     

β˜” View full report in Codecov by Sentry.
πŸ“’ Have feedback on the report? Share it here.

@juliusmarminge
Copy link
Contributor Author

struggling getting msw to intercept the requests πŸ€”

CleanShot 2024-01-25 at 23 15 52

@pi0
Copy link
Member

pi0 commented Jan 25, 2024

yeah. msw is not the best in class nor supports ofetch which I am aware of and also the fix but still hesitated because by doing it we add an overhead to every user of it only because of msw.

I will try to check it ASAP.

Copy link

nuxt-studio bot commented Jan 25, 2024

βœ… Live Preview ready!

Name Edit Preview Latest Commit
unstorage Edit on Studio β†—οΈŽ View Live Preview 4d55f54

@juliusmarminge
Copy link
Contributor Author

juliusmarminge commented Jan 30, 2024

@pi0 updated to use native custom ids now.

there are some tests failing however that's expected, since they relate to deletion of files.

UT is not a KV store, so when a file is deleted in UT, it is not instantly deleted but it may take up to 30 seconds before it is deleted at the storage provider, and then deleted from our db. some of the tests here assume that calling hasItem(key) directly after deleteItem(key) should return false, which is not the case for uploadthing.

idk how you wanna pursue given that knowledge. alter the tests? custom test suite for UT?


cc @markflorkowski we could, delete the customId field when a file is marked for deletion, so that a new file may be uploaded with that same customId even before the file is removed. idk if that would make sense for us to do, but it would be a workaround to fix this issue

another thing that we could do is to filter out the files marked for deletion in the getFileUrl endpoint. this would be a breaking change to the API but I don't see why they shouldn't be filtered out in the first place...

@@ -231,7 +231,7 @@ export function createStorage<T extends StorageValue>(
async setItems(items, commonOptions) {
await runBatch(items, commonOptions, async (batch) => {
if (batch.driver.setItems) {
await asyncCall(
return asyncCall(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fairly certain this was a bug before. you called setItems and then setItem for every file so files were getting uploaded twice.

noticed this since the newly introduced customids on uploadthing must be unique and we throw if that's not the case and we got duplicate requests when testing the setItems function before i changed the keyword here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. Do you mind splitting it into a new PR as well? πŸ™πŸΌ

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pi0
Copy link
Member

pi0 commented Jan 30, 2024

Exciting updates πŸš€ Sorry i'm quite busy still will try to come to this PR as soon as could also for failing tests.

Copy link
Contributor Author

@juliusmarminge juliusmarminge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: bump deps once properly released

@@ -92,6 +92,7 @@
"types-cloudflare-worker": "^1.2.0",
"typescript": "^5.3.3",
"unbuild": "^2.0.0",
"uploadthing": "6.3.2-canary.a35b49f",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"uploadthing": "6.3.2-canary.a35b49f",
"uploadthing": "6.4.0",

@@ -108,7 +109,8 @@
"@planetscale/database": "^1.13.0",
"@upstash/redis": "^1.28.1",
"@vercel/kv": "^0.2.4",
"idb-keyval": "^6.2.1"
"idb-keyval": "^6.2.1",
"uploadthing": "^6.0.0"
Copy link
Contributor Author

@juliusmarminge juliusmarminge Jan 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"uploadthing": "^6.0.0"
"uploadthing": "^6.4.0"

@pi0 pi0 added the driver label May 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants