-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[NFT Metadata Crawler] Asset Uploader Throttler #14904
[NFT Metadata Crawler] Asset Uploader Throttler #14904
Conversation
⏱️ 22m total CI duration on this PR
|
let mut queue = self_clone.asset_queue.lock(); | ||
queue.insert(asset); | ||
}; | ||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
its possible that server restarts when asset from in progress queue was already sent for uploading. so you might run into errors where it's already uploaded, so failed to upload again but DB wasn't updated yet due to server restart. we might need to handle that error specifically and write to DB even if it errors (also make a GET to cloudflare to get the uploaded url?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a lot of error handling around asset already exists 409 (5409) errors:
- If the case you described happens, the result is the asset existing in Cloudflare but not the DB. I introduce another endpoint to the worker that performs a best effort "lookup" operation to see if the asset_uri exists in Cloudflare. It needs to parse through the list of ALL images in Cloudflare to see if the asset we want is already there (there isn't a lookup option 😡). We help the lookup by additionally writing the original asset_uri in the metadata for each uploaded asset. This is an expensive operation, but it should basically never be used.
- For other 409s, we attempt to first grab the data from the parsed_aasset_uris table to see if there's already an exiting CDN URI there.
} | ||
|
||
impl Server for AssetUploaderThrottlerContext { | ||
fn build_router(&self) -> axum::Router { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm wonder do we need to have a fixed threads like 4 to upload assets? right now we're single threaded in asset uploading
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The uploading is actually all spun off onto a separate thread.
The only thing sequential about this is getting the next asset to upload, which is needed bc we want all the uploading to be in order
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remind me why we want all uploading to be in order? for fairness?
asset_queue: Arc<Mutex<BTreeSet<AssetUploaderRequestStatusesQuery>>>, | ||
in_progress_assets: Arc<Mutex<AHashSet<AssetUploaderRequestStatusesQuery>>>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we use single lock for both queues to avoid possible race conditions btw queue updates and uploads? where an asset could be added to the queue by update_queue
just as it's being removed and processed by handle_upload_assets
? chatgpt suggested https://docs.rs/crossbeam-queue/latest/crossbeam_queue/ as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oki doki
@@ -0,0 +1,253 @@ | |||
// Copyright © Aptos Foundation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i assume we only will have single throttler instance anytime with in memory queue? let say in future we replace cloudflare with another provider without rate limit concerns, i guess we can even safely remove throttler?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes and yes 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
overall lgtm! how to verify biz logics work as intended tho?
} | ||
|
||
impl Server for AssetUploaderThrottlerContext { | ||
fn build_router(&self) -> axum::Router { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remind me why we want all uploading to be in order? for fairness?
416cb79
into
justin/asset-uploader-api
* s * fix * edits * update model * upd oop * [NFT Metadata Crawler] Asset Uploader Throttler (#14904) * yay * upsert * awesome * AHHHHHHHHHHHHHHHHHHHHHHH * boom * boom * lint * lint
* s * fix * edits * update model * upd oop * [NFT Metadata Crawler] Asset Uploader Throttler (#14904) * yay * upsert * awesome * AHHHHHHHHHHHHHHHHHHHHHHH * boom * boom * lint * lint
Description
Implementation of the Asset Uploader Throttler.
Design: https://www.notion.so/aptoslabs/WIP-Asset-Uploader-cb33138051324d1fa03115ef1b55ed0b
How Has This Been Tested?
Key Areas to Review
src/asset_uploader/throttler/mod.rs
Type of Change
Which Components or Systems Does This Change Impact?
Checklist