Compute asset summary asynchronously for draft versions #1431
Conversation
Ideally we'd add that change to this PR, but (1) it's a separate (but intimately related) change and (2) I wouldn't want to hold up merging this PR, but it's important that the frontend changes are done soon after if not alongside this PR. @mvandenburgh, I'd like to be involved with designing the frontend changes. Please schedule a meeting with me and/or Dan to discuss. |
danlamanna
force-pushed
the
asynchronous-asset-summaries
branch
3 times, most recently
from
e3c9474
to
64d6639
Compare
danlamanna
force-pushed
the
asynchronous-asset-summaries
branch
from
64d6639
to
3d95ec5
Compare
|
@danlamanna, what's the current status of this PR? You had wanted to hold off before merging, but I see the PR is out of draft mode now. @mvandenburgh, did you have a chance to do a full review? (You left a comment-review rather than an approval, so just checking in with your status on this PR as well.) |
|
@waxlamp we discussed this PR earlier this week, and Dan made some changes based on that, he's just waiting on my review now (which I will do today). |
|
🚀 PR was released in |
For some background: modifying assets requires modifying the draft version to update the asset summary. This creates write contention on the version when assets that belong to the same version are being modified concurrently. The problem is complicated because the asset summary can take several seconds on larger dandisets and the time grows linearly with the asset count. To date this has only been a problem with our largest dandiset.
This PR moves the asset summary computation to an asynchronous task, similar to version metadata validation.
@mvandenburgh Can you take a look at this PR closely since it touches some of your recently modified publishing code?
This is a draft for right now, I want to look closer into the version lifecycle before I merge.