[Media Library] Allow uploading multiple images at once

[owenhoskins]

This issue is to bring attention to a feature that was mentioned by @indysigner in the now complete Media library UI #350 and Design Improvement: The Editor #180.

• Add an image upload function that allows uploading multiple images at once.

And a drag and drop uploader would be nice to have along with this.

I am comfortable with the frontend aspect but would need some pointers on how to handle the files within the context of Netlify CMS.

best,
Owen

[erquhart]

Thanks for taking this on @owenhoskins! I'll add some notes about file handling soon.

[erquhart]

A quick tour of file handling:

File upload handler for the UI:
https://github.com/netlify/netlify-cms/blob/master/src/components/MediaLibrary/MediaLibrary.js#L109-L131

Related Redux actions like persisting and deleting happen here:
https://github.com/netlify/netlify-cms/blob/master/src/actions/mediaLibrary.js

There are general backend hooks for persisting and deleting.

The GitHub backend implementation maps those backend hooks to API functions:
https://github.com/netlify/netlify-cms/blob/master/src/backends/github/implementation.js#L134
https://github.com/netlify/netlify-cms/blob/master/src/backends/github/implementation.js#L148

You'll notice the GitHub API function for persisting media is already set up to support multiple file upload, but deletion will need to be updated to do the same.

Finally, there are GitHub API level functions for persisting and deleting.

Git Gateway extends the GitHub backend and should automatically work with these improvements.

Lastly, the test repo (in-memory) backend would also need to be updated in it's implementation file for persisting and deleting.

Any specific questions I'd be super happy to help with. Gitter is the best place, but if something warrants being here in the issue, that works too.

[erquhart]

Also, you mentioned drag and drop uploads - I think uppy.js would be a great way to go if you want to give it a shot.

If all that stuff I put in my last comment is totally useless, let me know and we can set up a hangout to go over things.

[ww7]

Plus multiple deletion/selection.

[owenhoskins]

I've just been looking into uppy.js per your suggestion @erquhart. Very nice!

For the design aspect, I am thinking it could be nice to integrate a custom version of the uploading states from the Dashboard Modal into the Media Library modal:

We can configure Uppy to store its state in our Redux Store so that we can write a custom uploader UI component based on the their Dashboard Modal, for instance.

Curious to get some feedback on this design direction and I can sketch a few drafts of the integration tomorrow!

Example of a DnD hover state:

[erquhart]

Nice! Ideally we would use Uppy to power our own UI rather than using theirs, which is their stated goal for custom store implementations. To reduce strain on this particular PR, we could use Uppy just for DnD/bulk uploads in the current UI, and then figure out how best to pull in other available functionality moving forward, such as pausing/resuming/etc. Uppy will need to exist beneath surface so that we could even swap it out entirely in the future if the need arose. This is especially important considering our 2.0 goal of providing an API for custom asset storage immigrations.

Thoughts?

[owenhoskins]

I hear you, let's take this one step at a time! My previous comment was inspired by the potential of powering our own UI via Uppy's custom store implementation. From a user's perspective, I really appreciate how they've split the upload process into several very focused steps and could imagine a similar pattern for the Media Library.

Drag and Drop UI

So I dove into uppy.js today and created a test case with the DragDrop component and the uppy ReduxStore.

With Uppy only handling the Drag and Drop UI I am able to hook into the uppy/STATE_UPDATE action and pull the files out of the payload and pass them into the persistMedia method.

So far so good. The question on my mind now is if and how to utilize the ReduxStore in the future? Eg to leverage Uppy features such as a progress bar updating from our Git backend handling?

Handling multiple files

I've also been digging into how to expand the persist media handling to pass through an array of files at each of the steps. I'll follow up with you on gitter, @erquhart!

[owenhoskins]

I've just pushed some work at https://github.com/owenhoskins/netlify-cms/tree/media-library-1032 with support for multiple image upload to Github! This is done via the browser's file-picker; drag drop will follow.

I've also added support for multiple selection in preparation for multiple deletion, but my working knowledge of the Github API is minimal so I am not sure how to approach the extension of the Github API deleteFile method to support multiple files deletion (https://github.com/netlify/netlify-cms/blob/master/src/backends/github/API.js#L302).

From the looks of it, the persistMedia method first uploads the blobs via separate POST requests and then makes a single commit. The deleteFile method makes a single DELETE request with a commit message. https://developer.github.com/v3/repos/contents/#delete-a-file. Is there another strategy to delete multiple files via a single commit or can that actually be achieved with the DELETE method?

[erquhart]

Awesome!!

We'll need to move away from the simpler Contents API to delete multiple files.

Not sure how familiar you are with Git internals, but the GitHub backend's API file constructs trees (just an object of nested files essentially representing a directory structure), which we then push and use for API based commits. Deletion requires constructing a tree without whatever files should be deleted, so bulk deletion can happen with just two calls (pushing the tree + commit).

Check out the composeFileTree method in src/backends/github/api.js and it's usage for an example.

[owenhoskins]

I am new to Git internals so your guidance is much appreciated, @erquhart!

I've created a removeFiles method (owenhoskins@dd70f07) which grabs the current tree from Github, then I filter it against the files that are to be removed, then I pass that filtered tree to the composeTree method, then the new file tree is handled just like the persistFiles method options.mode === SIMPLE.

The result

Github creates a commit: owenhoskins/netlify-cms-1032@ed15f28 with these nested trees: root > static > img

However, no files are changed, but I am not sure what I am missing! 🤔

[erquhart]

Forgot the final step, updating the ref so the commit is actually applied to the branch. The GitHub API class has a patchRef method, which can be used via patchBranch:

https://github.com/netlify/netlify-cms/blob/e5b8af9f4d377b27d42fd284851d7177562372e0/src/backends/github/API.js#L617-L623

The persistFiles method is a good example of all three steps in sequence:

https://github.com/netlify/netlify-cms/blob/e5b8af9f4d377b27d42fd284851d7177562372e0/src/backends/github/API.js#L291-L293

[owenhoskins]

Hey @erquhart, thanks for the response! The method is calling patchBranch just like persistFiles so there must be something else that is amiss...

Looking into a commit and following the nested trees down to the file tree, the SHA of the files tree and the files aren't what I expected based on console logs from the updateTree method. It's as if another tree is somehow being referenced. Here is the SHA and modified tree with the files removed which was created by the updateTree method.

I am wondering how it is that the nested tree structure created by updateTree method could possibly refer to an already existing tree!

[erquhart]

Can you share your code?

[owenhoskins]

You can find it here owenhoskins@dd70f07, I linked to it a couple comments back!

[raf-vs]

Following with excitement! XD Looking forward to this feature being available. Will be a mayor + for netlify cms. (sorry for the totally un-constructive comment, just excited to see this being build 😄 )

Go @owenhoskins and @erquhart ! 👍

[erquhart]

@owenhoskins you're calling updateTree here: master...owenhoskins:media-library-1032#diff-d3cff7a0c66d0acce37277791446d509R330

That method calls getTree and uses it's result as the base tree, hence the issues you're describing in chat about ending up with the wrong tree.

[erquhart]

External media library integrations can upload multiple images at once! Check out #1602 for details.

[raf-vs]

@erquhart I jumped at the change to try this out and it went super easy. Until I hit a bit of a roadblock 😢
Uploadcare doesn't have an option to browse and select previously uploaded images/media... 😞
So although this fixes the multiple image upload, it's not a perfect replacement for the default media library in netlfiy cms imho.

Uploadcare does provide an example to implement "history" via local storage on the client side (see this codepen ) but I'm having trouble implementing this in the netlify cms "plugin"

Any help with this would be awesome!

[rvetere]

so what is the state of this nice work so far @owenhoskins? master...owenhoskins:media-library-1032

as far as i understand, only the "remove multiple" files is missing, but you got the multi-upload solved? could we separate these two features so we could maybe already benefit of the released multi-upload? :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[eur2]

Can't wait for this feature!

[erezrokah]

FYI, when opening the media library when editing an entry, all images for that entry will be uploaded together only after you save the entry.

[vimtor]

A current workaround was to put the images directly in the uploads folder.
New to Netlify CMS, and loving it so far, great work guys! ❤️

[alexboffey]

Yes as @papeloto suggests above, you can specify a media_folder in your config.yml file and move image files into it like so.

# config.yml

media_folder: static/img

Here is a link to the relevant documentation.

[kaboomdigital]

Can't wait for this feature!

You're not the only one ;)

[Jwiggiff]

Has there been any update on this or a predicted timeline? I'd really love to be able to upload multiple files in the media library.

[erezrokah]

Hi @Jwiggiff, a possible solution is described in #4965 so we would happy to receive a contribution for it.

[IRediTOTO]

Don't you have this feature now? @@