File management in Atomic-Server (with Commits)

[joepio]

Discussion about spec in docs

Requirements:

• User can upload a file
• API should feel familiar
• The metadata of the file should be described using Atomic Data
• Existing Atomic Data Authorization + Authentication should be used

I currently use actix-files for static file hosting. Not sure how this could be replaced. It does a lot of fancy stuff, such as setting mimetype, checking modified_at, etag...

Some approaches:

Use existing filesystem for storage

• We could keep using actix-files, and serve files directly from some pre-defined endpoint(s).
• Commits might still be described in the store. However, the 'initial commit' might have a risk of not existing with this approach. Perhaps add a single commit on initial load.
• How to deal with authorization? I'd rather not have every single static item public
• How to deal with indexing and search? Would a TPF query hit all these items?
• Easy to implement
• High degree of control over folder, easy to use, easy to manage
• Might cause OS-specific filesystem quirks (things with folders and rights, max file length, whatever)
• Hard to secure
• Does not scale well

Files as Resources in the Atomic Data Store

We could add a new type of resource, which has a filename, size, bytearray / string, mimetype. This would mean that we could use existing Resource logic - including things like commits and the TPF index.

However, including the bytearray would make things very slow for uploading / using the data, as it needs to be (de)serializes as base64 to appear in JSON-AD.

Store files in the DB

Files are a new datatype, which contains just a big u8 array + a MIME type or something

• Harder to implement - especially the HTTP side of things (ByteServing, mime headers for all files). I'd rather not.
• Forces me to think about how files work in atomic data - which is probably a good thing.
• Harder for users to manage - hard to beat a filesystem.
• Possible to encrypt files
• Possible for the server to give guarantees over files - server has more control over DB than filesystem
• Might be hard to optimize / make performant (even though I use a simple K/V store as DB)

Use an S3 like external object storage system

Instead on relying on Atomic server itself to deal with storing stuff, we can use some external object storage service. Mimio is an interesting open source contender.

• Scales well for more centralized servers working with Atomic
• Doesn't fit well with the no-dependencies proposition of Atomic.

Persist using IPFS library / addon / instance

IPFS #66 could be used as an addon / crate to atomic-server that deals with the complexities of IPFS (connecting to peers, hashing, resolving identifiers to binary arrays).

Maybe ipfs-embed is a nice option for this. Contrary to rust-ipfs, this one seems to support adding (inserting) byte arrays to IPFS.

[joepio]

Old usecase (removed from top level issue)

I'd like to manage how static pages look in my Atomic Server. Now, I could do this using git and all, but one of the core ideas for Atomic Data is that you don't need git - the Atomic Commits model kind of offers enough abstractions to properly manage stuff like web pages. Personally, I'd simply like to use the existing cli and do something like this:atomic-cli edit mypage.html html, which should open my default text editor, and on save a Commit should be made using my default agent. I should even be able to do this for Tera templates, without rebooting the server.

[jonassmedegaard]

One option (also since atomic-data already generally speaks http(s) to its users) is to use the Micropub standard.

[joepio]

API design thoughts

• Add an /upload Endpoint, to which the user can Post (multi-part) Form data.
• Or we could POST to any resource, and create an attachment for the newly uploaded item. That way, all files will have some Parent, which helps to deal with rights. However, I kind of like having a really clear endpoint with its own description.
• After upload, the server replies with a newly created URL (or a list of URLs if multiple files) that (when accessed in a web browser) downloads the file. If opened in Atomic-Data-Browser, or fetches using application/ad+json it shows the metadata.
• If we post to /upload, how do we know what the parent is? We could use a JSON filled multi-part form... Or maybe encode it in the URL as a query parameter, which is how Atomic Endpoints work.

Implementation thoughts

• Probably use actix_form_data + actix_multipart. https://docs.rs/actix-form-data/0.5.0/actix_form_data/enum.Value.html and https://docs.rs/actix-multipart/latest/actix_multipart/
• Store files on disk in a user-defined folder
• Store the paths to the files in the Atomic Data data of the newly created File resource.
• This metadata object should contain Authorization information (parent e.g.). On upload, we should set filename, filesize, mimetype data.

[joepio]

Here's an implementation example: https://github.com/saiumesh535/actix-server/pull/14/files