From fab6c7c51ef2bdd95001745574ab944fabc81b0b Mon Sep 17 00:00:00 2001 From: Erik Demaine Date: Fri, 4 Mar 2022 13:45:42 -0500 Subject: [PATCH] Docs update --- HISTORY.md | 10 +++--- README.md | 97 ++++++++++++++++++++++++++---------------------------- 2 files changed, 51 insertions(+), 56 deletions(-) diff --git a/HISTORY.md b/HISTORY.md index ffd5206..1d0b138 100644 --- a/HISTORY.md +++ b/HISTORY.md @@ -2,7 +2,7 @@ ### V2.0.0 -* Major rewrite, with three significant breaking changes +* Major rewrite, with three significant breaking changes: * BREAKING CHANGE: No more file locking, under the assumption that you write files once during upload and henceforth do concurrent reads. Make sure your app doesn't require locking. @@ -14,11 +14,11 @@ `insert` method has been replaced by `insertEmpty`. Relatedly, `allow` no longer has a `write` option; instead, POST/PUT and Resumable use `insert` permissions. -* In the new `insertStream`, `options.range.end` is exclusive. - (By contrast, `upsertStream`'s corresponding option was inclusive.) +* BREAKING CHANGE: In `readOneStream` and the new `insertStream`, + `options.range.end` is exclusive (before it was inclusive). This won't affect you if you're using the built-in GET handler, which - converts from inclusive to exclusive. If you've written your own GET - handler, you'll want to offset by 1. + converts from HTTP's inclusive Range to exclusive. + If you've written your own GET handler, you'll want to offset by 1. * Upgrade to MongoDB's v4 drivers, in particular the modern [GridFSBucket interface](https://mongodb.github.io/node-mongodb-native/4.4/classes/GridFSBucket.html) * No more `gridfs-locking-stream` dependency diff --git a/README.md b/README.md index 1de23e9..5040e86 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ Major features: * External changes to the underlying file store automatically synchronize with the Meteor collection * Designed for efficient handling of millions of small files as well as huge files 10GB and above -These features (and more) are possible because file-collection tightly integrates MongoDB [gridFS](http://docs.mongodb.org/manual/reference/gridfs/) with Meteor Collections, without any intervening plumbing or unnecessary layers of abstraction. +These features (and more) are possible because file-collection tightly integrates MongoDB [GridFS](http://docs.mongodb.org/manual/reference/gridfs/) with Meteor Collections, without any intervening plumbing or unnecessary layers of abstraction. #### Quick server-side example @@ -48,34 +48,23 @@ result = myFiles.remove(thatFile._id); ### Feature summary -Under the hood, file data is stored entirely within the Meteor MongoDB instance using a Mongo technology called [gridFS](http://docs.mongodb.org/manual/reference/gridfs/). Your file collection and the underlying gridFS collection remain perfectly in sync because they *are* the same collection. The file-collection package also provides a simple way to enable secure HTTP (GET, POST, PUT, DELETE) interfaces to your files, and additionally has built-in support for robust and resumable file uploads using the excellent [Resumable.js](http://www.resumablejs.com/) library. +Under the hood, file data is stored entirely within the Meteor MongoDB instance using a Mongo technology called [GridFS](http://docs.mongodb.org/manual/reference/gridfs/). Your file collection and the underlying GridFS collection remain perfectly in sync because they *are* the same collection. The file-collection package also provides a simple way to enable secure HTTP (GET, POST, PUT, DELETE) interfaces to your files, and additionally has built-in support for robust and resumable file uploads using the excellent [Resumable.js](http://www.resumablejs.com/) library. -### What's new in v1.3? +## Related work -* CORS/Cordova support via the ability to define custom HTTP OPTIONS request handlers -* Global and per-request file upload size limits via the new `maxUploadSize` option - -Additional changes are detailed in the HISTORY file. - -### Design philosophy - -**Update: CollectionFS appears to no longer be actively maintained, so caveat emptor.** - -My goal in writing this package was to stay true to the spirit of Meteor and build something efficient and secure that "just works" with a minimum of fuss. - -If you've been searching for ways to deal with file data on Meteor, you've probably also encountered [collectionFS](https://atmospherejs.com/cfs/standard-packages). If not, you should definitely check it out. It's a great set of packages written by smart people, and I even pitched in to help with a rewrite of their MongoDB gridFS support. - -Here's the difference in a nutshell: collectionFS is a Ferrari, and file-collection is a Fiat. - -They do approximately the same thing using some of the same technologies, but reflect different design priorities. file-collection is much simpler and somewhat less flexible; but if it meets your needs you'll find it has a lot fewer moving parts and may be significantly more efficient to work with and use. - -If you're trying to quickly prototype an idea or you know that you just need a straightforward way of dealing with files, you should definitely try file-collection. Because it is so much simpler, you may also find that it is easier to understand and customize for the specific needs of your project. +A major other library for supporting file uploads for Meteor is +[Meteor-Files](https://github.com/VeliovGroup/Meteor-Files/). +The main difference is that file-collection makes it really easy to +use GridFS (and that's all it supports), whereas Meteor-Files requires +[significant extra code](https://github.com/VeliovGroup/Meteor-Files/blob/master/docs/gridfs-bucket-integration.md#use-gridfs-with-gridfsbucket-as-a-storage) +to integrate, +[especially for advanced features like HTTP streaming](https://github.com/VeliovGroup/Meteor-Files/wiki/GridFS---206-Streaming). +So while Meteor-Files is a more flexible solution, file-collection is easier +to get up and running with your existing MongoDB database. ## Example -Enough words, time for some more code... - -The block below implements a `FileCollection` on server, including support for owner-secured HTTP file upload using `Resumable.js` and HTTP download. It also sets up the client to provide drag and drop chunked file uploads to the collection. The only things missing here are UI templates and some helper functions. See the [meteor-file-sample-app](https://github.com/vsivsi/meteor-file-sample-app) project for a complete working version written in [CoffeeScript](http://coffeescript.org/). +The block below implements a `FileCollection` on server, including support for owner-secured HTTP file upload using `Resumable.js` and HTTP download. It also sets up the client to provide drag and drop chunked file uploads to the collection. The only things missing here are UI templates and some helper functions. ```javascript // Create a file collection, and enable file upload and download using HTTP @@ -114,13 +103,14 @@ if (Meteor.isServer) { // The creator of a file owns it. UserId may be null. insert: function (userId, file) { // Assign the proper owner when a file is created - file.metadata = file.metadata || {}; + if (!file.metadata) file.metadata = {}; file.metadata.owner = userId; - return true; + return true; // always allow uploading + return Boolean(userId) // or require being logged + return (file.metadata.cool === true) // or depend on metadata }, - // Only owners can remove a file + // Only owners can delete a file remove: function (userId, file) { - // Only owners can delete return (userId === file.metadata.owner); }, // Only owners can retrieve a file via HTTP GET @@ -165,6 +155,11 @@ if (Meteor.isClient) { } ``` +For more complete examples, see: + +* [Coauthor](https://github.com/edemaine/coauthor/) is an actively maintained use of file-collection (including userId+group-based authentication via metadata when uploading files). +* [meteor-file-sample-app](https://github.com/vsivsi/meteor-file-sample-app) is old and unmaintained, so may need tweaking to get working again. + ## Installation To add to your project, run: @@ -205,7 +200,7 @@ Load `http://localhost:3000/` and the tests should run in your browser and on th ## Use -Below you'll find the [MongoDB gridFS `files` data model](http://docs.mongodb.org/manual/reference/gridfs/#the-files-collection). This is also the schema used by file-collection because a FileCollection *is* a gridFS collection. +Below you'll find the [MongoDB GridFS `files` data model](http://docs.mongodb.org/manual/reference/gridfs/#the-files-collection). This is also the schema used by file-collection because a FileCollection *is* a GridFS collection. ```javascript { @@ -222,9 +217,9 @@ Below you'll find the [MongoDB gridFS `files` data model](http://docs.mongodb.or } ``` -Here are a few things to keep in mind about the gridFS file data model: +Here are a few things to keep in mind about the GridFS file data model: -* Some of the attributes belong to gridFS, and you may **lose data** if you mess around with these. +* Some of the attributes belong to GridFS, and you may **lose data** if you mess around with these. * For this reason, `_id`, `length`, `chunkSize`, `uploadDate` and `md5` are read-only. * Some of the attributes belong to you. Your application can do whatever you want with them. * `filename`, `contentType`, `aliases` and `metadata` are yours. Go to town. @@ -233,7 +228,7 @@ Here are a few things to keep in mind about the gridFS file data model: Sound complicated? It really isn't and file-collection is here to help. -First off, when you create a new file you use `myFiles.insertStream(...)` or `myFiles.insertEmpty(...)` and just populate whatever attributes you care about. The file-collection package does the rest. You are guaranteed to get a valid gridFS file. +First off, when you create a new file you use `myFiles.insertStream(...)` or `myFiles.insertEmpty(...)` and just populate whatever attributes you care about. The file-collection package does the rest. You are guaranteed to get a valid GridFS file. Likewise, when you run `myFiles.update(...)` on the server, file-collection tries really hard to make sure that you aren't clobbering one of the "read-only" attributes with your update modifier. For safety, clients are never allowed to directly `update`, although you can selectively give them that power via `Meteor.methods`. @@ -249,7 +244,7 @@ file-collection offers no locking, so you shouldn't modify files after uploading ### Security -You may have noticed that the gridFS `files` data model says nothing about file ownership. That's your job. If you look again at the example code block above, you will see a bare bones `Meteor.userId` based ownership scheme implemented with the attribute `file.metadata.owner`. As with any Meteor Collection, allow/deny rules are needed to enforce and defend that document attribute, and file-collection implements that in *almost* the same way that ordinary Meteor Collections do. Here's how they're a little different: +You may have noticed that the GridFS `files` data model says nothing about file ownership. That's your job. If you look again at the example code block above, you will see a bare bones `Meteor.userId` based ownership scheme implemented with the attribute `file.metadata.owner`. As with any Meteor Collection, allow/deny rules are needed to enforce and defend that document attribute, and file-collection implements that in *almost* the same way that ordinary Meteor Collections do. Here's how they're a little different: * The `insert` allow/deny rules work just as you would expect for client `insertEmpty` calls, but more likely you want to upload a nonempty file via POST/PUT or Resumable, which are similarly protected by the `insert` rule. * The `remove` allow/deny rules work just as you would expect for client calls, and they also secure the HTTP DELETE method when it's used. @@ -273,7 +268,7 @@ The big losers are `insert()` and `upsert()`, which are not supported by `FileCo fc = new FileCollection('fs', // base name of collection { resumable: false, // Disable resumable.js upload support resumableIndexName: undefined, // Not used when resumable is false - chunkSize: 2*1024*1024 - 1024, // Use 2MB chunks for gridFS and resumable + chunkSize: 2*1024*1024 - 1024, // Use 2MB chunks for GridFS and resumable baseURL: '\gridfs\fs', // Default base URL for all HTTP methods http: [] // HTTP method definitions, none by default } @@ -282,7 +277,7 @@ fc = new FileCollection('fs', // base name of collection **Note:** The same `FileCollection` call should be made on both the client and server. -`name` is the root name of the underlying MongoDB gridFS collection. If omitted, it defaults to `'fs'`, the default gridFS collection name. Internally, three collections are used for each `FileCollection` instance: +`name` is the root name of the underlying MongoDB GridFS collection. If omitted, it defaults to `'fs'`, the default GridFS collection name. Internally, three collections are used for each `FileCollection` instance: * `[name].files` - This is the collection you actually see when using file-collection * `[name].chunks` - This collection contains the actual file data chunks. It is managed automatically. @@ -295,7 +290,7 @@ Here are the options `FileCollection` does support: * `options.resumable` - `` When `true`, exposes the [Resumable.js API](http://www.resumablejs.com/) on the client and the matching resumable HTTP support on the server. * `options.resumableIndexName` - `` When provided and `options.resumable` is `true`, this value will be the name of the internal-use MongoDB index that the server-side resumable.js support attempts to create. This is useful because the default index name MongoDB creates is long (94 chars out of a total maximum namespace length of 127 characters), which may create issues when combined with long collection and/or database names. If this collection already exists the first time an application runs using this setting, it will likely have no effect because an identical index will already exist (under a different name), causing MongoDB to ignore request to create a duplicate index with a different name. In this case, you must manually drop the old index and then restart your application to generate a new index with the requested name. -* `options.chunkSize` - `` Sets the gridFS and Resumable.js chunkSize in bytes. The default value of a little less than 2MB is probably a good compromise for most applications, with the maximum being 8MB - 1. Partial chunks are not padded, so there is no storage space benefit to using small chunk sizes. If you are uploading very large files over a fast network and upload spped matters, then a `chunkSize` of 8MB - 1KB (= 8387584) will likly optimize upload speed. However, if you elect to use such large `chunkSize` values, make sure that the replication oplog of your MongoDB instance is large enough to handle this, or you will risk having your client and server collections lose synchronization during uploads. Meteor's development mode only uses an oplog of 8 MB, which will almost certainly cause problems for high speed uploads to apps using a large `chunkSize`. +* `options.chunkSize` - `` Sets the GridFS and Resumable.js chunkSize in bytes. The default value of a little less than 2MB is probably a good compromise for most applications, with the maximum being 8MB - 1. Partial chunks are not padded, so there is no storage space benefit to using small chunk sizes. If you are uploading very large files over a fast network and upload spped matters, then a `chunkSize` of 8MB - 1KB (= 8387584) will likly optimize upload speed. However, if you elect to use such large `chunkSize` values, make sure that the replication oplog of your MongoDB instance is large enough to handle this, or you will risk having your client and server collections lose synchronization during uploads. Meteor's development mode only uses an oplog of 8 MB, which will almost certainly cause problems for high speed uploads to apps using a large `chunkSize`. For more information on Meteor's use of the MongoDB oplog, see: [Meteor livequery](https://www.meteor.com/livequery). * `options.baseURL` - `` Sets the base route for all HTTP interfaces defined on this collection. Default value is `/gridfs/[name]` * `option.maxUploadSize` - `` Maximum number of bytes permitted for any HTTP POST, PUT or resumable.js file upload. @@ -317,7 +312,7 @@ Each object in the `option.http` array defines one HTTP request interface on the When arranging http interface definition objects in the array provided to `options.http`, be sure to put more specific paths for a given HTTP method before more general ones. For example: `/hash/:md5` should come before `/:filename/:_id` because `"hash"` would match to filename, and so `/hash/:md5` would never match if it came second. Obviously this is a contrived example to demonstrate that order is significant. -Note that an authenticated userId is not provided to the `lookup` function. UserId based permissions should be managed using the allow/deny rules described later on. +Note that an authenticated userId is not provided to the `lookup` function. UserId-based permissions should be managed using the allow/deny rules described later on. Here are some example HTTP interface definition objects to get you started: @@ -547,7 +542,7 @@ fc.resumable.assignDrop($(".fileDrop")); // Assign a file drop target // When a file is dropped on the target (or added some other way) myData.resumable.on('fileAdded', function (file) { - // file contains a resumable,js file object, do something with it... + // file contains a resumable.js file object, do something with it... } ``` @@ -591,7 +586,7 @@ _id = fc.insert({ ); ``` -`fc.insertEmpty()` is similar as [Meteor's `Collection.insert()`](http://docs.meteor.com/#insert), except that the document is forced to be a [gridFS `files` document](http://docs.mongodb.org/manual/reference/gridfs/#the-files-collection), and it's not possible to specify the file contents (the file is permanently empty). All attributes not supplied get default values, non-gridFS attributes are silently dropped. Inserts from the client that do not conform to the gridFS data model will automatically be denied. Client inserts will additionally be subjected to any `'insert'` allow/deny rules (which default to deny all inserts). +`fc.insertEmpty()` is similar as [Meteor's `Collection.insert()`](http://docs.meteor.com/#insert), except that the document is forced to be a [GridFS `files` document](http://docs.mongodb.org/manual/reference/gridfs/#the-files-collection), and it's not possible to specify the file contents (the file is permanently empty). All attributes not supplied get default values, non-GridFS attributes are silently dropped. Inserts from the client that do not conform to the GridFS data model will automatically be denied. Client inserts will additionally be subjected to any `'insert'` allow/deny rules (which default to deny all inserts). ### fc.remove(selector, [callback]) #### Remove a file and all of its data. - Server and Client @@ -604,10 +599,10 @@ fc.remove( ); ``` -`fc.remove()` is nearly the same as [Meteor's `Collection.remove()`](http://docs.meteor.com/#remove), except that in addition to removing the file document, it also removes the file data chunks from the gridFS store. For safety, undefined and empty selectors (`undefined`, `null` or `{}`) are all rejected. Client calls are subjected to any `'remove'` allow/deny rules (which default to deny all removes). Returns the number of documents actually removed on the server, except when invoked on the client without a callback. In that case it returns the simulated number of documents removed from the local mini-mongo store. +`fc.remove()` is nearly the same as [Meteor's `Collection.remove()`](http://docs.meteor.com/#remove), except that in addition to removing the file document, it also removes the file data chunks from the GridFS store. For safety, undefined and empty selectors (`undefined`, `null` or `{}`) are all rejected. Client calls are subjected to any `'remove'` allow/deny rules (which default to deny all removes). Returns the number of documents actually removed on the server, except when invoked on the client without a callback. In that case it returns the simulated number of documents removed from the local mini-mongo store. ### fc.update(selector, modifier, [options], [callback]) -#### Update application controlled gridFS file attributes. - Server only +#### Update application controlled GridFS file attributes. - Server only Note: A local-only version of update is available on the client. See docs for `fc.localUpdate()` for details. @@ -626,8 +621,8 @@ fc.update( `fc.update()` is nearly the same as [Meteor's `Collection.update()`](http://docs.meteor.com/#update), except that it is a server-only method, and it will return an error if: -* any of the gridFS "read-only" attributes would be modified -* any standard gridFS document level attributes would be removed +* any of the GridFS "read-only" attributes would be modified +* any standard GridFS document level attributes would be removed * the `upsert` option is attempted Since `fc.update()` only runs on the server, it is *not* subjected to any allow/deny rules. @@ -672,8 +667,8 @@ Meteor.methods({ It will return an error if: -* any of the gridFS "read-only" attributes would be modified -* any standard gridFS document level attributes would be removed +* any of the GridFS "read-only" attributes would be modified +* any standard GridFS document level attributes would be removed * the `upsert` option is attempted Since `fc.localUpdate()` only changes data on the client, it is *not* subjected to any allow/deny rules. @@ -697,7 +692,7 @@ The parameters for callback functions for all three types of allow/deny rules ar ```js function (userId, file) { // userId is Meteor account if authenticated - // file is the gridFS file record for the matching file, + // file is the GridFS file record for the matching file, // or for insert rules, the initial data describing the file } ``` @@ -733,7 +728,7 @@ Other available options are `options.sort` and `options.skip` which have the sam The returned stream is a Mongo [GridFSBucketReadStream](https://mongodb.github.io/node-mongodb-native/4.4/classes/GridFSBucketReadStream.html). -When the stream has ended, the `callback` is called with the gridFS file document. +When the stream has ended, the `callback` is called with the GridFS file document. ### fc.insertStream(file, [options], [callback]) #### Create a file collection file and return a writable stream to its data. - Server only @@ -746,7 +741,7 @@ nyanStream = fc.insertStream({ filename: 'nyancat.flv', }); ``` -`fc.insertStream()` is similar to Meteor's `Collection.prototype.insert()`, but returning a writable stream for file contents. Optionally, the `file` parameter can specify an `_id` field to specify what ID to use for the file, but it will fail if another file already has that ID. If no `_id` is provided, then a new ID is created automatically. Any application-owned gridFS attributes (`filename`, `contentType`, `aliases`, `metadata`) that are present in the `file` parameter will be used for the file. +`fc.insertStream()` is similar to Meteor's `Collection.prototype.insert()`, but returning a writable stream for file contents. Optionally, the `file` parameter can specify an `_id` field to specify what ID to use for the file, but it will fail if another file already has that ID. If no `_id` is provided, then a new ID is created automatically. Any application-owned GridFS attributes (`filename`, `contentType`, `aliases`, `metadata`) that are present in the `file` parameter will be used for the file. The `options` argument is currently ignored; it is allowed just for interface similarly with `Collection.prototype.insert`. @@ -755,7 +750,7 @@ Once that is done, `fc.insertStream()` returns a [writable stream](http://nodejs *NOTE! Breaking Change*! Prior to file-collection v2.0, this function was named `upsertStream` and it supported (re)writing data for existing files. `insertStream` only works for nonexisting files. -When the write stream has closed, the `callback` is called as `callback(error, file)`, where file is the gridFS file document following the write. +When the write stream has closed, the `callback` is called as `callback(error, file)`, where file is the GridFS file document following the write. ### fc.exportFile(selector, filePath, callback) #### Export a file collection file to the local fileSystem. - Server only @@ -771,7 +766,7 @@ fc.exportFile({ 'filename': 'nyancat.flv'}, `fc.exportFile()` is a convenience method that [pipes](http://nodejs.org/api/stream.html#stream_readable_pipe_destination_options) the readable stream produced by `fc.findOneStream()` into a local [file system writable stream](http://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options). -The `selector` parameter works as it does with `fc.findOneStream()`. The `filePath` is the String directory path and filename in the local filesystem to write the file data to. The value of the `filename` attribute in the found gridFS file document is ignored. The callback is mandatory and will be called with a single parameter that will be either an `Error` object or `null` depending on the success of the operation. +The `selector` parameter works as it does with `fc.findOneStream()`. The `filePath` is the String directory path and filename in the local filesystem to write the file data to. The value of the `filename` attribute in the found GridFS file document is ignored. The callback is mandatory and will be called with a single parameter that will be either an `Error` object or `null` depending on the success of the operation. ### fc.importFile(filePath, file, callback) #### Import a local filesystem file into a file collection file. - Server only @@ -790,4 +785,4 @@ fc.importFile('/funtimes/lolcat_183.gif', `fc.importFile()` is a convenience method that [pipes](http://nodejs.org/api/stream.html#stream_readable_pipe_destination_options) a local [file system readable stream](http://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options) into the writable stream produced by a call to `fc.upsertStream()`. -The `file` parameter works as it does with `fc.upsertStream()`. The `filePath` is the String directory path and filename in the local filesystem of the file to open and copy into the gridFS file. The callback is mandatory and will be called with the same callback signature as `fc.upsertStream()`. +The `file` parameter works as it does with `fc.upsertStream()`. The `filePath` is the String directory path and filename in the local filesystem of the file to open and copy into the GridFS file. The callback is mandatory and will be called with the same callback signature as `fc.upsertStream()`.