forked from apache/incubator-kie-tools
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Fork LightningFS into
kie-sandbox-fs (apache#708)
* Initial commit * Add package-lock.json * Add tab-delimited printer/parser and abstract backend. * Add a makeIndex utility * Add HTTP fallback mechanism * Update README * fix: Azure pipelines tarball filepath * fix: bind methods to fs instance * fix: temporarily disable Safari MacOS testing (sauce seems broken today?) * fix: add semantic-release npm script * fix: ACTUALLY INSTALL semantic-release * fix: trimStart is apparently too new * chore: add UMD build to npm package * fix: readFile returns UInt8Array instead of Buffer BREAKING CHANGE: Reduces UMD build size from 34kb to 12kb so I think its worth it. * chore: fix bug in tests only for Safari Mobile 11.x * feat: add 'rename' function BREAKING CHANGE: The IndexedDB database format changed - the primary key is now an inode number rather than a filepath string. Adding this layer of indirection make renaming files and directories fast. (Before, renaming a directory would have required recursively copying all the files to a new location.) Rather than bloat the code with a migration script, I recommend simply creating a fresh filesystem or blowing the old filesystem away with the `wipe` argument. Maybe you can load v2 and v3 at the same time, and recursively read from the v2 instance and write to the v3 instance? Database migrations are hard, and I apologize. But this should be the first and last backwards incompatible change to the database format. BREAKING CHANGE: the `stat` function now returns `mtimeMs` rather than `mtimeSeconds` and `mtimeNanoseconds` to match what Node's `stat` function returns instead of catering to an implementation detail of `isomorphic-git`. * fix: remove console.log * fix: bind 'rename' to fs instance * fix: 'rename' was forgetting to save the superblock * fix: fix awful inode autoincrement bug * feat: symlink support (apache#11) * feat: add fs.promises (apache#12) * fix: ignore trailing slash (apache#14) * Update README.md (apache#17) * feat: make threadsafe with mutex (apache#19) * fix(HttpBackend): fs.readFile should throw on 404s (apache#24) * fix(rename): don't accidentally file if new filename invalid (apache#26) * fix(unlink): deleting symlinks should not delete file contents (apache#30) * fix(writeFile): preserve file mode when saving files (apache#33) * fix(readdir, rmdir): should throw ENOTDIR if used on files (apache#34) * fix: @wmhilton/idb-keyval -> @isomorphic-git/idb-keyval (apache#25) * Fix for reading from nested symlinks (apache#35) * feat: add 'backFile' function to add http-backed files to superblock (apache#32) * fix: reading through symlinks to http-backed files (apache#36) * chore: add note that `readdir` output is not sorted * fix: a shot in the dark at fixing the "Mutex double-freed" error (apache#39) * a shot in the dark at fixing the "Mutex double-freed" error * skip MS Edge for now since it is not launching * chore: disable Android testing too for now sheesh * fix: eliminate reliance on setInterval (apache#40) BREAKING CHANGE: bumped the supported / tested browsers to match [email protected]. * fix: add a margin of safety to renew mutex early * feat: reinitialize FS via 'init' and provide custom store names (apache#42) * feat: add du method (apache#45) * fix: add du to README (apache#46) * fix: default mode for symlinks should be 0o120000 not 0o666 Co-authored-by: Chris Thompson <[email protected]> * feat: add 'defer' param to 'init' * chore: rename 'master' branch to 'main' * chore: sauce labs no longer has Safari 13.0 * Add typescript typings * Minor fix * Moved the promises to the correct location. * Replaced never with void where appropriate * Added path typings. * Accidentally overwrote the last commit. * Made the option arguments of the promise api optional! * Moved PromisifiedFS into the FS namespace. * Update index.d.ts Co-authored-by: Alex <[email protected]> * feat: add custom `backend` parameter (apache#63) * fix: allow changing 'backend' during 'init' (apache#65) * Adding idbBackendDelegate and writeFileBulk * Add readFileBulk * Add unlinkBulk * Changing TypeScript definitions * Undo formatting * Adding Dexie as devDependency and bulk tests * Fix tests * Prefix test names to differentiate * Add 'dexie' as peerDependency * Removing transactions from DexieBackend * Tests passing * Forking LightningFS into kie-sandbox-fs * Fix name Co-authored-by: William Hilton <[email protected]> Co-authored-by: Jakub T. Jankiewicz <[email protected]> Co-authored-by: fuzzyTew <[email protected]> Co-authored-by: Chris <[email protected]> Co-authored-by: Chris Thompson <[email protected]> Co-authored-by: raldone01 <[email protected]> Co-authored-by: Alex <[email protected]>
- Loading branch information
Showing
60 changed files
with
7,344 additions
and
814 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| MIT License | ||
|
|
||
| Copyright (c) 2018 wmhilton | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,290 @@ | ||
| # @kie-tools/kie-sandbox-fs | ||
|
|
||
| This package is a fork of [LightningFS](https://github.com/isomorphic-git/lightning-fs). | ||
|
|
||
| Main differences so far: | ||
|
|
||
| - No HTTP capabilities | ||
| - Bulk operations on IndexedDB using Dexie. See the [new APIs](./src/index.d.ts); | ||
| - Allows in-memory operations for improved performance with some hacks. | ||
|
|
||
| Next steps: | ||
|
|
||
| - Rewrite in TypeScript | ||
| - Expose transactions to allow for more complex use-cases | ||
| - Improve `autoinc` performance | ||
| - Re-evaluate the debouncing mechanism for writing the superblock. | ||
| - Revisit the locking mechanism and remove the obsolete version of it. | ||
| - Include in-memory capabilities inside the FS itself. | ||
|
|
||
| Below is the original README.md. | ||
|
|
||
| --- | ||
|
|
||
| # @isomorphic-git/lightning-fs | ||
|
|
||
| A lean and fast 'fs' for the browser | ||
|
|
||
| ## Motivation | ||
|
|
||
| I wanted to see if I could make something faster than [BrowserFS](https://github.com/jvilk/BrowserFS) or [filer](https://github.com/filerjs/filer) that still implements enough of the `fs` API to run the [`isomorphic-git`](https://github.com/isomorphic-git/isomorphic-git) test suite in browsers. | ||
|
|
||
| ## Comparison with other libraries | ||
|
|
||
| This library does not even come close to implementing the full [`fs`](https://nodejs.org/api/fs.html) API. | ||
| Instead, it only implements [the subset used by isomorphic-git 'fs' plugin interface](https://isomorphic-git.org/docs/en/plugin_fs) plus the [`fs.promises`](https://nodejs.org/dist/latest-v10.x/docs/api/fs.html#fs_fs_promises_api) versions of those functions. | ||
|
|
||
| Unlike BrowserFS, which has a dozen backends and is highly configurable, `lightning-fs` has a single configuration that should Just Work for most users. | ||
|
|
||
| ## Philosophy | ||
|
|
||
| ### Basic requirements: | ||
|
|
||
| 1. needs to work in all modern browsers | ||
| 2. needs to work with large-ish files and directories | ||
| 3. needs to persist data | ||
| 4. needs to enable performant web apps | ||
|
|
||
| Req #3 excludes pure in-memory solutions. Req #4 excludes `localStorage` because it blocks the DOM and cannot be run in a webworker. Req #1 excludes WebSQL and Chrome's FileSystem API. So that leaves us with IndexedDB as the only usable storage technology. | ||
|
|
||
| ### Optimization targets (in order of priority): | ||
|
|
||
| 1. speed (time it takes to execute file system operations) | ||
| 2. bundle size (time it takes to download the library) | ||
| 3. memory usage (will it work on mobile) | ||
|
|
||
| In order to get improve #1, I ended up making a hybrid in-memory / IndexedDB system: | ||
|
|
||
| - `mkdir`, `rmdir`, `readdir`, `rename`, and `stat` are pure in-memory operations that take 0ms | ||
| - `writeFile`, `readFile`, and `unlink` are throttled by IndexedDB | ||
|
|
||
| The in-memory portion of the filesystem is persisted to IndexedDB with a debounce of 500ms. | ||
| The files themselves are not currently cached in memory, because I don't want to waste a lot of memory. | ||
| Applications can always _add_ an LRU cache on top of `lightning-fs` - if I add one internally and it isn't tuned well for your application, it might be much harder to work around. | ||
|
|
||
| ### Multi-threaded filesystem access | ||
|
|
||
| Multiple tabs (and web workers) can share a filesystem. However, because SharedArrayBuffer is still not available in most browsers, the in-memory cache that makes KieSandboxFs fast cannot be shared. If each thread was allowed to update its cache independently, then you'd have a complex distributed system and would need a fancy algorithm to resolve conflicts. Instead, I'm counting on the fact that your multi-threaded applications will NOT be IO bound, and thus a simpler strategy for sharing the filesystem will work. Filesystem access is bottlenecked by a mutex (implemented via polling and an atomic compare-and-replace operation in IndexedDB) to ensure that only one thread has access to the filesystem at a time. If the active thread is constantly using the filesystem, no other threads will get a chance. However if the active thread's filesystem goes idle - no operations are pending and no new operations are started - then after 500ms its in-memory cache is serialized and saved to IndexedDB and the mutex is released. (500ms was chosen experimentally such that an [isomorphic-git](https://github.com/isomorphic-git/isomorphic-git) `clone` operation didn't thrash the mutex.) | ||
|
|
||
| While the mutex is being held by another thread, any fs operations will be stuck waiting until the mutex becomes available. If the mutex is not available even after ten minutes then the filesystem operations will fail with an error. This could happen if say, you are trying to write to a log file every 100ms. You can overcome this by making sure that the filesystem is allowed to go idle for >500ms every now and then. | ||
|
|
||
| ## Usage | ||
|
|
||
| ### `new FS(name, opts?)` | ||
|
|
||
| First, create or open a "filesystem". (The name is used to determine the IndexedDb store name.) | ||
|
|
||
| ```js | ||
| import FS from "@isomorphic-git/lightning-fs"; | ||
|
|
||
| const fs = new FS("testfs"); | ||
| ``` | ||
|
|
||
| **Note: It is better not to create multiple `FS` instances using the same name in a single thread.** Memory usage will be higher as each instance maintains its own cache, and throughput may be lower as each instance will have to compete over the mutex for access to the IndexedDb store. | ||
|
|
||
| Options object: | ||
|
|
||
| | Param | Type [= default] | Description | | ||
| | --------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ||
| | `wipe` | boolean = false | Delete the database and start with an empty filesystem | | ||
| | `url` | string = undefined | Let `readFile` requests fall back to an HTTP request to this base URL | | ||
| | `urlauto` | boolean = false | Fall back to HTTP for every read of a missing file, even if unbacked | | ||
| | `fileDbName` | string | Customize the database name | | ||
| | `fileStoreName` | string | Customize the store name | | ||
| | `lockDbName` | string | Customize the database name for the lock mutex | | ||
| | `lockStoreName` | string | Customize the store name for the lock mutex | | ||
| | `defer` | boolean = false | If true, avoids mutex contention during initialization | | ||
| | `backend` | IBackend | If present, none of the other arguments (except `defer`) have any effect, and instead of using the normal KieSandboxFs stuff, KieSandboxFs acts as a wrapper around the provided custom backend. | | ||
|
|
||
| #### Advanced usage | ||
|
|
||
| You can procrastinate initializing the FS object until later. | ||
| And, if you're really adventurous, you can _re-initialize_ it with a different name to switch between IndexedDb databases. | ||
|
|
||
| ```js | ||
| import FS from "@isomorphic-git/lightning-fs"; | ||
|
|
||
| const fs = new FS(); | ||
|
|
||
| // Some time later... | ||
| fs.init(name, options); | ||
|
|
||
| // Some time later... | ||
| fs.init(different_name, different_options); | ||
| ``` | ||
|
|
||
| ### `fs.mkdir(filepath, opts?, cb)` | ||
|
|
||
| Make directory | ||
|
|
||
| Options object: | ||
|
|
||
| | Param | Type [= default] | Description | | ||
| | ------ | ---------------- | ---------------------- | | ||
| | `mode` | number = 0o777 | Posix mode permissions | | ||
|
|
||
| ### `fs.rmdir(filepath, opts?, cb)` | ||
|
|
||
| Remove directory | ||
|
|
||
| ### `fs.readdir(filepath, opts?, cb)` | ||
|
|
||
| Read directory | ||
|
|
||
| The callback return value is an Array of strings. NOTE: _To save time, it is NOT SORTED._ (Fun fact: Node.js' `readdir` output is not guaranteed to be sorted either. I learned that the hard way.) | ||
|
|
||
| ### `fs.writeFile(filepath, data, opts?, cb)` | ||
|
|
||
| `data` should be a string of a Uint8Array. | ||
|
|
||
| If `opts` is a string, it is interpreted as `{ encoding: opts }`. | ||
|
|
||
| Options object: | ||
|
|
||
| | Param | Type [= default] | Description | | ||
| | ---------- | ------------------ | -------------------------------- | | ||
| | `mode` | number = 0o777 | Posix mode permissions | | ||
| | `encoding` | string = undefined | Only supported value is `'utf8'` | | ||
|
|
||
| ### `fs.readFile(filepath, opts?, cb)` | ||
|
|
||
| The result value will be a Uint8Array or (if `encoding` is `'utf8'`) a string. | ||
|
|
||
| If `opts` is a string, it is interpreted as `{ encoding: opts }`. | ||
|
|
||
| Options object: | ||
|
|
||
| | Param | Type [= default] | Description | | ||
| | ---------- | ------------------ | -------------------------------- | | ||
| | `encoding` | string = undefined | Only supported value is `'utf8'` | | ||
|
|
||
| ### `fs.unlink(filepath, opts?, cb)` | ||
|
|
||
| Delete a file | ||
|
|
||
| ### `fs.rename(oldFilepath, newFilepath, cb)` | ||
|
|
||
| Rename a file or directory | ||
|
|
||
| ### `fs.stat(filepath, opts?, cb)` | ||
|
|
||
| The result is a Stat object similar to the one used by Node but with fewer and slightly different properties and methods. | ||
| The included properties are: | ||
|
|
||
| - `type` ("file" or "dir") | ||
| - `mode` | ||
| - `size` | ||
| - `ino` | ||
| - `mtimeMs` | ||
| - `ctimeMs` | ||
| - `uid` (fixed value of 1) | ||
| - `gid` (fixed value of 1) | ||
| - `dev` (fixed value of 1) | ||
|
|
||
| The included methods are: | ||
|
|
||
| - `isFile()` | ||
| - `isDirectory()` | ||
| - `isSymbolicLink()` | ||
|
|
||
| ### `fs.lstat(filepath, opts?, cb)` | ||
|
|
||
| Like `fs.stat` except that paths to symlinks return the symlink stats not the file stats of the symlink's target. | ||
|
|
||
| ### `fs.symlink(target, filepath, cb)` | ||
|
|
||
| Create a symlink at `filepath` that points to `target`. | ||
|
|
||
| ### `fs.readlink(filepath, opts?, cb)` | ||
|
|
||
| Read the target of a symlink. | ||
|
|
||
| ### `fs.backFile(filepath, opts?, cb)` | ||
|
|
||
| Create or change the stat data for a file backed by HTTP. Size is fetched with a HEAD request. Useful when using an HTTP backend without `urlauto` set, as then files will only be readable if they have stat data. | ||
| Note that stat data is made automatically from the file `/.superblock.txt` if found on the server. `/.superblock.txt` can be generated or updated with the [included standalone script](src/superblocktxt.js). | ||
|
|
||
| Options object: | ||
|
|
||
| | Param | Type [= default] | Description | | ||
| | ------ | ---------------- | ---------------------- | | ||
| | `mode` | number = 0o666 | Posix mode permissions | | ||
|
|
||
| ### `fs.du(filepath, cb)` | ||
|
|
||
| Returns the size of a file or directory in bytes. | ||
|
|
||
| ### `fs.promises` | ||
|
|
||
| All the same functions as above, but instead of passing a callback they return a promise. | ||
|
|
||
| ## Providing a custom `backend` (advanced usage) | ||
|
|
||
| There are only two reasons I can think of that you would want to do this: | ||
|
|
||
| 1. The `fs` module is normally a singleton. KieSandboxFs allows you to safely(ish) hotswap between various data sources by calling `init` multiple times with different options. (It keeps track of file system operations in flight and waits until there's an idle moment to do the switch.) | ||
|
|
||
| 2. KieSandboxFs normalizes all the lovely variations of node's `fs` arguments: | ||
|
|
||
| - `fs.writeFile('filename.txt', 'Hello', cb)` | ||
| - `fs.writeFile('filename.txt', 'Hello', 'utf8', cb)` | ||
| - `fs.writeFile('filename.txt', 'Hello', { encoding: 'utf8' }, cb)` | ||
| - `fs.promises.writeFile('filename.txt', 'Hello')` | ||
| - `fs.promises.writeFile('filename.txt', 'Hello', 'utf8')` | ||
| - `fs.promises.writeFile('filename.txt', 'Hello', { encoding: 'utf8' })` | ||
|
|
||
| And it normalizes filepaths. And will convert plain `StatLike` objects into `Stat` objects with methods like `isFile`, `isDirectory`, etc. | ||
|
|
||
| If that fits your needs, then you can provide a `backend` option and KieSandboxFs will use that. Implement as few/many methods as you need for your application to work. | ||
|
|
||
| **Note:** If you use a custom backend, you are responsible for managing multi-threaded access - there are no magic mutexes included by default. | ||
|
|
||
| Note: throwing an error with the correct `.code` property for any given situation is often important for utilities like `mkdirp` and `rimraf` to work. | ||
|
|
||
| ```tsx | ||
| type EncodingOpts = { | ||
| encoding?: "utf8"; | ||
| }; | ||
|
|
||
| type StatLike = { | ||
| type: "file" | "dir" | "symlink"; | ||
| mode: number; | ||
| size: number; | ||
| ino: number | string | BigInt; | ||
| mtimeMs: number; | ||
| ctimeMs?: number; | ||
| }; | ||
|
|
||
| interface IBackend { | ||
| // highly recommended - usually necessary for apps to work | ||
| readFile(filepath: string, opts: EncodingOpts): Awaited<Uint8Array | string>; // throws ENOENT | ||
| writeFile(filepath: string, data: Uint8Array | string, opts: EncodingOpts): void; // throws ENOENT | ||
| unlink(filepath: string, opts: any): void; // throws ENOENT | ||
| readdir(filepath: string, opts: any): Awaited<string[]>; // throws ENOENT, ENOTDIR | ||
| mkdir(filepath: string, opts: any): void; // throws ENOENT, EEXIST | ||
| rmdir(filepath: string, opts: any): void; // throws ENOENT, ENOTDIR, ENOTEMPTY | ||
|
|
||
| // recommended - often necessary for apps to work | ||
| stat(filepath: string, opts: any): Awaited<StatLike>; // throws ENOENT | ||
| lstat(filepath: string, opts: any): Awaited<StatLike>; // throws ENOENT | ||
|
|
||
| // suggested - used occasionally by apps | ||
| rename(oldFilepath: string, newFilepath: string): void; // throws ENOENT | ||
| readlink(filepath: string, opts: any): Awaited<string>; // throws ENOENT | ||
| symlink(target: string, filepath: string): void; // throws ENOENT | ||
|
|
||
| // bonus - not part of the standard `fs` module | ||
| backFile(filepath: string, opts: any): void; | ||
| du(filepath: string): Awaited<number>; | ||
|
|
||
| // lifecycle - useful if your backend needs setup and teardown | ||
| init?(name: string, opts: any): Awaited<void>; // passes initialization options | ||
| activate?(): Awaited<void>; // called before fs operations are started | ||
| deactivate?(): Awaited<void>; // called after fs has been idle for a while | ||
| destroy?(): Awaited<void>; // called before hotswapping backends | ||
| } | ||
| ``` | ||
|
|
||
| ## License | ||
|
|
||
| MIT |
Oops, something went wrong.