Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve local storage of assets & cached image files #258

Closed
michaelbromley opened this issue Feb 7, 2020 · 0 comments
Closed

Improve local storage of assets & cached image files #258

michaelbromley opened this issue Feb 7, 2020 · 0 comments
Assignees

Comments

@michaelbromley
Copy link
Member

Is your feature request related to a problem? Please describe.
Currently, when using the AssetServerPlugin, it will use the LocalAssetStorageStrategy which is part of Vendure core. This storage strategy dumps all assets into a single directory.

For a very large number of files, this approach can have performance implications - thousands of files in a directory can be slower to read.

Describe the solution you'd like
We can use an approach known as a "hashed directory structure" in which a hash is made from the filename (e.g. md5) and then this hash is used to place the file in a sub-directory structure according to the first part of the hash. This is e.g. how git stores files in its /objects directory.

A reasonable way to store a small multitude of files on a file system is through a hashed directory structure with details on the files kept in a database table. One advantage of this method is that it keeps directory reads and file access times quick by not putting too many files into a single directory. source

I propose the following:

  1. Create a new HashedDirectoryStorageStrategy
  2. Make the storage strategy in the AssetServerPlugin configurable. This allows users to continue using the existing LocalAssetStorageStrategy to avoid breaking existing projects.
  3. Set the new HashedDirectoryStorageStrategy as the default in the AssetServerPlugin.
  4. Use the same approach for storing cached images - this can be hard-coded as it is an internal implementation detail of the AssetServerPlugin.
@michaelbromley michaelbromley self-assigned this Feb 7, 2020
michaelbromley added a commit that referenced this issue Feb 13, 2020
Relates to #258. The developer can now provide a custom AssetStorageStrategy e.g. to store files with a cloud storage provider.

BREAKING CHANGE: The `LocalAssetStorageStrategy` class has been removed from `@vendure/core` and now lives in the `@vendure/asset-server-plugin` package.
michaelbromley added a commit that referenced this issue Feb 13, 2020
Relates to #258. This strategy for storing files limits the total number of files in a given directory, which otherwise can impact performance when the size grows too large.
michaelbromley added a commit that referenced this issue Feb 13, 2020
Relates to #258.

BREAKING CHANGE: The AssetServerPlugin has a new default naming strategy - instead of dumping all assets & previews into a single directory, it will now split sources & previews into subdirectories and in each of them will use hashed directories to ensure that the total number of files in a single directory does not grow too large (as this can have a negative performance impact). If you wish to keep the current behavior, then you must manually set the `namingStrategy: new DefaultAssetNamingStrategy()` in the `AssetServerPlugin.init()` method.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant