diff --git a/README.md b/README.md index 5a8260a1393..7e45c69cf63 100644 --- a/README.md +++ b/README.md @@ -115,8 +115,6 @@ OpenDAL offers a unified data access layer, empowering users to seamlessly and e | Cache Storage Services | [ghac] [memcached] [mini_moka] [moka] [vercel_artifacts] | | Git Based Storage Services | [huggingface] | -> Welcome to add any services that are not currently supported [here](https://github.com/apache/opendal/issues/5). - [sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02 [webdav]: https://datatracker.ietf.org/doc/html/rfc4918 diff --git a/core/README.md b/core/README.md index 2b72a3fd965..6b577d6064a 100644 --- a/core/README.md +++ b/core/README.md @@ -1,4 +1,4 @@ -# Apache OpenDALâ„¢ +# Apache OpenDALâ„¢: *Access Data Freely* [![Build Status]][actions] [![Latest Version]][crates.io] [![Crate Downloads]][crates.io] [![chat]][discord] @@ -10,108 +10,149 @@ [chat]: https://img.shields.io/discord/1081052318650339399 [discord]: https://opendal.apache.org/discord -**OpenDAL** is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. +OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services. -- Documentation: [stable](https://docs.rs/opendal/) | [main](https://opendal.apache.org/docs/rust/opendal/) -- [Release notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html) - -![](https://github.com/apache/opendal/assets/5351546/87bbf6e5-f19e-449a-b368-3e283016c887) - -## Services - -
-Standard Storage Protocols (like ftp, webdav) - -- ftp: FTP and FTPS -- http: HTTP read-only services -- sftp: [SFTP](https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02) services *being worked on* -- webdav: [WebDAV](https://datatracker.ietf.org/doc/html/rfc4918) Service - -
- -
-Object Storage Services (like s3, gcs, azblob) - -- azblob: [Azure Storage Blob](https://azure.microsoft.com/en-us/services/storage/blobs/) services -- cos: [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) services -- gcs: [Google Cloud Storage](https://cloud.google.com/storage) Service -- obs: [Huawei Cloud Object Storage](https://www.huaweicloud.com/intl/en-us/product/obs.html) Service (OBS) -- oss: [Aliyun Object Storage Service](https://www.aliyun.com/product/oss) (OSS) -- s3: [AWS S3](https://aws.amazon.com/s3/) alike services -- supabase: [Supabase Storage](https://supabase.com/docs/guides/storage) Service *being worked on* - -
- -
-File Storage Services (like fs, azdls, hdfs) - -- fs: POSIX alike file system -- azdls: [Azure Data Lake Storage Gen2](https://azure.microsoft.com/en-us/products/storage/data-lake-storage/) services (As known as [abfs](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver)) -- hdfs: [Hadoop Distributed File System](https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)(HDFS) -- ipfs: [InterPlanetary File System](https://ipfs.tech/) HTTP Gateway -- ipmfs: [InterPlanetary File System](https://ipfs.tech/) MFS API *being worked on* -- webhdfs: [WebHDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html) Service - -
- -
-Consumer Cloud Storage Service (like gdrive, onedrive, icloud drive) - -- gdrive: [Google Drive](https://www.google.com/drive/) *being worked on* -- onedrive: [OneDrive](https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage) *being worked on* -- icloud: [Icloud Drive](https://www.icloud.com.cn/iclouddrive/) *being worked on* - -
+OpenDAL Architectural -
-Key-Value Storage Service (like rocksdb, sled) +## Useful Links -- cacache: [cacache](https://crates.io/crates/cacache) backend -- dashmap: [dashmap](https://github.com/xacrimon/dashmap) backend -- memory: In memory backend -- persy: [persy](https://crates.io/crates/persy) backend -- redis: [Redis](https://redis.io/) services -- rocksdb: [RocksDB](http://rocksdb.org/) services -- sled: [sled](https://crates.io/crates/sled) backend -- redb: [redb](https://crates.io/crates/redb) backend -- tikv: [tikv](https://tikv.org/) backend -- atomicserver: [Atomicserver](https://github.com/atomicdata-dev/atomic-server) services - -
- -
-Cache Storage Service (like memcached, moka) - -- ghac: [GitHub Action Cache](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows) Service -- memcached: [Memcached](https://memcached.org/) service -- mini_moka: [Mini Moka](https://github.com/moka-rs/mini-moka) backend -- moka: [Moka](https://github.com/moka-rs/moka) backend -- vercel_artifacts: [Vercel Remote Caching](https://vercel.com/docs/concepts/monorepos/remote-caching) Service *being worked on* - -
- -> Welcome to add any services that are not currently supported [here](https://github.com/apache/opendal/issues/5). - -## Features - -Access data **freely** - -- Access different storage services in the same way -- Behavior tests for all services - -Access data **painlessly** - -- **100%** documents covered -- Powerful [`Layers`](https://docs.rs/opendal/latest/opendal/layers/index.html) -- Automatic [retry](https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html) support -- Full observability: [logging](https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html), [tracing](https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html), [metrics](https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html). -- [Native chaos testing](https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html) +- Documentation: [stable](https://docs.rs/opendal/) | [main](https://opendal.apache.org/docs/rust/opendal/) +- [Release Notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html) +- [Upgrade Guide](https://docs.rs/opendal/latest/opendal/docs/upgrade/index.html) +- [RFC List](https://docs.rs/opendal/latest/opendal/docs/rfcs/index.html) -Access data **efficiently** +## Services -- Zero cost: Maps to API calls directly -- Best effort: Automatically selects best read/seek/next based on services -- Avoid extra calls: Reuses metadata when possible +OpenDAL supports the following storage [services](https://docs.rs/opendal/latest/opendal/services/index.html): + +| Type | Services | +|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| +| Standard Storage Protocols | ftp http [sftp] [webdav] | +| Object Storage Services | [azblob] [cos] [gcs] [obs] [oss] [s3]
[b2] [openstack_swift] [upyun] [vercel_blob] | +| File Storage Services | fs [alluxio] [azdls] [azfile] [chainsafe] [compfs]
[dbfs] [gridfs] [hdfs] [hdfs_native] [ipfs] [webhdfs] | +| Consumer Cloud Storage Service | [aliyun_drive] [gdrive] [onedrive] [dropbox] [icloud] [koofr]
[pcloud] [seafile] [yandex_disk] | +| Key-Value Storage Services | [cacache] [cloudflare_kv] [dashmap] memory [etcd]
[foundationdb] [persy] [redis] [rocksdb] [sled]
[redb] [tikv] [atomicserver] | +| Database Storage Services | [d1] [mongodb] [mysql] [postgresql] [sqlite] [surrealdb] | +| Cache Storage Services | [ghac] [memcached] [mini_moka] [moka] [vercel_artifacts] | +| Git Based Storage Services | [huggingface] | + +[sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02 +[webdav]: https://datatracker.ietf.org/doc/html/rfc4918 + +[azblob]: https://azure.microsoft.com/en-us/services/storage/blobs/ +[cos]: https://www.tencentcloud.com/products/cos +[gcs]: https://cloud.google.com/storage +[obs]: https://www.huaweicloud.com/intl/en-us/product/obs.html +[oss]: https://www.aliyun.com/product/oss +[s3]: https://aws.amazon.com/s3/ +[b2]: https://www.backblaze.com/ +[openstack_swift]: https://docs.openstack.org/swift/latest/ +[upyun]: https://www.upyun.com/ +[vercel_blob]: https://vercel.com/docs/storage/vercel-blob + +[alluxio]: https://docs.alluxio.io/os/user/stable/en/api/REST-API.html +[azdls]: https://azure.microsoft.com/en-us/products/storage/data-lake-storage/ +[azfile]: https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api +[chainsafe]: https://storage.chainsafe.io/ +[compfs]: https://github.com/compio-rs/compio/ +[dbfs]: https://docs.databricks.com/en/dbfs/index.html +[gridfs]: https://www.mongodb.com/docs/manual/core/gridfs/ +[hdfs]: https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html +[hdfs_native]: https://github.com/Kimahriman/hdfs-native +[ipfs]: https://ipfs.tech/ +[webhdfs]: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html + +[aliyun_drive]: https://www.aliyundrive.com/ +[gdrive]: https://www.google.com/drive/ +[onedrive]: https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage +[dropbox]: https://www.dropbox.com/ +[icloud]: https://www.icloud.com/iclouddrive +[koofr]: https://koofr.eu/ +[pcloud]: https://www.pcloud.com/ +[seafile]: https://www.seafile.com/ +[yandex_disk]: https://360.yandex.com/disk/ + +[cacache]: https://crates.io/crates/cacache +[cloudflare_kv]: https://developers.cloudflare.com/kv/ +[dashmap]: https://github.com/xacrimon/dashmap +[etcd]: https://etcd.io/ +[foundationdb]: https://www.foundationdb.org/ +[persy]: https://crates.io/crates/persy +[redis]: https://redis.io/ +[rocksdb]: http://rocksdb.org/ +[sled]: https://crates.io/crates/sled +[redb]: https://crates.io/crates/redb +[tikv]: https://tikv.org/ +[atomicserver]: https://github.com/atomicdata-dev/atomic-server + +[d1]: https://developers.cloudflare.com/d1/ +[mongodb]: https://www.mongodb.com/ +[mysql]: https://www.mysql.com/ +[postgresql]: https://www.postgresql.org/ +[sqlite]: https://www.sqlite.org/ +[surrealdb]: https://surrealdb.com/ + +[ghac]: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows +[memcached]: https://memcached.org/ +[mini_moka]: https://github.com/moka-rs/mini-moka +[moka]: https://github.com/moka-rs/moka +[vercel_artifacts]: https://vercel.com/docs/concepts/monorepos/remote-caching + +[huggingface]: https://huggingface.co/ + +## Layers + +OpenDAL supports the following storage [layers](https://docs.rs/opendal/latest/opendal/layers/index.html) to extend the behavior: + +| Name | Depends | Description | +|---------------------------|------------------------|---------------------------------------------------------------------------------------| +| [`AsyncBacktraceLayer`] | [async-backtrace] | Add Efficient, logical 'stack' traces of async functions for the underlying services. | +| [`AwaitTreeLayer`] | [await-tree] | Add a Instrument await-tree for actor-based applications to the underlying services. | +| [`BlockingLayer`] | [tokio] | Add blocking API support for non-blocking services. | +| [`ChaosLayer`] | [rand] | Inject chaos into underlying services for robustness test. | +| [`ConcurrentLimitLayer`] | [tokio] | Add concurrent request limit. | +| [`DtraceLayer`] | [probe] | Support User Statically-Defined Tracing(aka USDT) on Linux | +| [`LoggingLayer`] | [log] | Add log for every operations. | +| [`MetricsLayer`] | [metrics] | Add metrics for every operations. | +| [`MinitraceLayer`] | [minitrace] | Add minitrace for every operations. | +| [`OtelTraceLayer`] | [opentelemetry::trace] | Add opentelemetry::trace for every operations. | +| [`PrometheusClientLayer`] | [prometheus_client] | Add prometheus metrics for every operations. | +| [`PrometheusLayer`] | [prometheus] | Add prometheus metrics for every operations. | +| [`RetryLayer`] | [backon] | Add retry for temporary failed operations. | +| [`ThrottleLayer`] | [governor] | Add a bandwidth rate limiter to the underlying services. | +| [`TimeoutLayer`] | [tokio] | Add timeout for every operations to avoid slow or unexpected hang operations. | +| [`TracingLayer`] | [tracing] | Add tracing for every operations. | + +[`AsyncBacktraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.AsyncBacktraceLayer.html +[async-backtrace]: https://github.com/tokio-rs/async-backtrace +[`AwaitTreeLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.AwaitTreeLayer.html +[await-tree]: https://github.com/risingwavelabs/await-tree +[`BlockingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.BlockingLayer.html +[tokio]: https://github.com/tokio-rs/tokio +[`ChaosLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html +[rand]: https://github.com/rust-random/rand +[`ConcurrentLimitLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ConcurrentLimitLayer.html +[`DtraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.DtraceLayer.html +[probe]: https://github.com/cuviper/probe-rs +[`LoggingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html +[log]: https://github.com/rust-lang/log +[`MetricsLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html +[metrics]: https://github.com/metrics-rs/metrics +[`MinitraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.MinitraceLayer.html +[minitrace]: https://github.com/tikv/minitrace-rust +[`OtelTraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.OtelTraceLayer.html +[opentelemetry::trace]: https://docs.rs/opentelemetry/latest/opentelemetry/trace/index.html +[`PrometheusClientLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusClientLayer.html +[prometheus_client]: https://github.com/prometheus/client_rust +[`PrometheusLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusLayer.html +[prometheus]: https://github.com/tikv/rust-prometheus +[`RetryLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html +[backon]: https://github.com/Xuanwo/backon +[`ThrottleLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ThrottleLayer.html +[governor]: https://github.com/boinkor-net/governor +[`TimeoutLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.TimeoutLayer.html +[`TracingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html +[tracing]: https://github.com/tokio-rs/tracing ## Quickstart @@ -159,6 +200,12 @@ The examples are available at [here](../examples/rust). Check out the [CONTRIBUTING](CONTRIBUTING.md) guide for more details on getting started with contributing to this project. +## Branding + +The first and most prominent mentions must use the full form: **Apache OpenDALâ„¢** of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization. + +For more details, see the [Apache Product Name Usage Guide](https://www.apache.org/foundation/marks/guide). + ## License and Trademarks Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0