Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Polish opendal rust core README #4745

Merged
merged 1 commit into from
Jun 17, 2024
Merged

docs: Polish opendal rust core README #4745

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,6 @@ OpenDAL offers a unified data access layer, empowering users to seamlessly and e
| Cache Storage Services | [ghac] [memcached] [mini_moka] [moka] [vercel_artifacts] |
| Git Based Storage Services | [huggingface] |

> Welcome to add any services that are not currently supported [here](https://github.com/apache/opendal/issues/5).

[sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02
[webdav]: https://datatracker.ietf.org/doc/html/rfc4918

Expand Down
243 changes: 145 additions & 98 deletions core/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Apache OpenDAL™
# Apache OpenDAL™: *Access Data Freely*

[![Build Status]][actions] [![Latest Version]][crates.io] [![Crate Downloads]][crates.io] [![chat]][discord]

Expand All @@ -10,108 +10,149 @@
[chat]: https://img.shields.io/discord/1081052318650339399
[discord]: https://opendal.apache.org/discord

**OpenDAL** is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
OpenDAL offers a unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services.

- Documentation: [stable](https://docs.rs/opendal/) | [main](https://opendal.apache.org/docs/rust/opendal/)
- [Release notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html)

![](https://github.com/apache/opendal/assets/5351546/87bbf6e5-f19e-449a-b368-3e283016c887)

## Services

<details>
<summary>Standard Storage Protocols (like ftp, webdav)</summary>

- ftp: FTP and FTPS
- http: HTTP read-only services
- sftp: [SFTP](https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02) services *being worked on*
- webdav: [WebDAV](https://datatracker.ietf.org/doc/html/rfc4918) Service

</details>

<details>
<summary>Object Storage Services (like s3, gcs, azblob)</summary>

- azblob: [Azure Storage Blob](https://azure.microsoft.com/en-us/services/storage/blobs/) services
- cos: [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) services
- gcs: [Google Cloud Storage](https://cloud.google.com/storage) Service
- obs: [Huawei Cloud Object Storage](https://www.huaweicloud.com/intl/en-us/product/obs.html) Service (OBS)
- oss: [Aliyun Object Storage Service](https://www.aliyun.com/product/oss) (OSS)
- s3: [AWS S3](https://aws.amazon.com/s3/) alike services
- supabase: [Supabase Storage](https://supabase.com/docs/guides/storage) Service *being worked on*

</details>

<details>
<summary>File Storage Services (like fs, azdls, hdfs)</summary>

- fs: POSIX alike file system
- azdls: [Azure Data Lake Storage Gen2](https://azure.microsoft.com/en-us/products/storage/data-lake-storage/) services (As known as [abfs](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver))
- hdfs: [Hadoop Distributed File System](https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)(HDFS)
- ipfs: [InterPlanetary File System](https://ipfs.tech/) HTTP Gateway
- ipmfs: [InterPlanetary File System](https://ipfs.tech/) MFS API *being worked on*
- webhdfs: [WebHDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html) Service

</details>

<details>
<summary>Consumer Cloud Storage Service (like gdrive, onedrive, icloud drive)</summary>

- gdrive: [Google Drive](https://www.google.com/drive/) *being worked on*
- onedrive: [OneDrive](https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage) *being worked on*
- icloud: [Icloud Drive](https://www.icloud.com.cn/iclouddrive/) *being worked on*

</details>
<img src="https://opendal.apache.org/img/architectural.png" alt="OpenDAL Architectural" width="61.8%" />

<details>
<summary>Key-Value Storage Service (like rocksdb, sled)</summary>
## Useful Links

- cacache: [cacache](https://crates.io/crates/cacache) backend
- dashmap: [dashmap](https://github.com/xacrimon/dashmap) backend
- memory: In memory backend
- persy: [persy](https://crates.io/crates/persy) backend
- redis: [Redis](https://redis.io/) services
- rocksdb: [RocksDB](http://rocksdb.org/) services
- sled: [sled](https://crates.io/crates/sled) backend
- redb: [redb](https://crates.io/crates/redb) backend
- tikv: [tikv](https://tikv.org/) backend
- atomicserver: [Atomicserver](https://github.com/atomicdata-dev/atomic-server) services

</details>

<details>
<summary>Cache Storage Service (like memcached, moka)</summary>

- ghac: [GitHub Action Cache](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows) Service
- memcached: [Memcached](https://memcached.org/) service
- mini_moka: [Mini Moka](https://github.com/moka-rs/mini-moka) backend
- moka: [Moka](https://github.com/moka-rs/moka) backend
- vercel_artifacts: [Vercel Remote Caching](https://vercel.com/docs/concepts/monorepos/remote-caching) Service *being worked on*

</details>

> Welcome to add any services that are not currently supported [here](https://github.com/apache/opendal/issues/5).

## Features

Access data **freely**

- Access different storage services in the same way
- Behavior tests for all services

Access data **painlessly**

- **100%** documents covered
- Powerful [`Layers`](https://docs.rs/opendal/latest/opendal/layers/index.html)
- Automatic [retry](https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html) support
- Full observability: [logging](https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html), [tracing](https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html), [metrics](https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html).
- [Native chaos testing](https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html)
- Documentation: [stable](https://docs.rs/opendal/) | [main](https://opendal.apache.org/docs/rust/opendal/)
- [Release Notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html)
- [Upgrade Guide](https://docs.rs/opendal/latest/opendal/docs/upgrade/index.html)
- [RFC List](https://docs.rs/opendal/latest/opendal/docs/rfcs/index.html)

Access data **efficiently**
## Services

- Zero cost: Maps to API calls directly
- Best effort: Automatically selects best read/seek/next based on services
- Avoid extra calls: Reuses metadata when possible
OpenDAL supports the following storage [services](https://docs.rs/opendal/latest/opendal/services/index.html):

| Type | Services |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| Standard Storage Protocols | ftp http [sftp] [webdav] |
| Object Storage Services | [azblob] [cos] [gcs] [obs] [oss] [s3] <br> [b2] [openstack_swift] [upyun] [vercel_blob] |
| File Storage Services | fs [alluxio] [azdls] [azfile] [chainsafe] [compfs] <br> [dbfs] [gridfs] [hdfs] [hdfs_native] [ipfs] [webhdfs] |
| Consumer Cloud Storage Service | [aliyun_drive] [gdrive] [onedrive] [dropbox] [icloud] [koofr] <br> [pcloud] [seafile] [yandex_disk] |
| Key-Value Storage Services | [cacache] [cloudflare_kv] [dashmap] memory [etcd] <br> [foundationdb] [persy] [redis] [rocksdb] [sled] <br> [redb] [tikv] [atomicserver] |
| Database Storage Services | [d1] [mongodb] [mysql] [postgresql] [sqlite] [surrealdb] |
| Cache Storage Services | [ghac] [memcached] [mini_moka] [moka] [vercel_artifacts] |
| Git Based Storage Services | [huggingface] |

[sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02
[webdav]: https://datatracker.ietf.org/doc/html/rfc4918

[azblob]: https://azure.microsoft.com/en-us/services/storage/blobs/
[cos]: https://www.tencentcloud.com/products/cos
[gcs]: https://cloud.google.com/storage
[obs]: https://www.huaweicloud.com/intl/en-us/product/obs.html
[oss]: https://www.aliyun.com/product/oss
[s3]: https://aws.amazon.com/s3/
[b2]: https://www.backblaze.com/
[openstack_swift]: https://docs.openstack.org/swift/latest/
[upyun]: https://www.upyun.com/
[vercel_blob]: https://vercel.com/docs/storage/vercel-blob

[alluxio]: https://docs.alluxio.io/os/user/stable/en/api/REST-API.html
[azdls]: https://azure.microsoft.com/en-us/products/storage/data-lake-storage/
[azfile]: https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api
[chainsafe]: https://storage.chainsafe.io/
[compfs]: https://github.com/compio-rs/compio/
[dbfs]: https://docs.databricks.com/en/dbfs/index.html
[gridfs]: https://www.mongodb.com/docs/manual/core/gridfs/
[hdfs]: https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
[hdfs_native]: https://github.com/Kimahriman/hdfs-native
[ipfs]: https://ipfs.tech/
[webhdfs]: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

[aliyun_drive]: https://www.aliyundrive.com/
[gdrive]: https://www.google.com/drive/
[onedrive]: https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage
[dropbox]: https://www.dropbox.com/
[icloud]: https://www.icloud.com/iclouddrive
[koofr]: https://koofr.eu/
[pcloud]: https://www.pcloud.com/
[seafile]: https://www.seafile.com/
[yandex_disk]: https://360.yandex.com/disk/

[cacache]: https://crates.io/crates/cacache
[cloudflare_kv]: https://developers.cloudflare.com/kv/
[dashmap]: https://github.com/xacrimon/dashmap
[etcd]: https://etcd.io/
[foundationdb]: https://www.foundationdb.org/
[persy]: https://crates.io/crates/persy
[redis]: https://redis.io/
[rocksdb]: http://rocksdb.org/
[sled]: https://crates.io/crates/sled
[redb]: https://crates.io/crates/redb
[tikv]: https://tikv.org/
[atomicserver]: https://github.com/atomicdata-dev/atomic-server

[d1]: https://developers.cloudflare.com/d1/
[mongodb]: https://www.mongodb.com/
[mysql]: https://www.mysql.com/
[postgresql]: https://www.postgresql.org/
[sqlite]: https://www.sqlite.org/
[surrealdb]: https://surrealdb.com/

[ghac]: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
[memcached]: https://memcached.org/
[mini_moka]: https://github.com/moka-rs/mini-moka
[moka]: https://github.com/moka-rs/moka
[vercel_artifacts]: https://vercel.com/docs/concepts/monorepos/remote-caching

[huggingface]: https://huggingface.co/

## Layers

OpenDAL supports the following storage [layers](https://docs.rs/opendal/latest/opendal/layers/index.html) to extend the behavior:

| Name | Depends | Description |
|---------------------------|------------------------|---------------------------------------------------------------------------------------|
| [`AsyncBacktraceLayer`] | [async-backtrace] | Add Efficient, logical 'stack' traces of async functions for the underlying services. |
| [`AwaitTreeLayer`] | [await-tree] | Add a Instrument await-tree for actor-based applications to the underlying services. |
| [`BlockingLayer`] | [tokio] | Add blocking API support for non-blocking services. |
| [`ChaosLayer`] | [rand] | Inject chaos into underlying services for robustness test. |
| [`ConcurrentLimitLayer`] | [tokio] | Add concurrent request limit. |
| [`DtraceLayer`] | [probe] | Support User Statically-Defined Tracing(aka USDT) on Linux |
| [`LoggingLayer`] | [log] | Add log for every operations. |
| [`MetricsLayer`] | [metrics] | Add metrics for every operations. |
| [`MinitraceLayer`] | [minitrace] | Add minitrace for every operations. |
| [`OtelTraceLayer`] | [opentelemetry::trace] | Add opentelemetry::trace for every operations. |
| [`PrometheusClientLayer`] | [prometheus_client] | Add prometheus metrics for every operations. |
| [`PrometheusLayer`] | [prometheus] | Add prometheus metrics for every operations. |
| [`RetryLayer`] | [backon] | Add retry for temporary failed operations. |
| [`ThrottleLayer`] | [governor] | Add a bandwidth rate limiter to the underlying services. |
| [`TimeoutLayer`] | [tokio] | Add timeout for every operations to avoid slow or unexpected hang operations. |
| [`TracingLayer`] | [tracing] | Add tracing for every operations. |

[`AsyncBacktraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.AsyncBacktraceLayer.html
[async-backtrace]: https://github.com/tokio-rs/async-backtrace
[`AwaitTreeLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.AwaitTreeLayer.html
[await-tree]: https://github.com/risingwavelabs/await-tree
[`BlockingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.BlockingLayer.html
[tokio]: https://github.com/tokio-rs/tokio
[`ChaosLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html
[rand]: https://github.com/rust-random/rand
[`ConcurrentLimitLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ConcurrentLimitLayer.html
[`DtraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.DtraceLayer.html
[probe]: https://github.com/cuviper/probe-rs
[`LoggingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html
[log]: https://github.com/rust-lang/log
[`MetricsLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html
[metrics]: https://github.com/metrics-rs/metrics
[`MinitraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.MinitraceLayer.html
[minitrace]: https://github.com/tikv/minitrace-rust
[`OtelTraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.OtelTraceLayer.html
[opentelemetry::trace]: https://docs.rs/opentelemetry/latest/opentelemetry/trace/index.html
[`PrometheusClientLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusClientLayer.html
[prometheus_client]: https://github.com/prometheus/client_rust
[`PrometheusLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusLayer.html
[prometheus]: https://github.com/tikv/rust-prometheus
[`RetryLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html
[backon]: https://github.com/Xuanwo/backon
[`ThrottleLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ThrottleLayer.html
[governor]: https://github.com/boinkor-net/governor
[`TimeoutLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.TimeoutLayer.html
[`TracingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html
[tracing]: https://github.com/tokio-rs/tracing

## Quickstart

Expand Down Expand Up @@ -159,6 +200,12 @@ The examples are available at [here](../examples/rust).

Check out the [CONTRIBUTING](CONTRIBUTING.md) guide for more details on getting started with contributing to this project.

## Branding

The first and most prominent mentions must use the full form: **Apache OpenDAL™** of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.

For more details, see the [Apache Product Name Usage Guide](https://www.apache.org/foundation/marks/guide).

## License and Trademarks

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
Expand Down
Loading