discuss: OpenDAL VISION Polish to "One Layer, All Storage."

[Xuanwo]

This discussion replaced by #5301

---

Hello, everyone. I’m starting this thread as the first part of an ongoing discussion: #5189

As mentioned in that thread:

Our current vision, "access data freely," occasionally confuses our users, as many associate the term "data" with databases, data lakes, and SQL in a broader context. We should refine it to better reflect OpenDAL's objectives.

In light of this, I propose a new vision for the OpenDAL project:

One Layer, All Storage.

This vision is a high-level abstraction of what we aim to achieve. It removes the confusing term data and replaces it with storage to clearly indicate that OpenDAL deals with storage rather than databases or tables (viewing databases as storage is another perspective). It also emphasizes that OpenDAL is a layer, serving as an abstraction layer for storage access, and it can support additional layers on top of it.

This vision could be used as:

OpenDAL: One Layer, All Storage

or in a sentence:

OpenDAL is an Open Data Access Layer that aims to achieve One Layer, All Storage.

---

I personally love this vision and am eager to share it with the whole community.

Please let me know what you think, and feel free to propose your version of the vision.

[flaneur2020]

while we aim to treat all storage types behind a same abstraction, i suppose positioning object storage as a first-class citizen might offers more advantages for branding.

i'd love to live in a world which "all storages are equal, but object storage is more equal than others" 😲

[xxchan]

I think one common confusion is "what is object storage's position in OpenDAL"?
Some users have concern that OpenDAL might deviate from object storage API and will not satisfy their need. So they prefer object_store instead.

I think it can help adoption if we clarify this point.

[erickguan]

From my first read, I feel that “all storage is (somewhat) equal.” Of course, different backend support varies in reality.

“First-class support” and “all storage is equal” are two different things. If I read “first-class support for file systems and object storage,” I understand these two types of storage receive relatively high priority, presumably based on their relative importance. “All storage” could mean different things to me. It could be:

• Supports many services and backends
• Supports a subset of all storage
• Supports storage with different levels of support

I also asked ChatGPT to generate a few suggestions. I prompted file system and object storage, which I don’t like, but food for thought:

1. OpenDAL: Unified Access to Object Storage and Beyond
2. OpenDAL: One Layer for All Storage—From Object to File Systems
3. OpenDAL: Your Gateway to Object and File Storage Solutions
4. OpenDAL: The Unified Layer for First-Class Object Storage and More
5. OpenDAL: One Interface, All Storage—Empowering Object and File Systems
6. OpenDAL: Seamless Integration with Object Storage and Local File Systems

[Xuanwo]

Thank you, @flaneur2020, @A-Stupid-Sun, @xxchan, and @erickguan, for sharing your thoughts.

In the current implementation of OpenDAL, object storage is prioritized. However, I feel that emphasizing this in the vision is not ideal. It's quite hard to convey so much information within the vision.

After this VISION, we will present our tenets. With these tenets, we will describe how OpenDAL implements our VISION. Perhaps we can prioritize object storage there. I plan to discuss those tenets in a separate thread since they have no meaning without an established vision.

[Xuanwo]

Perhaps we should follow a process similar to hyper's Define VISION #2772, which defines vision and tenets simultaneously. It seems we can't simply agree with just four words.

[erickguan]

For what is worth, I'm okay with "One Layer, All Storage".

[Gnosnay]

considering about branding, the following things should be taken into consideration:

• short and easy to remember
• highlight the core value of this library
• target the clients

then how about One Layer, All Storage, With Zero-Cost Abstraction

[morristai]

Or One Layer, All Storage, Zero Overhead?

[Xuanwo]

Thank you @Gnosnay and @morristai for this.

I'm a bit hesitant about adding "zero overhead" to the vision. I feel that "zero overhead" is one of our principles for implementing the vision, not our ultimate goal. OpenDAL doesn't aim to achieve zero overhead; instead, we achieve "one layer, all storage" in a zero overhead manner.

Furthermore, some things are more important than zero overhead. For example, we might add additional error checks to ensure errors are handled correctly instead of failing silently. For example, as we do in

opendal/core/src/services/s3/writer.rs

Lines 163 to 173 in 3146fcc

	StatusCode::OK => {
	// still check if there is any error because S3 might return error for status code 200
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_Example_4
	let (parts, body) = resp.into_parts();
	let maybe_error: S3Error =
	quick_xml::de::from_reader(body.reader()).map_err(new_xml_deserialize_error)?;
	if !maybe_error.code.is_empty() {
	return Err(from_s3_error(maybe_error, parts));
	}
	Ok(())

.

I believe this comment is very good, but we could consider mentioning it in other places.

[Xuanwo]

Hello everyone, thank you all for joining the discussion. I have started a new thread with all the tenets we want and use cases we aim to support at #5301. Feel free to check it out there.