Looking for GSoC 2024 Project Ideas

[Xuanwo]

Hello, OpenDAL Community!

The Google Summer of Code is about to begin, and the Apache Software Foundation (ASF) has registered as a mentoring organization. The ASF is now inviting us to submit our project ideas for consideration in their mentoring program.

This year is the first year for OpenDAL to take part in GSoC. I believe it's a great opportunity to attract more contributors

Do you have ideas to submit? And do you want to be a mentor for it? Welcome to leave your comments here.

More details could be found at:

• Guide to being a mentor
• Apache GSoC Page

---

DEADLINE: Feb 03

[pacoxu]

Some colleagues have discussed on integrating OpenDAL with clusterperdia(written in golang). Is there something like rust binding from zig to go.

[Xuanwo]

We offer a Golang binding built on top of our C bindings, although it is still in the early stages of development.

[AbhishekCS3459]

@Xuanwo I want to be a part of GSOC 2024 as a contributor. I have some ideas too, how to proceed with them?

[Xuanwo]

👋, welcome! You can present idea here directly so we can find mentors who interested in them.

[oowl]

@AbhishekCS3459 We can discuss it in the issue, draft an RFC doc to propose, and then propose it.

[Xuanwo]

@AbhishekCS3459 We can discuss it in the issue, draft an RFC doc to propose, and then propose it.

We just need ideas here. There ideas will be reviewed by both ASF and GSoC. We can fill the RFC after our ideas been accepted.

[chamber909]

I have some ideas but before I go, I have a question regarding this , what kind of project like some new feature, implementation in openDAL or building projects that will use OpenDAL API?

[Xuanwo]

Hi, thank you for your interest. However, I am unable to understand your comment. Could you please provide further clarification?

[chamber909]

So can I give some ideas (as you mentioned in the discussion) about new feature in openDAL or something like an application (project ideas) which uses this openDAL or anything. Actually I am beginner in this field so it's confusing for me to figure out.

[Xuanwo]

Both of them are ok. We can discuss about them case by case.

[mav3ri3k]

Hi, I recently found the project. What instantly striked me was the scale of project and the number of bindings it aims to provide which is great!
I think another approach for the same might be to implement them once in web assembly. The new wasm component model would be great fit for this. These can then be consumed by other programming languages with minimal performance penalty. This will also greatly reduce the amount of bindings and code that needs to be maintained to one. This project would also be a good candidate for the same since web assembly has first class support for rust.

[Xuanwo]

It indeed a goal for opendal to be able to run under wasm32-unknown-unknown target, there are some progress at #3803.

Your idea is to create something similar to opendal-wasm and enable users to import it in wasm?

[mav3ri3k]

I want to utilize wasi which is a system interface for webassembly. From #3803, I believe that you are targeting wasm32-unknown-unknown target. It is my understanding that wasm32-wasi is better positioned for syscalls. From the documentation:

It's an API designed by the Wasmtime project that provides access to several operating-system-like features, including files and filesystems, Berkeley sockets, clocks, and random numbers, that we'll be proposing for standardization.

This might help work around some issues like

• this using wasi-http
• Add Origin Private File System support #2442 using wasi-filesystem

However, I must add that I still do not fully understand what the requirements or vision is for wasm support for this project which might decide whether using wasm32-unknown-unknown or wasm32-wasi is a better option. I found this to be a good comparison.

My original suggestion

Currently bindings for each language need to be maintained separately. My suggestion was to try such that the rust crate for opendal is build into a wasm component, example rust function to wasm component. Now these components could be used in any language for which runtime like wasitime supports the new component model. The list of supported languages is small but that is especially because how new the specification is and I expect it to grow rapidly.

[simonsan]

Currently bindings for each language need to be maintained separately. My suggestion was to try such that the rust crate for opendal is build into a wasm component, example rust function to wasm component. Now these components could be used in any language for which runtime like wasitime supports the new component model. The list of supported languages is small but that is especially because how new the specification is and I expect it to grow rapidly.

Keep in mind that pulling in a runtime is not always possible for some projects and not useful (performancewise). So having bindings in the language is probably better, I would assume. So I personally would see WASM/WASI not as a replacement, but an addition to the ecosystem.

[mav3ri3k]

Yes, any change should be seen as addition first, for it is all talk now.
However I must add that wasm runtimes like wasmtime have or atleast claims to have low-overhead calls between the embedder and wasm. The performace penalty is largely dependent on type of language. Components in low-level languages are expected to have close to native code performance.

[Xuanwo]

However, I must add that I still do not fully understand what the requirements or vision is for wasm support for this project which might decide whether using wasm32-unknown-unknown or wasm32-wasi is a better option. I found this to be a good comparison.

They are not conflict goals. OpenDAL's VISION is *access data freely: We want opendal to run on both wasm32-unknown-unknown and wasm32-wasi.

Supporting wasm32-wasi has a more detail discussion here: #3970

[chamber909]

hey, can we make the CLI tool more automated towards storing data?

[Xuanwo]

How about presenting your ideas more clearly? What improvements do you want to make?

[chamber909]

Ok, What I mean is that I want to provide built-in functionality for dynamic data storage allocation based on database load or capacity.
So the CLI tool will store data on its own based on where storage is high or low.

The improvements I want to make :
1)Implement a monitoring system that continuously tracks the load and capacity of each database (PostgreSQL, MySQL, Google Cloud).

2)Develop logic that analyzes the monitored data to determine which database has the lowest load or the most available capacity for storing new data.

3)Use the OpenDAL CLI or its programming interfaces to interact with the databases based on the decisions made by users logic.

4)When a user initiates data storage, your system automatically selects the appropriate database (PostgreSQL, MySQL, or Google Cloud) based on the analysis of load and capacity.

5. Implementing load balancing mechanisms to distribute data storage evenly across databases. Additionally, include failover mechanisms to handle cases where a database becomes unavailable or overloaded.

6)Continuously monitor the performance of users data storage system and adjust the allocation logic as needed based on real-world usage patterns.

[Xuanwo]

Thank you for proposing the idea. However, I believe it is complex enough to warrant being a separate project based on OpenDAL instead part of OpenDAL or oli.

[chamber909]

so can we do only one thing in those points which is :
Develop logic or functionality in this framework that analyzes the monitored data to determine which database has the lowest load or the most available capacity for storing new data.

[Xuanwo]

I'm sorry, but I believe it falls outside the scope of opendal.

[oowl]

Later proposal, I want to propose a kio binding for OpenDAL which was mentioned before in our issues #3042, it is related to our cpp binding.
we can base on https://github.com/KDE/kio-s3 project to implement our opendal-kio submodule.

[Xuanwo]

Hi, as a PMC memebr of OpenDAL, You could do anything that I can do!

Would you like to try submit an issue like #4130 and create a ticket like https://issues.apache.org/jira/browse/GSOC-255? I'm willing to help you to address any possible problems.