Tracking issues of OpenDAL wasm32-unknown-unknown target support #3803

Xuanwo · 2023-12-22T09:17:50Z

WebAssembly (WASM) is a modern binary instruction format for a stack-based virtual machine. It is designed as a portable compilation target for high-level languages like C/C++ and Rust, enabling deployment on the web for client and server applications. WASM aims to enable near-native performance for web applications, providing a more efficient and secure alternative to traditional JavaScript. Its key features include fast loading times, efficient execution, and the ability to run alongside JavaScript, allowing developers to build complex client-side web applications that perform better than those written solely in JavaScript.

By adding WASM support for opendal, we will allow users to use opendal in targets like wasm32-unknown-unknown. In a short word, users can use opendal in web browser!

We have addressed most blockers by #3796 and adding a basic test for s3 #3802. It's time for us to move forward.

This tracking issue is used to track our current progress of wasm support. Welcome to join in us!

Tasks

Add OPFS support

After opendal is ready for WASM, we can start to work for opfs now!

Add Origin Private File System support #2442

Add support for more services

Please run the following command to check if opendal can build under wasm32 arch:

cargo build --target wasm32-unknown-unknown --no-default-features --features=services-s3

Visit #3796 for possible fix or ask directly in this issue.

It's possible that some services can't support wasm at all, like fs. Please also provide your feedback here. We will collect them into a seperate doc.

Testing

Integrate behavior test for wasm target

Benchmark

Add benchmarks for opendal wasm

Existing issues

wasm-pack test --node doesn't work for s3_read_on_wasm
wasm-pack test --chrome --headless doesn't work for s3_read_on_wasm #3809

The text was updated successfully, but these errors were encountered:

Zheaoli · 2023-12-22T16:08:11Z

For now, I think the service based on the TCP socket is not working because the wasm32-unknown-unknown is not support TCP yet, the wasm32-unknown-emscripten can support TCP socket but need some hackc on Server

The TCP service is some service like following bellow:

MySQL
PostgreSQL
MongoDB
Redis

Zheaoli · 2023-12-22T17:03:16Z

Can we be used on wasmtime?

Can we be used on wasmedge?

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

The WASI runtime is not support for the wasm-bindgen which is dependent by reqwest now.

If we want to support the wasi runtime, we may need the other HTTP client which is compatible with the WASI spec

Xuanwo · 2023-12-23T06:00:00Z

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

Thanks, let's remove them. I just found that support wasm32-unknown-unkown is totally different from support wasm32-wasi. Lesson learnt.

Zheaoli · 2023-12-23T06:32:03Z

For now, the memory service can both on wasi(means can run in wasmtime and wasmedge, tested) and wasm32. The fs services need remove the tokio::fs

Zheaoli · 2023-12-23T07:00:51Z

For the record,

wasm-pack test --chrome --headless and wasm-pack test --chrome is both working.

But need some special S3 settings like

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "POST",
            "PUT",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

suyanhanx · 2023-12-23T07:58:28Z

incompatible crate found: [email protected]
When I tried to compile with services-gcs, error reported:
No available targets are compatible with triple "wasm32-unknown-unknown"

cargo tree -i ring
ring v0.17.5
├── jsonwebtoken v9.2.0
│   └── reqsign v0.14.6
│       └── opendal v0.43.0 
├── rustls v0.21.9
│   ├── hyper-rustls v0.24.2
│   │   └── reqwest v0.11.22
│   │       ├── opendal v0.43.0 
│   │       └── reqsign v0.14.6 (*)
│   ├── reqwest v0.11.22 (*)
│   └── tokio-rustls v0.24.1
│       ├── hyper-rustls v0.24.2 (*)
│       └── reqwest v0.11.22 (*)
├── rustls-webpki v0.101.7
│   └── rustls v0.21.9 (*)
└── sct v0.7.1
    └── rustls v0.21.9 (*)

Related issue: briansmith/ring#657
It looks like ring isn't going to support wasm for a while.

Xuanwo · 2023-12-23T10:56:47Z

When I tried to compile with services-gcs, error reported:

Let's mark gcs as unsupported since gcs will require jsonwebtoken (and ring) for JWT.

gabrielgrant · 2023-12-24T10:18:16Z

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: briansmith/ring#657 (comment)

the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?

Xuanwo · 2023-12-25T03:22:59Z

it seems surprising that ring would be failing in that way

Thanks for raising this up! I re-visit the gcs support on wasm, and confirmed that gcs is able to compile on wasm32-unknown-unknown. I will start a PR to address this.

suyanhanx · 2023-12-25T05:46:37Z

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: briansmith/ring#657 (comment)

the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?

Thank you for pointing that out!
I've tried to compile ring before but it kept failing. Finally, I realized that the llvm comes with macos doesn't have a backend of wasm. I reinstalled llvm and it compiled successfully. Sorry for the confusion!

fyears · 2023-12-31T18:17:05Z

Hi,

I am very interested in trying the wasm version of opendal for my project But I am not a professional rust developer though (I know something about javascript/typescript but none of rust), so I have some questions/ suggestions:

any doc to use the wasm version? by importing any library in js/typescript? any npm package?
would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .
would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.
in js world, there is a technique called “tree shaking” (by importing part of the package). For example developers only interested in s3 might only import opendal/s3 without importing others. It’s especially useful to support this because users have to download the assets in browser environment. Is it possible for opendal to support tree shaking?

Thank you so much and happy new year!

Young-Flash · 2024-01-01T02:48:06Z

It is still in a early stage and has not yet reached the stage where it can be released as a lib.

Maybe you could take a look at OpenDAL node.js binding.

Xuanwo · 2024-01-01T03:13:33Z

Hi @fyears, thank you for your interest! As @Young-Flash mentioned, our WebAssembly support is still in the early stages and not yet ready for practical use.

any doc to use the wasm version? by importing any library in js/typescript? any npm package?

Not yet. WASM support is still in active development, and we need to address existing bugs such as #3810 before proceeding.

We haven't released it yet. (However, you can create a wasm library by running cargo add opendal and building it.)

would you like to consider adding hooks for js fetch function? For example, I plan to use opendal in a special js environment and I have to use a special fetch function to get around the annoying cors limitatioms. Moreover, users might need to intercept/monitor/debug some network connections .

We are using web-sys now, and I'm not sure if web-sys support this use case.

would you like to consider adding hooks for js fs functiom? For example, in browser environment, file system might need some mocking by using indexeddb.

We will not support the fs service on wasm, which means there's no need to provide hooks either. We have a plan to support opfs: #2442. If users require IndexedDB, we can offer native support for it.

in js world, there is a technique called “tree shaking” (by importing part of the package). For example developers only interested in s3 might only import opendal/s3 without importing others. It’s especially useful to support this because users have to download the assets in browser environment. Is it possible for opendal to support tree shaking?

Nice question. We want this as well, but I'm not sure how to implement it. We can re-visit this after we have build a usable opendal wasm lib.

fyears · 2024-01-01T05:47:24Z

web-sys

After a quick search, it seems that web-sys has a clear mapping to js web api. However I think the most dirty work comes to the data crossing boundaries: wasm (prepare the net calls) -> js (fetch hook) -> wasm (reading the result)

Let's wait for the stable wasm releases first... LOL

Furthermore, my intention is to use opendal in my 320k downloaded project (remotely-save if you are interested). If you want to talk about the user case, you are welcome to DM me (username fyears) on discord .

Xuanwo · 2024-01-01T07:48:48Z

Furthermore, my intention is to use opendal in my 320k downloaded project (remotely-save if you are interested). If you want to talk about the user case, you are welcome to DM me (username fyears) on discord .

Interesting use case! I personally want OpenDAL to be usable in web browsers and Electron as well.

waynexia · 2024-03-28T12:52:58Z

I've played with the HTTP operator and it works well. Except for the CORS problem mentioned before. It needs the support from server to add CORS header (like the S3 operator)

fyears · 2024-03-28T14:00:19Z

may i ask is it usable now?

Xuanwo added good first issue Good for newcomers help wanted Extra attention is needed labels Dec 22, 2023

This comment was marked as resolved.

Sign in to view

Xuanwo mentioned this issue Dec 23, 2023

Unify the comment style. #3807

Closed

Young-Flash mentioned this issue Dec 23, 2023

feat: make services-gdrive compile for wasm target #3808

Merged

Xuanwo mentioned this issue Dec 23, 2023

wasm-pack test --chrome --headless doesn't work for s3_read_on_wasm #3809

Closed

Xuanwo mentioned this issue Dec 25, 2023

feat(core): Make gcs available on wasm32 arch #3816

Merged

Xuanwo mentioned this issue Jan 1, 2024

Can we make opendal-nodejs running on browser or electron? #3887

Open

Xuanwo changed the title ~~Tracking issues of OpenDAL WASM support~~ Tracking issues of OpenDAL wasm32-unknown-unknown target support Jan 11, 2024

Xuanwo mentioned this issue Jan 11, 2024

idea: Provide OpenDAL as host functions for wasm runtimes #3970

Open

This was referenced Apr 7, 2024

Can't fetch http query set waynexia/datafusion-wasm#1

Open

Can't fetch http query set datafusion-contrib/datafusion-wasm-bindings#1

Open

waynexia mentioned this issue Apr 9, 2024

feat: ?Send async trait for HttpBackend when the target is wasm32 #4444

Merged

KKould mentioned this issue Sep 2, 2024

Support WASM tonbo-io/tonbo#118

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking issues of OpenDAL wasm32-unknown-unknown target support #3803

Tracking issues of OpenDAL wasm32-unknown-unknown target support #3803

Xuanwo commented Dec 22, 2023 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

Zheaoli commented Dec 22, 2023

Zheaoli commented Dec 22, 2023

Xuanwo commented Dec 23, 2023 •

edited

Loading

Zheaoli commented Dec 23, 2023

Zheaoli commented Dec 23, 2023 •

edited

Loading

suyanhanx commented Dec 23, 2023 •

edited

Loading

Xuanwo commented Dec 23, 2023

gabrielgrant commented Dec 24, 2023

Xuanwo commented Dec 25, 2023

suyanhanx commented Dec 25, 2023

fyears commented Dec 31, 2023 •

edited

Loading

Young-Flash commented Jan 1, 2024

Xuanwo commented Jan 1, 2024

fyears commented Jan 1, 2024 •

edited

Loading

Xuanwo commented Jan 1, 2024

waynexia commented Mar 28, 2024

fyears commented Mar 28, 2024

Tracking issues of OpenDAL wasm32-unknown-unknown target support #3803

Tracking issues of OpenDAL wasm32-unknown-unknown target support #3803

Comments

Xuanwo commented Dec 22, 2023 • edited Loading

Tasks

Add OPFS support

Add support for more services

Testing

Benchmark

Existing issues

This comment was marked as resolved.

This comment was marked as resolved.

Zheaoli commented Dec 22, 2023

Zheaoli commented Dec 22, 2023

Xuanwo commented Dec 23, 2023 • edited Loading

Zheaoli commented Dec 23, 2023

Zheaoli commented Dec 23, 2023 • edited Loading

suyanhanx commented Dec 23, 2023 • edited Loading

Xuanwo commented Dec 23, 2023

gabrielgrant commented Dec 24, 2023

Xuanwo commented Dec 25, 2023

suyanhanx commented Dec 25, 2023

fyears commented Dec 31, 2023 • edited Loading

Young-Flash commented Jan 1, 2024

Xuanwo commented Jan 1, 2024

fyears commented Jan 1, 2024 • edited Loading

Xuanwo commented Jan 1, 2024

waynexia commented Mar 28, 2024

fyears commented Mar 28, 2024

Xuanwo commented Dec 22, 2023 •

edited

Loading

Xuanwo commented Dec 23, 2023 •

edited

Loading

Zheaoli commented Dec 23, 2023 •

edited

Loading

suyanhanx commented Dec 23, 2023 •

edited

Loading

fyears commented Dec 31, 2023 •

edited

Loading

fyears commented Jan 1, 2024 •

edited

Loading