feat: Implement a basic Extism Go SDK #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I added support for loading a plugin from a manifest, but not sure if the approach is correct or if it's the best way to do it, please let me know if you have any feedback. |
|
Hey @zshipko, I have been trying to implement Namely:
|
There is already
I will have to take a closer look at the code to be sure, but it might be possible to use context.WithValue for this. You could set the "CurrentPlugin" key on the context to a |
zshipko
reviewed
Jul 18, 2023
… for WasmUrl and WasmFile
Ah, I don't know how I couldn't see it. Thanks!
Yup, that works |
|
@zshipko I have faced a new roadblock: wazero can't initialize two modules with the same name. And we are implmenting some PDK functions in the kernel module and some of them in a hosted module (for example: I have tried these work-arounds:
env, err := r.NewHostModuleBuilder("env").
NewFunctionBuilder().
WithFunc(func(v uint32) {
fmt.Println("log_i32 >>", v)
}).
Export("log_i32").
NewFunctionBuilder().
WithFunc(func() uint32 {
if envYear, err := strconv.ParseUint(os.Getenv("CURRENT_YEAR"), 10, 64); err == nil {
return uint32(envYear) // Allow env-override to prevent annual test maintenance!
}
return uint32(time.Now().Year())
}).
Export("current_year").
Instantiate(ctx)
if err != nil {
log.Panicln(err)
}
// try to call `current_year`
rr, err := env.ExportedFunction("current_year").Call(ctx)
if err != nil {
log.Panicln(err)
}output:
Should we file an issue on wazero for that? One other way would be to change the namespace of the non-kernel functions. But this would be a breaking change for all of the PDKs |
|
Yeah the lack of a linker similar to wasmtime makes this confusing. I think makes the most sense in terms of how to proceed. I will experiment with that a little today and make a PR against this branch if I come up with something. If being able to add imports to a module is just a matter of adding a new function to wazero that could be a viable option as well. |
That'd be great if possible |
feat: Allow `env` module to be extended by the extism runtime
fix: add glob package to go.mod file
mhmd-azeez
commented
Jul 30, 2023
Comment on lines
+9
to
+15
| //export __wasm_call_ctors | ||
| func __wasm_call_ctors() | ||
|
|
||
| //export _initialize | ||
| func _initialize() { | ||
| __wasm_call_ctors() | ||
| } |
There was a problem hiding this comment.
@zshipko we can export our own _initialize in TinyGo
Closed
|
There are still some things to do, but I am marking the PR ready for review |
…runtime is initialized and cleaned up
|
I have changed the host function callback signature to include userData and use the same stack used by wazero. But tbh, I don't think we don't need userData anymore here as there are more idiomatic ways to pass in some state |
|
For the CI, do we need to test on multiple OSes? since the whole thing is written in pure Go |
@mhmd-azeez - is this statement true for all SDK languages, or just Go? We just need to be mindful of the compatibility across hosts/guests -- I'm certain you are already thinking of this :) just clarifying! |
I think starting with |
|
To test this SDK, I think it would be nice to update this repo to use it: https://github.com/extism/extism-fsnotify |
If you mean compatibility between Host APIs, I was under the impression that the host SDK APIs didn't have to match each other, as long as they're all capable of doing the same set of operations. So this statement would be true for any language that supports closures. And if you mean compatibility between the GO SDK and the guest PDKs, the But in the end, I don't have a strong opinion about |
|
I think the userData argument is mostly because of the C API, it doesn't make as much sense in this case. |
|
Also, this PR is looking great! Unless @nilslice disagrees or there are some major API design changes you're working on I think you're good to merge this when you feel ready. Let me know if there's anything you need specific feedback on before merging! |
|
I'm on the same page as @zshipko - it's got enough eyes on it to merge and we can follow up with some more narrowly scoped PRs Great work!! |
|
Agreed, this is a great base and we can follow up with more focused PRs. Nice work!
…On Aug 2, 2023 at 7:24 PM -0500, Steve Manuel ***@***.***>, wrote:
I'm on the same page as @zshipko - it's got enough eyes on it to merge and we can follow up with some more narrowly scoped PRs
Great work!!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
nilslice
approved these changes
Aug 3, 2023
|
Great, in this case I will merge this PR and continue working on follow up PRs |
mhmd-azeez
added a commit
to extism/extism
that referenced
this pull request
Jul 25, 2024
This is a rough POC for allowing people to whitelist a dir as readonly. When the source path is prefixed with `ro:`, the dir is considered as readonly. This preserved backward compatibility. This suggestion came up in extism/go-sdk#1 (comment) Readonly: ```rs let manifest = Manifest::new([url]) .with_allowed_path("ro:D:/x/rust/fs/data".to_string(), "/data") .with_config_key("path", "/data/data.txt"); ``` ``` trying to read file: "Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\n" ----------------------------------------------------- trying to write file: thread '<unnamed>' panicked at src\lib.rs:24:34: called `Result::unwrap()` on an `Err` value: Os { code: 58, kind: Unsupported, message: "Not supported" } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'main' panicked at runtime\examples\fs.rs:27:6: called `Result::unwrap()` on an `Err` value: error while executing at wasm backtrace: 0: 0x234d2 - fs.wasm!__rust_start_panic 1: 0x232a1 - fs.wasm!rust_panic 2: 0x231da - fs.wasm!std::panicking::rust_panic_with_hook::hd3fb69bc0aea298a 3: 0x22467 - fs.wasm!std::panicking::begin_panic_handler::{{closure}}::h4d99b90b43f79472 4: 0x223ca - fs.wasm!std::sys_common::backtrace::__rust_end_short_backtrace::h5691573a73161cb1 5: 0x22bca - fs.wasm!rust_begin_unwind 6: 0x303e9 - fs.wasm!core::panicking::panic_fmt::hdb62f5cdb45533e4 7: 0x3234d - fs.wasm!core::result::unwrap_failed::h30d23efcc9e41efc 8: 0x36c2 - fs.wasm!fs::try_write::inner::h0b3b0df8e129f5cc 9: 0x29cd - fs.wasm!try_write 10: 0x35e4a - fs.wasm!try_write.command_export note: using the `WASMTIME_BACKTRACE_DETAILS=1` environment variable may show more debugging information Caused by: wasm trap: wasm `unreachable` instruction executed Stack backtrace: 0: std::backtrace_rs::backtrace::dbghelp64::trace at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\..\..\backtrace\src\backtrace\dbghelp64.rs:99 1: std::backtrace_rs::backtrace::trace_unsynchronized at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\..\..\backtrace\src\backtrace\mod.rs:66 2: std::backtrace::Backtrace::create at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\backtrace.rs:331 3: std::backtrace::Backtrace::capture at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\backtrace.rs:296 4: anyhow::error::impl$1::from<wasmtime_environ::trap_encoding::Trap> at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\anyhow-1.0.86\src\error.rs:565 5: core::convert::impl$3::into<wasmtime_environ::trap_encoding::Trap,anyhow::Error> at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\convert\mod.rs:759 6: wasmtime_environ::impl$1::into_anyhow<wasmtime_environ::trap_encoding::Trap> at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-environ-22.0.0\src\lib.rs:90 7: wasmtime::runtime::trap::from_runtime_box at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\trap.rs:118 8: wasmtime::runtime::func::invoke_wasm_and_catch_traps::closure$0<extism::current_plugin::CurrentPlugin,wasmtime::runtime::func::impl$1::call_unchecked_raw::closure_env$0<extism::current_plugin::CurrentPlugin> > at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1597 9: enum2$<core::result::Result<tuple$<>,alloc::boxed::Box<wasmtime::runtime::vm::traphandlers::Trap,alloc::alloc::Global> > >::map_err<tuple$<>,alloc::boxed::Box<wasmtime::runtime::vm::traphandlers::Trap,alloc::alloc::Global>,anyhow::Error,wasmtime::runtime: at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\result.rs:829 10: wasmtime::runtime::func::invoke_wasm_and_catch_traps<extism::current_plugin::CurrentPlugin,wasmtime::runtime::func::impl$1::call_unchecked_raw::closure_env$0<extism::current_plugin::CurrentPlugin> > at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1597 11: wasmtime::runtime::func::Func::call_unchecked_raw<extism::current_plugin::CurrentPlugin> at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1063 12: wasmtime::runtime::func::Func::call_unchecked<ref_mut$<wasmtime::runtime::store::context::StoreContextMut<extism::current_plugin::CurrentPlugin> > > at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1049 13: wasmtime::runtime::func::Func::call_impl_do_call<extism::current_plugin::CurrentPlugin> at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1243 14: wasmtime::runtime::func::Func::call<ref_mut$<wasmtime::runtime::store::Store<extism::current_plugin::CurrentPlugin> > > at C:\Users\muham\.cargo\registry\src\index.crates.io-6f17d22bba15001f\wasmtime-22.0.0\src\runtime\func.rs:1002 15: extism::plugin::Plugin::raw_call<ref$<str$>,ref$<str$> > at .\src\plugin.rs:753 16: extism::plugin::Plugin::call<ref$<str$>,ref$<str$>,ref$<str$> > at .\src\plugin.rs:900 17: fs::main at .\examples\fs.rs:25 18: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> > at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250 19: core::hint::black_box at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\hint.rs:337 20: std::sys_common::backtrace::__rust_begin_short_backtrace<void (*)(),tuple$<> > at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\sys_common\backtrace.rs:155 21: std::rt::lang_start::closure$0<tuple$<> > at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\rt.rs:166 22: std::rt::lang_start_internal at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\rt.rs:148 23: std::rt::lang_start<tuple$<> > at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\std\src\rt.rs:165 24: main 25: invoke_main at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78 26: __scrt_common_main_seh at D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288 27: BaseThreadInitThunk 28: RtlUserThreadStart stack backtrace: 0: std::panicking::begin_panic_handler at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\std\src\panicking.rs:645 1: core::panicking::panic_fmt at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\panicking.rs:72 2: core::result::unwrap_failed at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library\core\src\result.rs:1654 3: enum2$<core::result::Result<ref$<str$>,anyhow::Error> >::unwrap at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\result.rs:1077 4: fs::main at .\examples\fs.rs:25 5: core::ops::function::FnOnce::call_once<void (*)(),tuple$<> > at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\ops\function.rs:250 6: core::hint::black_box at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6\library\core\src\hint.rs:337 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. error: process didn't exit successfully: `D:\dylibso\extism\target\debug\examples\fs.exe` (exit code: 101) ``` Writable: ```rs let manifest = Manifest::new([url]) .with_allowed_path("D:/x/rust/fs/data".to_string(), "/data") .with_config_key("path", "/data/data.txt"); ``` ``` trying to read file: "Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\n" ----------------------------------------------------- trying to write file: "Hello World at 1719851282.5109031sHello World at 1719851299.0819795sHello World at 1719851317.8934608s\nHello World at 1719851500.7803263s\n" done! ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements https://github.com/dylibso/JIRA/issues/35
extism/go-sdk), implement the full Go SDK API (where it makes sense, @zshipko can clarify here a bit) & make improvements to the APIs as you go.Cache http wasm pluginsAs far as I can see, this is not being used in the Rust runtime either.extism_http_request), configured to disallow every request by default, and allow hostnames as defined in the Manifestextism_http_requestextism_http_status_codecontextpackage should make this more straightforward)envmoduleThink about host functions in modules we don't own (from wasm binaries). Maybe we can wrap the guest module in a host module just like the env module. I think we should not support this until a user explicitly asks for it, so that we understand the use case better_startfunction)extism-runtime.wasmQuestions
wazero.Runtimelightweight? Yes. See feat: Implement a basic Extism Go SDK #1 (comment)