Add `wit_import` attribute macro to import functions from Wasm modules #944

jvff · 2023-08-09T19:19:36Z

Motivation

The new Witty crate should allow calling functions exported from Wasm guest modules.

Proposal

Use a wit_import attribute macro that parses a trait definition and generates the code to call functions from a guest Wasm instance defined by the trait.

In order to make the behavior independent from the host runtime, some traits were defined to abstract away the exported function's signature and how parameters and results are converted to the necessary WIT primitives to call the function.

Test Plan

Some simple tests were written to ensure that the wit_import macro generates code that tries to load functions from a mock Wasm instance and allows calling them.

The test_case crate together with the TestInstanceFactory trait were used in order to make it simple for future runtime implementations to reuse the same tests.

Links

Part of #906

Release Plan

All good!
Need to bump the major/minor version number in the next release of the crates.
Need to update the developer manual.
This PR is adding or removing Cargo features.
Release is blocked and/or tracked by other issues (see links above)

Reviewer Checklist

The title and the summary of the PR are short and descriptive.
The proposed solution achieves the goals stated in the PR.
The test plan is reproducible and provides sufficient coverage.
The release plan is adequate.
The commits correspond to distinct logical changes.
The code follows the coding guidelines.
The proposed changes look correct.
The CI is passing.
All of the above!

Missing from some of the unit tests.

Don't require a name and type pair, allowing macros to be applied on lists of either just names or just types as well.

Make it easier to convert a host parameters type into the flat type parameters for an imported guest function.

Allow converting from the results received from an imported guest Wasm function into the host results type.

ma2bd

LGTM!

linera-witty/src/imported_function_interface/mod.rs

ma2bd · 2023-08-09T21:42:43Z

linera-witty/src/imported_function_interface/mod.rs

+        <Instance::Runtime as Runtime>::Memory: RuntimeMemory<Instance>;
+}
+
+impl<Parameters, Results> ImportedFunctionInterface for (Parameters, Results)


Is this the only implementation of ImportedFunctionInterface? Did you consider a helper struct struct ImportedFunctionInterface<P, R> { parameters: P, results: R }, then defining a block impl<P, R> ImportedFunctionInterface<P, R> where .. { .. } ?

I guess it depends if you need the output associated types Guest* as a short alias.

Yes, unfortunately associated types in types isn't stable yet :/

linera-witty/tests/common/test_instance.rs

Use the `FlatHostParameters` and `FlatHostResults` traits in a single trait that can be used to determine the interface to an imported function from a guest based on the host inteface types.

Prepare to add support for a new `wit_import` attribute procedural macro. The attribute requires receiving an extra parameter to determine the package of the imported functions, and the namespace can also be specified as a parameter. Implement extraction of the namespace by parsing the attribute parameters. It will expect the package to be specified, and will use the namespace if provided as a parameter or fallback to the trait name if it's not.

Parse the functions from a trait and extract the information to later be used to generate code to import functions.

A type to allow `TokenStream`s to be used as an element in a `HashSet`.

Given a trait interface, generate the code for a type with equivalent methods that load a function from the guest and calls it.

Allow generated code to use it.

Generate a trait alias for the instance constraints for the implementation of the imported functions. Use this trait alias in the generated code as well instead of using the constraints directly.

And `FakeRuntime` to `MockRuntime`. Prepare to allow mocking exported functions in the mocked instance.

Keep track of the mock exported functions and allow handlers to be called when the exported functions are called.

Prepare to keep track of which handler should be called based on the exported function name.

Prepare to use the same mechanism for the canonical ABI memory functions.

Implement `InstanceWithFunction` for all valid parameters and results types.

Improve the ergonomics of mocking an exported function and checking that it's called correctly.

Check that a function without parameters or return types can be imported from a mock instance.

Check that functions that have return types can be imported from a mock instance.

Check that functions that have single parameters can be imported from a mock instance.

Check that functions that have multiple parameters and return values can be imported from a mock instance.

jvff added 4 commits August 9, 2023 18:45

Add missing lift and lowering round-trip tests

ab790ee

Missing from some of the unit tests.

Refactor repeat_macro! to support lone elements

cc105c7

Don't require a name and type pair, allowing macros to be applied on lists of either just names or just types as well.

Create a FlatHostParameters helper trait

766014c

Make it easier to convert a host parameters type into the flat type parameters for an imported guest function.

Create a FlatHostResults helper trait

390dd63

Allow converting from the results received from an imported guest Wasm function into the host results type.

jvff added the enhancement New feature or request label Aug 9, 2023

jvff added this to the Devnet milestone Aug 9, 2023

jvff requested a review from ma2bd August 9, 2023 19:19

jvff self-assigned this Aug 9, 2023

jvff mentioned this pull request Aug 9, 2023

Add a new crate to generate host WIT code #906

Closed

14 tasks

jvff force-pushed the wit-import branch from 55dfd78 to 851b778 Compare August 9, 2023 19:54

ma2bd approved these changes Aug 9, 2023

View reviewed changes

jvff added 18 commits August 9, 2023 23:56

Create an ImportedFunctionInterface helper trait

8bc1217

Use the `FlatHostParameters` and `FlatHostResults` traits in a single trait that can be used to determine the interface to an imported function from a guest based on the host inteface types.

Collect information for functions to import

cfde95c

Parse the functions from a trait and extract the information to later be used to generate code to import functions.

Create a TokensSetItem helper type

c8798ec

A type to allow `TokenStream`s to be used as an element in a `HashSet`.

Generate code to import functions from guests

0cfa881

Given a trait interface, generate the code for a type with equivalent methods that load a function from the guest and calls it.

Export InstanceWithFunction trait

7afecc5

Allow generated code to use it.

Export Instance trait

99d0c83

Allow generated code to use it.

Refactor to use a trait alias for instances

b48cb96

Generate a trait alias for the instance constraints for the implementation of the imported functions. Use this trait alias in the generated code as well instead of using the constraints directly.

Rename FakeInstance to MockInstance

9f4b5a3

And `FakeRuntime` to `MockRuntime`. Prepare to allow mocking exported functions in the mocked instance.

Allow adding mocked exported functions to instance

74f0458

Keep track of the mock exported functions and allow handlers to be called when the exported functions are called.

Use String as the export type

8405d3e

Prepare to keep track of which handler should be called based on the exported function name.

Register canonical ABI functions an mocks

23a047a

Prepare to use the same mechanism for the canonical ABI memory functions.

Allow mock exported functions to be called

1a4d980

Implement `InstanceWithFunction` for all valid parameters and results types.

Create a MockExportedFunction helper type

4077038

Improve the ergonomics of mocking an exported function and checking that it's called correctly.

Test importing a simple function

8af3b5c

Check that a function without parameters or return types can be imported from a mock instance.

Test importing functions that return values

521a670

Check that functions that have return types can be imported from a mock instance.

Test importing functions with single parameters

80820a7

Check that functions that have single parameters can be imported from a mock instance.

Test importing input/output functions

253cce5

Check that functions that have multiple parameters and return values can be imported from a mock instance.

jvff force-pushed the wit-import branch from 851b778 to 253cce5 Compare August 9, 2023 23:56

jvff merged commit 15bddb1 into linera-io:main Aug 10, 2023
3 checks passed

jvff deleted the wit-import branch August 10, 2023 00:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `wit_import` attribute macro to import functions from Wasm modules #944

Add `wit_import` attribute macro to import functions from Wasm modules #944

jvff commented Aug 9, 2023 •

edited by ma2bd

Loading

ma2bd left a comment

ma2bd Aug 9, 2023

ma2bd Aug 9, 2023

jvff Aug 9, 2023

Add wit_import attribute macro to import functions from Wasm modules #944

Add wit_import attribute macro to import functions from Wasm modules #944

Conversation

jvff commented Aug 9, 2023 • edited by ma2bd Loading

Motivation

Proposal

Test Plan

Links

Release Plan

Reviewer Checklist

ma2bd left a comment

Choose a reason for hiding this comment

ma2bd Aug 9, 2023

Choose a reason for hiding this comment

ma2bd Aug 9, 2023

Choose a reason for hiding this comment

jvff Aug 9, 2023

Choose a reason for hiding this comment

Add `wit_import` attribute macro to import functions from Wasm modules #944

Add `wit_import` attribute macro to import functions from Wasm modules #944

jvff commented Aug 9, 2023 •

edited by ma2bd

Loading