python ffi creation #59

jasoncolburne · 2023-02-03T23:36:51Z

Rationale

Wanted to set us up for success. I created the beginnings of the Rust API layer and the Python FFI. Comments welcome.

Also, this unlocks parallel streams of work and tasks with a lower bar to entry for the project.

Changes

Exposes Rust API and maintains standard crate configuration.
Creates Python API with a simple Matter class exposed.
Adds a Makefile and some targets for easier scripting (abusing make a bit but it's available everywhere)
Updates README.md with instructions for new developers.

I am pretty happy with how I cleaned up the directory structure after a few iterations and separated the python code from the main library. The only changes to the core library are to support a simple #[pyclass] above the definition of the struct.

Testing

You are encouraged to check out the branch and try these commands.

> make clean libs preflight python-shell
Python 3.10.6 (main, Aug 11 2022, 13:49:25) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from cesride import Matter
>>> m = Matter(qb64="BGlOiUdp5sMmfotHfCWQKEzWR91C72AH0lT84c0um-Qj")
>>> qb2 = m.qb2()
>>> print(qb2)
[4, 105, 78, 137, 71, 105, 230, 195, 38, 126, 139, 71, 124, 37, 144, 40, 76, 214, 71, 221, 66, 239, 96, 7, 210, 84, 252, 225, 205, 46, 155, 228, 35]
>>> m2 = Matter(qb2=bytes(qb2))
>>> m2.qb64()
'BGlOiUdp5sMmfotHfCWQKEzWR91C72AH0lT84c0um-Qj'
>>> m3 = Matter(code=m.code(), raw=bytes(m.raw()), raw_size=len(m.raw()))
>>> m3.qb64()
'BGlOiUdp5sMmfotHfCWQKEzWR91C72AH0lT84c0um-Qj'
>>> Matter(qb64="_" * 44)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: unexpected code error: op code start
>>> Matter(qb64="-" * 44)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Exception: unexpected code error: count code start

Screenshots (updated README.md)

Here is a live link to a preview of this file.

codecov · 2023-02-03T23:40:33Z

Codecov Report

Merging #59 (7b93440) into main (a3beb9a) will increase coverage by 0.10%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #59      +/-   ##
==========================================
+ Coverage   97.72%   97.83%   +0.10%     
==========================================
  Files           9        9              
  Lines         969      968       -1     
==========================================
  Hits          947      947              
+ Misses         22       21       -1

Impacted Files	Coverage Δ
src/core/counter/mod.rs	`95.55% <ø> (ø)`
src/core/diger.rs	`100.00% <ø> (ø)`
src/core/matter/mod.rs	`95.35% <ø> (+0.40%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

jasoncolburne · 2023-02-04T10:18:06Z

Makefile

+
+python:
+	@head -n $(LINE) Cargo.toml > cargo/$@/Cargo.toml
+	@echo "pyo3 = { version = \"0.18.0\", features = [\"abi3\", \"extension-module\"] }" >> cargo/$@/Cargo.toml


This was pretty gross but I was in a rush.

Would be nice to use a toml parser or something to edit the section and re-export.

Or, template everything.

yeah add an issue so we track it I think it's fine for now

jasoncolburne · 2023-02-04T10:21:09Z

src/core/matter/mod.rs

+        let result = if let Some(code) = code {
+            let (raw, raw_size) = if raw.is_none() || raw_size.is_none() {
+                return Err(PyValueError::new_err("code present, raw and raw_size missing"));
+            } else {
+                (raw.unwrap(), raw_size.unwrap())
+            };
+
+            Self::new_with_code_and_raw(code, raw, raw_size)
+        } else if let Some(qb64) = qb64 {
+            Self::new_with_qb64(qb64)
+        } else if let Some(qb64b) = qb64b {
+            Self::new_with_qb64b(qb64b)
+        } else if let Some(qb2) = qb2 {
+            Self::new_with_qb2(qb2)
+        } else {


Someone should add validation if this gets merged, that multiple things aren't set. Example: qb2 and qb64 both specified = bad.

This comment is only outdated because code moved, we should still do this.

src/lib_python.rs

jasoncolburne · 2023-02-04T15:17:27Z

src/core/matter/mod.rs


 use crate::core::util;
 use crate::error::{err, Error, Result};

+pub mod tables;
+
+#[cfg_attr(feature = "python", pyclass)]


Just this line in the core code, and the required gated-use.

jasoncolburne · 2023-02-04T17:34:41Z

Side note: the tarpaulin action occasionally fails, we should look at using a make or shell command rather than the action.

m00sey

🚀

m00sey · 2023-02-04T19:00:14Z

Makefile

+	cargo fix
+	cargo fmt
+
+preflight:


I've been wanting to do this !!

m00sey · 2023-02-04T19:29:28Z

src/core/mod.rs

+pub mod cigar;
+pub mod counter;
+pub mod diger;
+pub mod matter;


Does Matter want to be public? I wouldn't expect people to be initializing it as it's a base for the others?

Possibly not, but in the Rust syntax we'd probably need a type because using a trait you initialize like this:

let verfer = <Matter as Verfer>::new_with_code_and_raw(/* ... */);

.. we may be able to alias all these? I can look into this in another issue.

m00sey · 2023-02-04T19:33:16Z

Makefile

+
+python:
+	@head -n $(LINE) Cargo.toml > cargo/$@/Cargo.toml
+	@echo "pyo3 = { version = \"0.18.0\", features = [\"abi3\", \"extension-module\"] }" >> cargo/$@/Cargo.toml


yeah add an issue so we track it I think it's fine for now

Artemkaaas · 2023-02-06T06:26:03Z

@jasoncolburne I would consider usage of uniffi Rust crate for FFI bindings.
It can generate bindings for multiple languages at once (Kotlin, Swift, Python, Ruby).
You may have the following project structure:

cesride-rs rust package published to crates.io
cesride crate depending on cesride-rs which provides system library and FFI bindings with using uniffi.

jasoncolburne · 2023-02-06T07:16:58Z

@Artemkaaas Thanks for this, it sounds less hacky than what I threw together! Someone mentioned it recently but it went past me and I didn't really know what it offered. I created a ticket for investigation: #65

jasoncolburne force-pushed the ffideas branch from e967e1c to a90002a Compare February 4, 2023 09:58

jasoncolburne commented Feb 4, 2023

View reviewed changes

src/lib_python.rs Outdated Show resolved Hide resolved

jasoncolburne force-pushed the ffideas branch from a90002a to fe530c2 Compare February 4, 2023 13:50

jasoncolburne changed the title ~~ffi spike~~ python ffi creation Feb 4, 2023

jasoncolburne force-pushed the ffideas branch 3 times, most recently from 578062b to eb6efd2 Compare February 4, 2023 15:14

jasoncolburne commented Feb 4, 2023

View reviewed changes

jasoncolburne force-pushed the ffideas branch 2 times, most recently from 132f2b4 to c8496d6 Compare February 4, 2023 15:54

jasoncolburne marked this pull request as ready for review February 4, 2023 15:59

jasoncolburne requested a review from m00sey as a code owner February 4, 2023 15:59

jasoncolburne force-pushed the ffideas branch 3 times, most recently from 221c1f7 to 321caad Compare February 4, 2023 17:16

python ffi

7b93440

jasoncolburne force-pushed the ffideas branch from 321caad to 7b93440 Compare February 4, 2023 17:24

m00sey approved these changes Feb 4, 2023

View reviewed changes

jasoncolburne merged commit 817fda3 into WebOfTrust:main Feb 4, 2023

jasoncolburne deleted the ffideas branch February 4, 2023 20:16

jasoncolburne mentioned this pull request Feb 6, 2023

Explore using uniffi #65

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python ffi creation #59

python ffi creation #59

jasoncolburne commented Feb 3, 2023 •

edited

Loading

codecov bot commented Feb 3, 2023 •

edited

Loading

jasoncolburne Feb 4, 2023

jasoncolburne Feb 4, 2023

jasoncolburne Feb 4, 2023

m00sey Feb 4, 2023

jasoncolburne Feb 4, 2023

jasoncolburne Feb 4, 2023 •

edited

Loading

jasoncolburne Feb 4, 2023

jasoncolburne commented Feb 4, 2023

m00sey left a comment

m00sey Feb 4, 2023

m00sey Feb 4, 2023

jasoncolburne Feb 4, 2023

m00sey Feb 4, 2023

Artemkaaas commented Feb 6, 2023

jasoncolburne commented Feb 6, 2023

python ffi creation #59

python ffi creation #59

Conversation

jasoncolburne commented Feb 3, 2023 • edited Loading

Rationale

Changes

Testing

Screenshots (updated README.md)

codecov bot commented Feb 3, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasoncolburne Feb 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasoncolburne commented Feb 4, 2023

m00sey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Artemkaaas commented Feb 6, 2023

jasoncolburne commented Feb 6, 2023

jasoncolburne commented Feb 3, 2023 •

edited

Loading

codecov bot commented Feb 3, 2023 •

edited

Loading

jasoncolburne Feb 4, 2023 •

edited

Loading