Add browser support #63

bryanchriswhite · 2022-06-17T12:58:00Z

Summary

This PR re-organizes the work from fetchai/blst-ts#1 in an attempt to make it easier to review and to revert unnecessary changes. This work has also been released on npm as @fetchai/blst-ts.

While all applicable tests are passing in a browser environment, I'm reluctant to say that this "closes" #18 because it is missing bindings to the constants:

BLS12_381_G1
BLS12_381_NEG_G1
BLS12_381_G2
BLS12_381_NEG_G2

The rest of the bindings were to functions, which is well documented, but I was neither able to to find documentation on nor come up with a way to bind to C/C++ constants in the time I've spent on this so far. I would be extremely receptive to any help in this area! 😄

~~The blst submodule is pointed at the current HEAD of supranational/blst#121 but should be updated to the main branch when possible.~~

Dependencies

~~supranational/blst#121~~

CLAassistant · 2022-06-17T13:02:53Z

All committers have signed the CLA.

dot-asm · 2022-06-19T10:55:36Z

I hope you don't mind if I ask some questions and comment. The goal is to gather some understanding of the platform and corresponding challenges. Maybe there is something that would be appropriate to do slightly differently on the blst side, more specifically in blst.hpp...

dot-asm · 2022-06-19T11:19:58Z

prebuild/emscripten/blst.d.ts

+// BLS12_381_G1: P1_Affine;
+// BLS12_381_NEG_G1: P1_Affine;
+// BLS12_381_G2: P2_Affine;
+// BLS12_381_NEG_G2: P2_Affine;


I'd suggest not aiming for providing access to these. There are generator() methods that serve the purpose. Though one can make a case for constant generators. I mean generator() methods create mutable copies... Which can be viewed as an extra burden, because as far as I understand you're responsible for freeing all the objects explicitly. One can reformulate the question as following. Are there cases when constant generator points would be ultimately useful? One common case is the PT constructor, but it should be noted that the underlying subroutine, blst_miller_loop, actually replaces null references with generators...

This one has me in a bit of an awkward position. 😅 My original intention was to replace the native binding, leaving the API layer unchanged. I don't know enough to have strong opinion on this but perhaps @dapplion can weigh in here?

Well, sometimes one has to recognize the limitations. I mean webasm interface is purely functional and one can't share constant data in the same way. As already mentioned, there are generator() methods and they are available even in native bindings. So that it's possible to adapt the existing code to avoid the limitation of one [platform]... But sure, let's first hear how bad is it. In worst case one can define them as constants, and have C++ side recognize them and replace with pointers to constant data...

Is there are way to execute some code upon module load? There should be... It looks like it would be possible to instantiate the objects by having contructors recognize the constants and returning pointers to constant data... Though problem would be that the data won't be actually constant. I mean unlike the native bindings where attempt to manipulate would trigger a crash, here a buggy program would get away with otherwise erroneous modification, which would incur hard-to-find bugs later on... Again, generator() methods would be preferable... But again, let's first hear...

Yes, see Module.onRuntimeInitialized. I am using it in post.js:982-988.

Though problem would be that the data won't be actually constant... hard-to-find bugs later on...

On closer examination maybe it's not that bad. In sense that the constant points in question are affine and the affine class doesn't have methods that would modify the data. In other words an application won't have an easily accessible way to wreck havoc...

(additional lifecycle hooks are also available)

This one has me in a bit of an awkward position. sweat_smile My original intention was to replace the native binding, leaving the API layer unchanged. I don't know enough to have strong opinion on this but perhaps @dapplion can weigh in here?

Since this library is eventually wrapped by chainsafe/bls I don't think it necessary that bindings and wasm API are the same. If wasm exports only a functional API that fits consumer ergonomics, then that's fine.

(additional lifecycle hooks are also available)

Could you provide some pointers? Like a real-life usage example and a specification or something...

Unfortunately, no. Usage of the Module.onRuntineInitialized hook and the emscripten docs that I referenced above contain the full extent of my understanding at the moment.

dot-asm · 2022-06-19T11:30:15Z

prebuild/emscripten/post.js

+    case "Uint8Array":
+      return [ensureUint8(val), val.length];
+    // TODO:
+    // case BigInt:


This one would go something like this

if (val < 0) { throw new Error(`expecting unsigned value`); } var temp = []; while (val != 0) { temp.push(Number(val & 255n)); val >>= 8n; } return [ensureUint8(temp), temp.length];

[As you can tell, I don't have a firm grasp on how things get to the wasm memory. I only know that it's separate, that Javascript can read and write it, while wasm can not read, let alone write, the Javascript memory. Right?]

Thanks! This binary logic seems very on-brand 😜

I don't think webpack will like the bigint literals, so maybe don't be surprised if BigInt(...)s come around later.

In essence the idea would be to fully harmonize with blst.hpp.ts.

dot-asm · 2022-06-19T11:46:05Z

prebuild/emscripten/post.js

+  var out = ensureCache.alloc(new Uint8Array(32), HEAPU8);
+  _emscripten_bind_SecretKey_to_lendian_1(self, out);
+  return new Uint8Array(HEAPU8.subarray(out, out + 32));
+  // TODO: free `out`?


My understanding is that Uint8Array(wasm-pointer, ...) provides direct access to wasm memory. My question is [if] HEAP8.subarray() makes a copy of the data from wasm memory to the Javascript pool. (I apologize if the question is stupid, I'm not that familiar with interactions between wasm and Javascript.)

The Internet suggests that one might just as well return a pointer to the webasm stack, which you then immediately copy to the Javascript pool. I suppose this means that data sits safely on the stack till the next call ~~from~~ to the corresponding webasm module. This way you don't need to allocate a temporary buffer and figure out when it can be freed...

My question is [if] HEAP8.subarray() makes a copy of the data from wasm memory to the Javascript pool.

From what I gather .subarray doesn't, but in the context new Uint8Array does the copy. As for freeing out. My understanding is that ensureCache allocates a chunk of wasm memory and reuses it specifically for argument passing. It has no free method.

dot-asm · 2022-06-19T14:35:35Z

prebuild/emscripten/post.js

+  if (ikm && typeof ikm === 'object') {
+    if (ikm instanceof Uint8Array) {
+      ikm_len = ikm.length;
+      ikm = ensureUint8(ikm, ikm.length);


ensureUint8 doesn't take length. So passing ikm.length is redundant, right? What might be appropriate is to double-check that the length is not shorter than 32.

ensureUint8 doesn't take length. So passing ikm.length is redundant, right?

Haha, so it would seem. 🤦

dot-asm · 2022-06-19T14:53:10Z

prebuild/emscripten/post.js

+    }
+  } else {
+    ikm_len = ikm.length;
+    ikm = ensureString(ikm);


Formally speaking the ikm_len can be deceptive in the context, String. The [String's] length method returns amount of characters, while what should reach C is array of bytes. The lengths of the two are not the same if there are non-ASCII characters. BTW, I don't see the intArrayFromString implementation and it doesn't seem to be standard. Where is it? In some dependency? Just in case, expectation is that String would be encoded as UTF-8...

I don't see the intArrayFromString implementation

It's provided by emscripten as part for the current module interface, right? As for ikm_len, I gather it should be ikm_len = lengthBytesUTF8(ikm);

intArrayFromString is part of emscripten's preamble.js API. When emcc builds the WASM module, it also dumps the preamble into the accompanying JavaScript (before dead-code elimination).

Just in case, expectation is that String would be encoded as UTF-8...

Correct, is there a potential pitfall there?

Just in case, expectation is that String would be encoded as UTF-8...

This came out wrong. Expectation is that all strings, not "String" as in "Javascript String", that are given to blst would be encoded as UTF-8. And the trouble is that "Javascript String"-s are stored as UTF-16. intArrayFromString does the UTF-16->UTF-8 conversion, but lengths' meanings are different. On the Javascript side it's amount of UTF-16 characters, while on blst side it's the amount of bytes it takes to encode the string in question in UTF-8. To give an example. In Javascript "Straße".length is 6, while the amount of bytes corresponding UTF-8 string takes is 7.

Understood, thanks for clarifying 👍

dot-asm · 2022-06-19T15:00:07Z

prebuild/emscripten/post.js

+  var self = this.ptr;
+  ensureCache.prepare();
+  if (typeof input == 'object') { input = ensureInt8(input); }
+  _emscripten_bind_SecretKey_from_bendian_1(self, input);


Who deallocates input? BTW, why is it copied to stack in the bind method? Is it actually required? BTW, I would argue that a buffer length check would be appropriate here.

I'm used to believing that webasm can't access the Javascript pool. But now I see things about emscripten::val and it looks like one can create a memoryView of webasm memory in C++ and call its "set" method to copy data from the Javascript pool. Do you have any experience with that? Idea would be that instead of allocating input from Javascript one would pass the Uint8Array Javascript object, have C++ copy data to stack and deserialize it into SecretKey. Same technique can be applied to all deserialization methods...

Who deallocates input?

No need, it's in ensureCache, it will be reused next time one passes arguments to a method. Length check would still be more than appropriate.

emscripten::val

It doesn't seem to be possible to pass them as arguments.

dot-asm · 2022-06-19T15:18:56Z

Is it correct understanding that the glue code was first auto-generated and then manually fixed up? If so, maybe it would be appropriate to exercise approach similar to blst/bindings/go/generate.py. I.e. instead of applying same fix-ups to P1 and P2, one would put together P1 and then transliterate it to P2? So the each time P1 is modified, P2 is modified accordingly with zero effort. It's even possible to keep it in one single file. If there is interest, I can make more concrete suggestion...

dot-asm · 2022-06-19T16:04:48Z

exercise approach similar to blst/bindings/go/generate.py

One can even get more creative and interleave C++ and Javascript in the same file, method by method, for better overview...

dot-asm · 2022-06-19T16:32:21Z

exercise approach similar to blst/bindings/go/generate.py

One can even get more creative and interleave C++ and Javascript in the same file, method by method, for better overview...

Just to give an idea:

#!/usr/bin/env python3

js = """
P1_Affine.prototype['dup'] = P1_Affine.prototype.dup = /** @suppress {undefinedVars, duplicate} @this{Object} */function() {
  var self = this.ptr;
  return wrapPointer(_emscripten_bind_P1_Affine_dup_0(self), P1_Affine);
};;
"""
cpp = """
blst::P1_Affine* EMSCRIPTEN_KEEPALIVE emscripten_bind_P1_Affine_dup_0(blst::P1_Affine* self) {
  return new blst::P1_Affine(self->dup());
}
"""
js += """
P1_Affine.prototype['to_jacobian'] = P1_Affine.prototype.to_jacobian = /** @suppress {undefinedVars, duplicate} @this{Object} */function() {
  var self = this.ptr;
  return wrapPointer(_emscripten_bind_P1_Affine_to_jacobian_0(self), P1);
};;
"""
cpp += """
blst::P1* EMSCRIPTEN_KEEPALIVE emscripten_bind_P1_Affine_to_jacobian_0(blst::P1_Affine* self) {
  return new blst::P1(self->to_jacobian());
}
"""

print(cpp) # to cpp file, then replace P1 with P2 and emit again
print("8<------" * 10)
print(js)  # to js file, then replace P1 with P2 and emit again

dot-asm · 2022-06-20T18:15:29Z

My suggestion would be to provide an empty .idl file that simply lists the classes:

[Prefix="blst::"] interface P1_Affine { };
[Prefix="blst::"] interface P1 { };
[Prefix="blst::"] interface P2_Affine { };
[Prefix="blst::"] interface P2 { };
[Prefix="blst::"] interface SecretKey { };
[Prefix="blst::"] interface PT { };
[Prefix="blst::"] interface Pairing { };

This will be used to re-generate empty bindings and all those Javascript helpers [in case it's deemed necessary]. Then one would use a script that interleaves C++/Javascript snippets to append manually written methods to previously generated files. And override the constructors.

As for P1->P2 transliteration. A better way to do it, better than bindings/go/generate.py that is, is to do something along following:

def xchg_1vs2(matchobj):
    if matchobj.group(2) == '1':
        return matchobj.group(1) + '2'
    else:
        return matchobj.group(1) + '1'

...

print(cpp, file=cpp_fd)
print(js, file=js_fd)
print(re.sub(r'((?<!f)[pgPG])([12])', xchg_1vs2, cpp), file=cpp_fd)
print(re.sub(r'((?<!f)[pgPG])([12])', xchg_1vs2, js), file=js_fd)

[Just in case, I'm using this method to generate C# bindings.]

bryanchriswhite · 2022-06-21T10:30:02Z

My suggestion would be to provide an empty .idl file that simply lists the classes:

...

This will be used to re-generate empty bindings and all those Javascript helpers [in case it's deemed necessary]. Then one would use a script that interleaves C++/Javascript snippets to append manually written methods to previously generated files. And override the constructors.

I love this idea 😍! It's as if you've done this before 😉

dapplion · 2022-06-21T13:04:11Z

@bryanchriswhite so happy to see this PR here in a such a great form, I sincerely appreciate your perseverance ❤️

Also thank you @dot-asm for jumping in already with valuable suggestions. Your help was instrumental in delivering blst-ts in the first place.

Will do an in-depth review latter this week

dapplion · 2022-06-21T13:16:04Z

src/scripts/inline_bin.ts

+// fs.createReadStream(inputPath);
+const data = fs.readFileSync(inputPath);
+const js_contents = `
+import { Buffer } from "buffer";


Is the buffer library already used for some other purpose or only imported here to convert to base64?

Just for base64 decoding. Is there a more browser-idiomatic way you're aware of?

We've been using this library https://www.npmjs.com/package/uint8arrays

Gotcha, my thinking was that Buffer was already available via the webpack polyfill. Looking closer, I'm noticing that maybe all but one usage of Buffer is in tests.

dapplion · 2022-06-21T13:19:38Z

webpack.config.js

+    "P1.test.ts",
+    "P2.test.ts",
+    "Pairing.test.ts",
+    "SecretKey.test.ts",


Why is it necessary to list test files explicitly here?

Oh, probably no reason. My workflow involved commenting and uncommenting various sets of tests while debugging.

dapplion · 2022-06-21T13:20:29Z

test/unit/lib/index.test.ts

-describe("bls lib", () => {
-  const n = 3;
+// TODO: more robust wait for `Module` to initialize
+setTimeout(() => {


Should this use the maybeWaitForRuntime pattern too?

Indeed, nice catch!

dapplion · 2022-06-21T13:22:29Z

test/utils.ts

@@ -66,3 +68,11 @@ export function runInstanceTestCases<InstanceType extends {[key: string]: any}>(
    }
  }
 }
+
+export function maybeWaitForRuntime(callback: () => void): void {


To run some steps before running any test code mocha has some strategies, can you check if these ones play nice with web? Would definitely make the setup simpler.

https://mochajs.org/#root-hook-plugins

https://mochajs.org/#global-setup-fixtures

Thanks for pointing this out, I was unaware!

dapplion · 2022-06-21T13:25:37Z

I'm gonna think more what's the best way to provide both bindings and wasm in the same library. @bryanchriswhite do you think wasm has utility in NodeJS or may only be used in web?

CC @wemeetagain please take a look, specially how this integrates into chainsafe/bls and esm modules

bryanchriswhite · 2022-06-21T18:12:16Z

do you think wasm has utility in NodeJS or may only be used in web?

@dapplion I believe that node supports the same WebAssembly API as the browser these days and emscripten's preamble is intended to support both. So I'm pretty confident that one could use the wasm binding in node as well.

dot-asm · 2022-06-29T14:32:36Z

This will be used to re-generate empty bindings and all those Javascript helpers [in case it's deemed necessary]. Then one would use a script that interleaves C++/Javascript snippets to append manually written methods to previously generated files. And override the constructors.

This last part, overriding the constructors, is not possible. Hence https://github.com/dot-asm/blst/tree/emscripten/bindings/emscipten, as a starting point...

dapplion · 2022-06-29T15:18:41Z

@bryanchriswhite ping, reached out via email from your Github profile

dot-asm · 2022-07-04T14:06:22Z

There was typo in directory name, fixed. And runnable.js is equivalent of node.js/runnable.js now...

dot-asm · 2022-07-08T11:02:02Z

https://github.com/dot-asm/blst/tree/emscripten/bindings/emscripten is essentially ready to roll. "Essentially" is an invitation to check it out and provide feedback. Once it's done, it can go to the main blst repo. build.py creates blst.wasm and blst.js in current working directory. And it passes through additional command line flags to emcc, e.g. /some/where/blst/bindings/emscripten/build.py -O2 --closure 1. blst.js interface aims to be fully compatible with the node.js module.

matthewkeil · 2024-04-11T10:02:38Z

Closing this PR. Browser support is provided via https://github.com/supranational/blst/tree/master/bindings/emscripten and also via https://github.com/ChainSafe/bls . This repos now hosts napi async bindings for multi-threaded verification and is node specific. There is some discussion about building multi-threaded wasm or emscripten bindings but that is still a future "wish list" thing. If this work becomes relevant again we can open a fresh PR and refer back to this one for the comment history. Thank you for your work on this @bryanchriswhite. If you have any questions or concerns please feel free to reach out to us on our discord and we will be happy to chat 😄

bryanchriswhite added 13 commits June 17, 2022 14:16

refactor: move prebuild/* to prebuild/swig/*

0b1645e

chore: isolate paths members which require fs

19751a0

fix: make BINDINGS_DIR platform agnostic

2e6aeeb

chore: update blst submodule

1a00e65

feat: add emscripten binding code & types

94488e0

feat: add emscripten build & bin (wasm) inline scripts

81db0d4

tests: add deps and configs for webpack and karma

f8d22b7

fixup: emscripten build script

ab750be

tests: nest applicable tests in init callback

bb009e7

feat: switch to emscripten bindings if not node or BINDING=emscripten

abf76a9

todo: support remaining interface

daa96bb

chore: update yarn.lock

be02a38

docs: update readme

f18f08c

bryanchriswhite requested a review from a team as a code owner June 17, 2022 12:58

bryanchriswhite mentioned this pull request Jun 17, 2022

build.sh: better support for use with emscripten supranational/blst#121

Closed

dot-asm reviewed Jun 19, 2022

View reviewed changes

dapplion reviewed Jun 21, 2022

View reviewed changes

dapplion requested a review from wemeetagain June 21, 2022 13:29

wemeetagain force-pushed the master branch from 6c33ba5 to 302c0ef Compare December 15, 2022 17:04

philknows mentioned this pull request Feb 28, 2024

Stale PR Cleanup on External repos ChainSafe/lodestar#6494

Closed

86 tasks

matthewkeil closed this Apr 11, 2024

Add browser support #63

Add browser support #63

Conversation

bryanchriswhite commented Jun 17, 2022 • edited Loading

Summary

Dependencies

CLAassistant commented Jun 17, 2022 • edited Loading

dot-asm commented Jun 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dot-asm Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dapplion Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dot-asm Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

bryanchriswhite Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dot-asm Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

dot-asm Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dot-asm Jun 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanchriswhite Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

dot-asm Jun 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dot-asm commented Jun 19, 2022

dot-asm commented Jun 19, 2022

dot-asm commented Jun 19, 2022

dot-asm commented Jun 20, 2022 • edited Loading

bryanchriswhite commented Jun 21, 2022

dapplion commented Jun 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dapplion commented Jun 21, 2022

bryanchriswhite commented Jun 21, 2022 • edited Loading

dot-asm commented Jun 29, 2022

dapplion commented Jun 29, 2022

dot-asm commented Jul 4, 2022

dot-asm commented Jul 8, 2022

matthewkeil commented Apr 11, 2024

bryanchriswhite commented Jun 17, 2022 •

edited

Loading

CLAassistant commented Jun 17, 2022 •

edited

Loading

dot-asm Jun 21, 2022 •

edited

Loading

dapplion Jun 21, 2022 •

edited

Loading

dot-asm Jun 19, 2022 •

edited

Loading

bryanchriswhite Jun 21, 2022 •

edited

Loading

dot-asm Jun 19, 2022 •

edited

Loading

dot-asm Jun 19, 2022 •

edited

Loading

dot-asm Jun 19, 2022 •

edited

Loading

bryanchriswhite Jun 21, 2022 •

edited

Loading

dot-asm Jun 21, 2022 •

edited

Loading

dot-asm commented Jun 20, 2022 •

edited

Loading

bryanchriswhite commented Jun 21, 2022 •

edited

Loading