C/C++ transpiling for addons

[indutny]

I suggest exploring following way of writing addons.

Concept

Instead of including particular engine headers and working with engine's APIs (v8::*), addon may look like a plain C/C++ program:

void some_fn(int value, js_str_t str) {
  // ...
}

The transpiler will take such function and rewrite it in a way that will make it compatible with engine. For example, the code above may result in something like this:

void some_fn(const FunctionCallbackInfo<Value>& args) {
  int value = args[0]->ToInteger();
  js_str_t str;
  String::Utf8Value tmp(args[1]->ToString());

  str.data = *tmp;
  str.length = tmp.length();

  // addon code
}

Benefits

This approach provides three important benefits:

• Independence of particular JS engine
• No need for pre-compiling stuff, node.js will carry clang with transpiler
• Simple way to test functions using just C/C++ APIs without initializing V8 or whatever engine

Some further ideas

Struct-object

typedef struct my_obj_t {
  int x;
  js_str_t str;
};

void my_fn(my_obj_t* obj) {
  // Accessing object, or updating its properties will update relevant JS object
}

Function

typedef void (*ext_fn_t)(int x, js_str_t str);

void my_fn(ext_fn_t ext) {
  // This will invoke JS function referenced by `ext`
  ext(1, JS_STR("..."));
}

[indutny]

cc @nodejs/v8 @Fishrock123 @trevnorris @jasnell and everyone interested. Let's have some discussion about this.

[indutny]

Also @bnoordhuis

[bnoordhuis]

node.js will carry clang with transpiler

He said, as if it were nothing.

[kobalicek]

Having a working clang as a part of node-gyp would be nice, but limiting all node users to use only clang compiler to compile native addons is kind of wrong, sorry.

[stefanmb]

Maybe I'm missing something obvious, but part of the API WG discussion was to isolate module users from having to compile anything at all at install time, I'm not sure how that can be achieved if we need to bundle clang for source-to-source translation?

[kobalicek]

I see bundling clang as a part of node-gyp as an opportunity to silent those complaining about their broken Windows installations.

[indutny]

@bnoordhuis sarcasm != argument . What are our objections to it? Binary size? Licensing?

@kobalicek this is not going to be the only one way to compile addons. Just the suggested one.

@stefanmb sorry, I wasn't participating in that discussion. May I ask you what was the motivation for it?

[kobalicek]

@stefanmb I don't see a reason why to isolate developers from compiling native addons. Native addons are native. If you want to isolate users from compilation then the compilation can happen somewhere on the server-side and npm can check if it's already compiled for a particular version/arch/os, and then download the binary. But I think all of this effort would be just for Windows' users, has anybody using Linux/BSD ever complained about compilation?

[bnoordhuis]

sarcasm != argument . What are our objections to it? Binary size? Licensing?

@indutny It's not sarcasm. You're proposing to take on a dependency on a (very) large project. We'll be on the hook for making sure it builds and works on the platforms we support.

There is inevitably a maintenance cost, possibly a large one. I'm not saying it's a bad idea per se but your original post doesn't address that.

[indutny]

@bnoordhuis no-no-no, this is not what I suggest. I suggest to bundle binaries with us! :) There are already lots of prebuilt libraries for chromium, we can just do the same thing.

[kobalicek]

@indutny I think you approach has one problem - it completely hides the fact that the function is called by a VM. I'm author of only one addon, but it's a gigantic one and I can say this won't work for me. Calling functions without VM doesn't make any sense to me. It can work when you need to sum two variables, but it just doesn't work when you need to do something regular - check values, create objects, iterate objects, throw, catch, etc... it basically prevents any interaction with the VM.

Here is some work that I have done (http://blend2d.com/screencast.html) and basically I'm trying to develop an agnostic VM API that can address my use case. If I can address this then I'm sure it can be extended to address more use cases.

[indutny]

@kobalicek I think it is possible to add value type checks, object creation, and iteration into the proposed model. It just needs to be as abstract as possible.

Surely, it disallows direct interaction with VM. However, it may work for many use cases. For example, take bignum module which is quite popular. Technically, it is mostly about providing bindings to the OpenSSL's C API. There is no need for direct VM interaction there.

[joshgav]

@indutny your proposal may work for simple modules, or modules that only need a couple straightforward system calls. In fact, an FFI may be a similar but better approach for such modules, as an FFI with cooperation from VM vendors could completely shield the native developer from anything related to JS or V8.

If we could have everything proposed during the summit, I'd think FFI would address simple modules or modules with simple needs from native; and a true native API would address complex ones.

[kobalicek]

I think there should be no difference between a simple and complex module, that seems worse than anything else to me

[indutny]

@joshgav the difference between FFI and my approach is that my approach allows modifying JS objects and calling back to JS-land, while FFI does not really allow this.

Unfortunately, remote participation was really limited during the summit. I wish I could say more there.

[indutny]

@kobalicek I agree with you. What I personally mean by complex module is a kind of module that does coupling to particular JS engine. Say the module like ref or heapdump.

[kobalicek]

So having some kind of wrapper that still allows you to interact directly with the VM is fine no? This is actually the approach that I'm trying. I mean API :)

[indutny]

coupling != interaction. This mean reliance on particular API provided only by a particular VM.

[kobalicek]

What I mean is that if the neutral API provides the least common denominator, but some module requires some let's say V8 functionality that is really V8 specific there should be a way to access it. This is my main reason to maintain source-code compatibility, but to not enforce ABI compatibility.

[indutny]

Of course! This should not block anyone from writing vm-specific addons. I'm just saying that these are rare cases, that we should support, but not should not take as a major example.

[benjamingr]

@indutny so to be clear - you're not suggesting adding clang as a dependency of node - rather than that - a binary produced by it that performs some transformation?

Why would clang be used in particular? Can't the above transformations be written in a just-slightly-uglier way with regular templates and macros? I'm sure I'm not considering many cases but for the cases you've outlined above I think it's possible if we have a reasonable (c++11) compiler which we require anyway for building Node add-ins.

[jeisinger]

One of the stated goals for native modules was that it should be possible to update node within one LTS branch without recompiling. With this approach, this won't work.

(not having to compile modules at all was a goal for FFI, not for the native modules API in general).

On a high-level using clang to transpile is similar to the idea of describing the exposed surface using something like IDL and having a custom IDL compiler in node, so instead of transforming the function some_fn in place, we'd end up with separate code that calls to some_fn from JS bindings code and vice versa. My feeling is that this is a more tractable problem.

[indutny]

@jeisinger do you think IDL could be used to call JS from C/C++ too? What about property loads/stores? Do we want to just expose structs to JS and assume that JS will do the rest?

[ofrobots]

@indutny At the VM summit we discussed a dynamic, JIT supported FFI. I think it should be possible to support JS calls from C/C++. The FFI needs to generate/provide a C callable function that can wrap the JS function. Structs could be handled using proxies.

Or think about it differently; if you can write a transpiler to do a specific source transformation, there is no reason a dynamic FFI couldn't generate similar code at runtime. The biggest benefit of FFI above your proposal is ABI compatibility.

[jeisinger]

To call js from c++ or to modify js objects, you'd have to use a very strictly defined syntax that the transpiler would be able to recognize. At that point, you could just provide a c/c++ API for calling/modifying js

[indutny]

You are right. Closing. Thank you!