Replace gyp, node-gyp for addons by adding compiler abstraction

[eljefedelrodeodeljefe]

EXPERIMENTAL, WIP

Checklist

• make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

Adding platform_tools API

Reducing Build Dependencies

In order to achieve nodejs/CTC#2 I am proposing an API for building node addons, C/C++ sources and Node itself.

This would not be a drop in replacement fro GYP, but rather follow the philosophy that a build system is not necessary for tasks such as ours. The only heavy lifting it does is escaping and putting flags in the right order so calling the respective platforms compilers has a normalized API.

At it's core it does exactly what build systems do, namely calling on a list gcc [src] -c o [target].o and link all of those together gcc [1.o, 2.o, ... , n.o] [executable].o. In GYP and make you'd do that by declaring file locations in a configuration file and then have control flow elements like target and platform conditions. When libraries grow those files pivot to being written with declarative programming.

The dynamic nature of JS is seems to be more suited for that purpose and also does not require users to learn other (and declarative) languages, nor gluing this together with pyhton. Also it requires no dependency, except a node binary of the v4 lts line.

This is based on some experiments I conduct in a library here:
https://github.com/eljefedelrodeodeljefe/platform-tools

Discussion

Why this as API?

Does need to be one. But the intention behind is to curate building issues we have for a long time now and will, imo, not be able to fix them externally. Take gccgo / go build as example.

Native addons as feature should be promoted cross platform. Authoring native addons is really unpleasant.

Why not improve on Gyp?

gyp is a language, which is bad enough. Also its's really chromium specific and a more or less private Google repo. There won't be much improvements there. Also the python dependency is a big hurdle for any platform in order to allow for a nice build experience

Why no make?

Make is also a declarative language one needs to learn and also it is very implicit about parameter passing. The biggest issue with it is that is not cross platform, which makes vcbuild.bat necessary.

Why no vcbuild.bat?

Same as make

Why no IDE integration (for now and / or built-in)

Both Visual Studio and Xcode abstract away the tiniest dependencies, such as (VS) basic C headers like #include <stdio.h>, which requires a properly set up build environment. On windows that would mean calling vcvarsall.bat, but all it does adding those include dir to your path. This lib has the default paths built-in, assuming users have the compilers in default locations

Why no ninja

Ninja is rather a meta build system, which is nice and fast, but won't do much alone. It then requires a set-up build environment. Also it's a binary dependency, users need to install.

Why no build system at all?

Build systems abstract the compiler, linker duality quite a lot and offer some additional features. Features such as timestamp comparison and dependency management are nice on paper, but probably can be done more easily in JS by native JS programmers. Dependency management is probably the prime domain of JS, where every build dependency is a npm install away. Take as opposing example the installation of gyp or running gclient sync. Linker and compiler abstraction might be nice but imo is also dangerous, since you can be easily trapped in a black bock like Visual Studio which suggests, that is doing the heavy lifting, but actually is just running the 500 loc logic presented here. Also running CL.exe directly is more portable approach, since you pretty much run the same logic with gcc.

Can this be used now?

Yes, by omitting the node core compilation step and then run this API. The makes it integratable now and can be improved to build all others C/C++ deps with the goal of getting rid of python and gyp in the long run. Those can be done one by one, with V8 being the biggest bite.

What about sugar features of the other build systems

Change detection and a dynamic dependency management could be implemented by userland. This should just serve as very thin layer for normalizing compiler calls and build stuff barebones.

Modular Core

This would be a first dependency that could be developed outside of core, but be an essential part to it, as discussed in #7098

Known Issues

• I need some help hooking it in with NativeModule.require, since I think it's not possible to require from deps dynamically. Then the consequence is probably compiling all sources
• currently this has a number of external deps, of which the minimal sources are built in. Those being tar-stream and mkdirp. Those are necessary to build addons, since the node-gyp-like workflow needs headers that will be downloaded.

API

Compiling a C Program

int main(int argc, char const* argv[]) {
  return 1;
}

The below would be an example of emulating with Node.js

gcc -c exit_with_1
gcc -o exit_with_1.o
./exit_with_1

const platform_tools = require('platform-tools')
const spawn = require('child_process').spawn

let out = 'exit_with_1'
// first compile without linking
platform_tools.compile('exit_with_1.c', {output: `${out}.o`}, () => {
  // then link the object file (here an easy case)
  platform_tools.link(`${out}.o`, {output: out}, () => {
    // now execute the compiled binary and expect the C-program to end
    // with code 1
    const cp = spawn(out, [], {shell: true});
    cp.on('close', (code) => {
      assert(code === 1), 'Compiled binary exit_with_1 must exit with code 1')
    })
  })
})

Compiling Node Addon

  // the below emualates something very similar to $ on OS X:
  //  c++ -I/Users/jefe/repos/platform-tools/build/deps/node/6.2.0/include/node -I../node_modules/nan  -Os -gdwarf-2 -mmacosx-version-min=10.7 -Wall -Wendif-labels -W -Wno-unused-parameter -std=gnu++0x -fno-rtti -fno-exceptions -fno-threadsafe-statics -fno-strict-aliasing -c -o Release/obj.target/addon/addon.o ../addon.cc
  // c++ -bundle -undefined dynamic_lookup -Wl,-no_pie -Wl,-search_paths_first -mmacosx-version-min=10.7 -L./Release  -o Release/addon.node Release/obj.target/addon/addon.o
  //
  let out = `${process.cwd()}/test/fixtures/sources/addons/addon.cc`
  pt.compileAddon(`${out}`, {output: `addon`}, (err) => {
    if (err) {
      return t.fail(err, 'must not call error here')
    }

    let addon = require('../fixtures/sources/addons/addon')

    t.equal(addon.add(3, 5), 8)
    t.equal(addon.add(10, 5), 15)
  })

Compile Node

Obviously this is rather verbose, but a huge reduction to what we have in the repo and as dependencies.

# build dependencies
./configure --no-target-type && make -j8

Then run the custom node build logic.

'use strict';
const pt = require('../../')
const fs = require('fs')
const assert = require('assert')
const child_process = require('child_process')
const test = require('tape')

test('multiple_objects_exectuable test', function (t) {
  t.plan(2);
  try {
    fs.mkdirSync('build')
  } catch (e) {
    // ignore all errors
  }

  let out = `${process.cwd()}/build/node`

  const targetRoot = `${process.cwd()}/test/fixtures/sources/smoke/node`

  const sources = [
    `${targetRoot}/src/debug-agent.cc`,
    `${targetRoot}/src/async-wrap.cc`,
    `${targetRoot}/src/env.cc`,
    `${targetRoot}/src/fs_event_wrap.cc`,
    `${targetRoot}/src/cares_wrap.cc`,
    `${targetRoot}/src/handle_wrap.cc`,
    `${targetRoot}/src/js_stream.cc`,
    `${targetRoot}/src/node.cc`,
    `${targetRoot}/src/node_buffer.cc`,
    `${targetRoot}/src/node_constants.cc`,
    `${targetRoot}/src/node_contextify.cc`,
    `${targetRoot}/src/node_file.cc`,
    `${targetRoot}/src/node_http_parser.cc`,
    `${targetRoot}/src/node_javascript.cc`,
    `${targetRoot}/src/node_main.cc`,
    `${targetRoot}/src/node_os.cc`,
    `${targetRoot}/src/node_revert.cc`,
    `${targetRoot}/src/node_util.cc`,
    `${targetRoot}/src/node_v8.cc`,
    `${targetRoot}/src/node_stat_watcher.cc`,
    `${targetRoot}/src/node_watchdog.cc`,
    `${targetRoot}/src/node_zlib.cc`,
    `${targetRoot}/src/node_i18n.cc`,
    `${targetRoot}/src/pipe_wrap.cc`,
    `${targetRoot}/src/signal_wrap.cc`,
    `${targetRoot}/src/spawn_sync.cc`,
    `${targetRoot}/src/string_bytes.cc`,
    `${targetRoot}/src/stream_base.cc`,
    `${targetRoot}/src/stream_wrap.cc`,
    `${targetRoot}/src/tcp_wrap.cc`,
    `${targetRoot}/src/timer_wrap.cc`,
    `${targetRoot}/src/tty_wrap.cc`,
    `${targetRoot}/src/process_wrap.cc`,
    `${targetRoot}/src/udp_wrap.cc`,
    `${targetRoot}/src/uv.cc`,
    `${targetRoot}/src/util.cc`,
    `${targetRoot}/src/string_search.cc`,
    `${targetRoot}/src/node_crypto.cc`,
    `${targetRoot}/src/node_crypto_bio.cc`,
    `${targetRoot}/src/node_crypto_clienthello.cc`,
    `${targetRoot}/src/tls_wrap.cc`
  ]

  if (process.platform === 'darwin' || process.platform === 'sunos')
    sources.push(`${targetRoot}/src/node_dtrace.cc`)

  const options = {
    output: `${out}.o`,
    include_headers: [
      `${targetRoot}/src`,
      `${targetRoot}/tools/msvs/genfiles`,
      `${targetRoot}/deps/uv/src/ares`,
      `${targetRoot}/out/Release/obj/gen`,
      `${targetRoot}/deps/v8`,
      `${targetRoot}/deps/cares/include`,
      `${targetRoot}/deps/v8/include`,
      `${targetRoot}/deps/openssl/openssl/include`,
      `${targetRoot}/deps/zlib`,
      `${targetRoot}/deps/http_parser`,
      `${targetRoot}/deps/uv/include`
    ]
  }

  if (process.platform === 'win32') {
    options.include_headers.push(`${targetRoot}/debug/obj/global_intermediate`)

    options.compiler_flags = [
      '/ISRC',
      '/W3',
      '/WX-',
      '/Od',
      '/Oy-',
      '/GF',
      '/Gm-',
      '/RTC1',
      '/MTd',
      '/GS',
      '/fp:precise',
      '/Zc:wchar_t',
      '/Zc:forScope',
      '/Zc:inline',
      '/Gd',
      '/TP',
      '/wd4351',
      '/wd4355',
      '/wd4800',
      '/analyze-'
    ]
    options.defines = [
      'WIN32',
      '_CRT_SECURE_NO_DEPRECATE',
      '_CRT_NONSTDC_NO_DEPRECATE',
      '_HAS_EXCEPTIONS=0',
      'BUILDING_V8_SHARED=1',
      'BUILDING_UV_SHARED=1',
      '\"NODE_ARCH=\\\"ia32\\\"\"',
      'NODE_WANT_INTERNALS=1',
      'V8_DEPRECATION_WARNINGS=1',
      'HAVE_OPENSSL=1',
      // 'HAVE_ETW=0',
      // 'HAVE_PERFCTR=0',
      // 'HAVE_DTRACE=0',
      'FD_SETSIZE=1024',
      '\"NODE_PLATFORM=\\\"win32\\\"\"',
      '_UNICODE=1',
      'HTTP_PARSER_STRICT=0',
      'DEBUG',
      '_DEBUG'
    ]
  } else if (process.platform === 'darwin') {
    options.compiler_flags = [
      `-Os`,
      `-gdwarf-2`,
      `-Wall`,
      `-Wendif-labels`,
      `-W`,
      `-Wno-unused-parameter`,
      `-std=gnu++0x`,
      `-fno-rtti`,
      `-fno-exceptions`,
      `-fno-threadsafe-statics`,
      `-fno-strict-aliasing`
    ],
    options.defines = [
      '_DARWIN_USE_64_BIT_INODE=1',
      'NODE_ARCH="x64"',
      'NODE_WANT_INTERNALS=1',
      'V8_DEPRECATION_WARNINGS=1',
      'HAVE_OPENSSL=1',
      'HAVE_DTRACE=1',
      '__POSIX__',
      'NODE_PLATFORM="darwin"',
      'HTTP_PARSER_STRICT=0',
      '_LARGEFILE_SOURCE',
      '_FILE_OFFSET_BITS=64'
    ]

    pt.compilerUtil[process.platform].compiler.arch(process.arch)
      .forEach(el => options.compiler_flags.push(el))
    pt.compilerUtil[process.platform].osx_min_version('10.5')
      .forEach(el => options.compiler_flags.push(el))

  } else if (process.platform === 'linux') {
    options.compiler_flags = [
      '-pthread',
      '-Wall',
      '-Wextra',
      '-Wno-unused-parameter',
      '-O3',
      '-ffunction-sections',
      '-fdata-sections',
      '-fno-omit-frame-pointer',
      '-fno-rtti',
      '-fno-exceptions',
      '-std=gnu++0x',
    ],
    options.defines = [
      'NODE_ARCH="x64"',
      'NODE_PLATFORM="linux"',
      'NODE_WANT_INTERNALS=1',
      'V8_DEPRECATION_WARNINGS=1',
      'HAVE_OPENSSL=1',
      '__POSIX__',
      'HTTP_PARSER_STRICT=0',
      '_LARGEFILE_SOURCE',
      '_FILE_OFFSET_BITS=64',
      '_POSIX_C_SOURCE=200112'
    ]

    pt.compilerUtil[process.platform].compiler.arch(process.arch)
      .forEach(el => options.compiler_flags.push(el))
  }

  pt.compile(sources, options, (err, files) => {
    if (err)
      t.fail(err, 'Error must not be called')

    const options = {
      output: out,
      linker_flags: []
    }

    const archives = [
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libcares.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libv8_libplatform.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libopenssl.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libzlib.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libhttp_parser.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libuv.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libv8_base.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libv8_libbase.a`,
      `${process.cwd()}/test/fixtures/sources/smoke/node/out/Release/libv8_snapshot.a`,
    ]

    if (process.platform === 'win32') {
      [
        '/NXCOMPAT',
        '/DYNAMICBASE',
        '/MAPINFO:EXPORTS',
        'winmm.lib',
        'gdi32.lib',
        'user32.lib',
        'advapi32.lib',
        'iphlpapi.lib',
        'psapi.lib',
        'shell32.lib',
        'userenv.lib',
        'ws2_32.lib',
        '/MACHINE:X64',
        // '/SAFESEH',
        '/INCREMENTAL:NO',
        '/MAP',
        '/ERRORREPORT:queue',
        '/TLBID:1',
        '/debug',
        '/tlbid:1',
        '/MACHINE:X64',
        `/implib:\"${process.cwd()}/build\\node.lib\"`,
        `\"${targetRoot}\\debug\\lib\\cares.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_libplatform.lib\"`,
        `\"${targetRoot}\\debug\\lib\\openssl.lib\"`,
        `\"${targetRoot}\\debug\\lib\\zlib.lib\"`,
        `\"${targetRoot}\\debug\\lib\\http_parser.lib\"`,
        `\"${targetRoot}\\debug\\lib\\libuv.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_base_0.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_base_1.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_base_2.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_base_3.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_libbase.lib\"`,
        `\"${targetRoot}\\build\\debug\\lib\\v8_snapshot.lib\"`
      ].forEach(flag => options.linker_flags.push(flag))
    } else if (process.platform === 'darwin') {
       [
        `-Wl,-force_load,${targetRoot}/out/Release/libopenssl.a`,
        `-Wl,-force_load,${targetRoot}/out/Release/libv8_base.a`,
        '-Wl,-search_paths_first',
        '-mmacosx-version-min=10.5',
        '-arch', 'x86_64',
        '-framework',
        'CoreFoundation',
        '-lm'
      ].forEach(flag => options.linker_flags.push(flag))

      options.tail_flags = archives
    } else if (process.platform === 'linux') {
      [
        '-Wl,--whole-archive',
        `${targetRoot}/out/Release/libopenssl.a`,
        '-Wl,--no-whole-archive',
        '-Wl,--whole-archive',
        `${targetRoot}/out/Release/obj.target/deps/v8/tools/gyp/libv8_base.a`,
        '-Wl,--no-whole-archive',
        '-Wl,-z,noexecstack',
        '-pthread',
        '-rdynamic',
        '-ldl',
        '-lrt',
        '-lm'
      ].forEach(flag => options.linker_flags.push(flag))

      pt.compilerUtil[process.platform].linker.arch(process.arch)
        .forEach(el => options.linker_flags.push(el))

      options.tail_flags = archives
    }

    pt.link(files, options, (err, file) => {
      if (err) return t.fail(err, 'Error must not be called')

      const e = child_process.spawn(`${process.platform === 'win32' ? '': './'}node`, ['-v'], {cwd: 'build', shell: true});
      e.stdout.on('data', (data) => {
        t.ok(data.toString().indexOf('v4.4.5') === 0, 'Match version 4.4.5')
      });
      e.on('error', (err) => {
        if (err) return t.fail(err, 'Error must not be called')
      });
      e.on('close', (code) => {
        t.ok(code === 0, 'Compiled binary node must exit with code 0')
      });
    })
  })
})

Building V8

pending

Building libuv

pending

Building other C deps and js2c run

pending

[mscdex]

/cc @indutny

[indutny]

I really like the idea, but it is hard for me to tell how useful it will be for the current users. We need to gather more information about current binding.gyp files, and what GYP features they use. Certainly some of them are optional for these projects, but some are not.

The problem with simplified system is that it is pretty hard to build things that needs lots of features, and considering that this eventually may become the only build system - this may limit addons.

[rvagg]

/cc @nodejs/addon-api

I've been through a lot of ecosystem binding.gyp, and I think I can safely assert that they are almost all using only the basic declarative logic in there - if linux do this, if windows do this, if osx do this - they may as well just have a makefile and a .bat and most could probably be easily replaced with this kind of tooling.

Being able to export some defaults from core like we do with common.gypi would be nice, although that's a bit of a dogs breakfast and it'd be nice to do something much cleaner than that since it's the cause of a lot of binding.gyp pain that exists out there.

I'd like to see how much effort this takes to add v8 support for this, that seems like a pretty large job. Any idea on what amount of work that would be @eljefedelrodeodeljefe?

[jasnell]

@eljefedelrodeodeljefe ... still want to pursue this? The likely better course would be to engage with the new Node.js API (abstraction API) work

[eljefedelrodeodeljefe]

@jasnell not sure what you mean. I would want to pursue this, but it's likely to be crunched by the related discussions about the future of gyp.

I am a little bit wary arguing for a slim JS thingy.

[stevenvachon]

I think that this is worth reopening, here or elsewhere.

[eljefedelrodeodeljefe]

@stevevachon why do you think so? This will very likely never go forward.

[stevenvachon]

If it were backwards compatible with gyp, it would provide time to deprecate.

[refack]

I think that this is worth reopening, here or elsewhere.

There is hope, gyp.js is getting nearer:
nodejs/node-gyp#960
nodejs/node-gyp#962
nodejs/node-gyp#1092

[stevenvachon]

Yep, and while that's great, I don't think that gyp should be used for anything more than backwards-compatibility moving forward.

[refack]

Yep, and while that's great, I don't think that gyp should be used for anything more than backwards-compatibility moving forward.

@stevenvachon need a viable working alternative. Are you willing to commit to platform-tools? (see my comment in CTC#2)

[stevenvachon]

I don't have any prior gyp experience, but it might not be very difficult if another library can handle the heavy gyp parsing. platform-tools#14

[bnoordhuis]

GYP does quite a lot though. "Another library" would basically be a reimplementation of GYP.

You may be able to leave out some things but I think you'll end up being dismayed at the amount of functionality you will have to replicate.

[refack]

if another library can handle the heavy gyp parsing

we do have gyp.js now, so a marriage of the two sounds fantastic. gyp.js parsing then passing the ball to platform-tools (instead of ninja)... WIN!

[eljefedelrodeodeljefe]

@bnoordhuis I am not really convinced that gyp does a lot. Also I think this disregards a lot of things that are more crucial, like usability, which I hope we agree, is insufficient.

[refack]

like usability, which I hope we agree, is insufficient.

Ohh yea, 👎 on usability. But if you have a way to eat .gyp files you have a migration path forward...

[bnoordhuis]

I am not really convinced that gyp does a lot.

Famous last words! All I can say is try it and see how far you get before you run out of steam.

You could aim for a 80/20 solution but popular add-ons like node-sass and node-iconv are in that 20% category.

[eljefedelrodeodeljefe]

See, I already did one year ago https://github.com/eljefedelrodeodeljefe/platform-tools. I mean, you know that I like you, but you are quite conservative about initiatives like that :) While that might good for personal mental hygiene, you could at least be supportive stating that you like the general idea or not, apart from the efforts (you don't have to take). I don't care too much about my hygiene for example :)

[eljefedelrodeodeljefe]

The 80/20 things is of course correct. I would aim for Node addons and node proper first. That leaves complicated native addons and v8 / libuv. I think that already is great help for the ecosystem and node development (except those who integrate v8)

[refack]

node-sass and node-iconv

Challenge accepted ⚔ eljefedelrodeodeljefe/platform-tools#15

[bnoordhuis]

@eljefedelrodeodeljefe I'm supportive, I'd like to get rid of GYP and node-gyp, but I'm also a realist: a great heard of yaks are going to get a new haircut before any successor is a drop-in replacement.

There is a lot of institutional knowledge encoded in node-gyp and (particularly) GYP that you will have to figure out for yourself and replicate. I'll be happy to assist with moral support, though. :-)

(In all seriousness, I know a great deal about both projects. If you have questions about why something works the way it does, don't hesitate to ask.)

[BridgeAR]

Ping @eljefedelrodeodeljefe

[eljefedelrodeodeljefe]

I am gonna take a shot at an implementation in the second half of September. Until then I will review related discussions. I am happy that @bnoordhuis is kinda in support. Sorry for being abscent.

[jasnell]

Hey! Welcome back! No apology necessary!

[BridgeAR]

Ping @eljefedelrodeodeljefe again

[BridgeAR]

Closing due to long inactivity. @eljefedelrodeodeljefe please feel free to reopen if you would like to pursue this further.