node-pre-gyp

[3rd-Eden]

We should explore the possibility to pre-compile binaries using the node-pre-gyp library so people will have less failures when compiling code.

ht/ @lpinca

[DavidSouther]

+1, this is hitting me especially hard in hosted CI environments. The rebuild step can easily take an extra 30 seconds, and rarely outright fails, because of the rebuild.

[lpinca]

I've never used node-pre-gyp but it should be fun to play with it. The only problem I see here, given the popularity of this module, is how to cover S3 costs.

[3rd-Eden]

@leedm777 and that exactly why I haven't looked at it yet. Ideally, this should be something that is solved & stored by npm instead of using these kind of hacks.

I wonder if we could just create ws-<platform> packages and conditionally install them during the install phase so we can still leverage npm. Which might be even a bigger hack, but at least it's cheaper :-P

[DavidSouther]

@3rd-Eden @lpinca

ws is getting ~1 million downloads from the NPM registry a month.

The two binaries, bufferutil.node and validation.node, are 18kb each. Round that to 40kb per install.

US Standard Out pricing is $0.090 / GB. http://aws.amazon.com/s3/pricing/

40kb * 1 million * $0.090 / Gb = $28.80 / mo

My company, ThirdCat LLC, would love to donate $500/year to cover those costs.

[3rd-Eden]

@DavidSouther I'll see what I can do about. Now that we've got ws moved to it's own repository I plan on extracting the binary addons to it's own modules. As use them as optional dependencies in this project so the possiblity of build failures are reduced.

[rosskukulinski]

@3rd-Eden I've just gone through this same process for node-webrtc. I'd be happy to lend a hand if you'd like to move forward with pre-built binaries.

[3rd-Eden]

@rosskukulinski I don't know if it makes sense to pursue this much longer as we moved away from a silly install script which should have been doing optionalDependencies to the actual optionalDependencies support in npm. The primary reason for this feature was to reduce build failures, which seems to be somewhat solved now.

The only thing that I still want to explore is to either have node-gyp-pre publish the binaries to npm's registry or as download files from github so there is hosting overhead and additional accounts needed for contributors to release code.

[ErisDS]

Hi there, I've hunted around for a gitter or IRC to write this - but couldn't find anything, so please direct me elsewhere if there's a better place for this.

I'm currently researching the feasibility of adding ws as a dependency to Ghost and I'm concerned about the impact this would have on our install process. ws is already a dev dependency, and I've seen a few people run into build failures, although I think the versions of ws that are involved are quite old and may be prior to the changes mentioned.

Essentially, I'd like to find out (sorry couldn't find this anywhere) what impact the optional dependencies failing has on functionality and whether it is going to cause confusion due to the error messages. More importantly I am wondering if you would consider there being a greater need, and be more interested in potentially using node-pre-gyp, in order to support Ghost's use case?

node-pre-gyp has been very large factor in Ghost's success - so we're big fans! I completely understand the overheads it introduces though and would be keen to help minimise those in any way that we can.

[lpinca]

@ErisDS the UI when something goes wrong is horrible, but the optional dependencies have no impact on the functionality of the module.
This is the output of npm install ws when VCBuild.exe is not found:

C:\Users\luigi\gh-402
λ npm i ws
\
> [email protected] install C:\Users\luigi\gh-402\node_modules\ws\node_modules\utf-8-validate
> node-gyp rebuild


C:\Users\luigi\gh-402\node_modules\ws\node_modules\utf-8-validate {git}
{lamb} if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (rebuild)
Compilazione dei progetti nella soluzione uno alla volta. Per abilitare la compilazione parallela, aggiungere l'opzione "/m".
MSBUILD : error MSB3428: impossibile caricare il componente di Visual C++ "VCBuild.exe". Per risolvere il prob lema, 1) installare .NET Framework 2.0 SDK, 2) Microsoft Visual Studio 2005 oppure 3) aggiungere al percorso d i sistema il percorso del componente se questo è installato altrove.  [C:\Users\luigi\gh-402\node_modules\ws\n ode_modules\utf-8-validate\build\binding.sln]
gyp ERR! build error
gyp ERR! stack Error: `C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:269:23)
gyp ERR! stack     at ChildProcess.emit (events.js:110:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\luigi\gh-402\node_modules\ws\node_modules\utf-8-validate
gyp ERR! node -v v0.12.2
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok

> [email protected] install C:\Users\luigi\gh-402\node_modules\ws\node_modules\bufferutil
> node-gyp rebuild


C:\Users\luigi\gh-402\node_modules\ws\node_modules\bufferutil {git}
{lamb} if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (rebuild)
Compilazione dei progetti nella soluzione uno alla volta. Per abilitare la compilazione parallela, aggiungere l'opzione "/m".
MSBUILD : error MSB3428: impossibile caricare il componente di Visual C++ "VCBuild.exe". Per risolvere il prob lema, 1) installare .NET Framework 2.0 SDK, 2) Microsoft Visual Studio 2005 oppure 3) aggiungere al percorso d i sistema il percorso del componente se questo è installato altrove.  [C:\Users\luigi\gh-402\node_modules\ws\n ode_modules\bufferutil\build\binding.sln]
gyp ERR! build error
gyp ERR! stack Error: `C:\Windows\Microsoft.NET\Framework\v4.0.30319\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:269:23)
gyp ERR! stack     at ChildProcess.emit (events.js:110:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Users\luigi\gh-402\node_modules\ws\node_modules\bufferutil
gyp ERR! node -v v0.12.2
gyp ERR! node-gyp -v v1.0.3
gyp ERR! not ok
npm WARN optional dep failed, continuing [email protected]
npm WARN optional dep failed, continuing [email protected]
[email protected] node_modules\ws
├── [email protected]
└── [email protected]

C:\Users\luigi\gh-402
λ

This nodejs/node#1010 (comment) explains why the optional utf-8-validate is being used. The other optional dependency (bufferutil) is only used to apply the mask efficiently.

Have you considered faye-websocket? This is what SockJS/Meteor uses.

[3rd-Eden]

@ErisDS As @lpinca explained, the UTF-8 dependency is used to validate that we're receiving correct UTF8. If this module is not installed, it just accepts all the UTF8. This is not really a big problem as you should be always be validating user input in your application so your application logic will already cover this case. As for the bufferutil it provides the performance boost which makes ws the fastest module for websockets in node.

If you are uncertain about the potential impact that ws will have I would suggest taking a look at primus which allows you to transparently switch between different websocket providers and real-time frameworks without having to rewrite your application logic. https://github.com/primus/primus (it also fixes various of browser and implementation bugs so it also results in a more stable connection)

As for the node-pre-gyp, I was hoping that this step would no longer be necessary now that ws has fully transitioned to optional dependencies. If the output of a failing installation affects the user experience it will be more an issue of how npm handles it. The output as seen above should really be hidden by default as it's an optional dependency, so it doesn't matter why it failed anyways.

[lpinca]

Closing this as native addons are no longer dependencies of any kind. They must be installed separately.