Prebuilt binaries

[coderaiser]

Would be great if node-pty use something like prebuild to avoid compiling on every install. Especially on windows. Alot people will appreciate it.

[Tyriar]

While I'm not the biggest fan of including pre-built binaries it really makes sense particularly for winpty which will never change and takes quite a while to compile. I believe that is also the lib that has the dependencies that causes issues.

I'd rather not build and ship a binary for every single version of node though.

[Tyriar]

WIP branch https://github.com/Tyriar/node-pty/tree/46_include_winpty_binaries

[Tyriar]

Remaining work:

• Both ia32 and x64 architectures of winpty need to be supported depending on the requested build type and linked correctly in binding.gyp.
• Get it working in VS Code, last time I checked ./demo/fork_demo.js worked fine but when I tried to include it in VS Code it wasn't working.
• Figure out how to best clean the node-pty module binaries so that only the current arch is included on install. Downloading the relevant binaries might be the best idea, I tried to programmatically extract the zip from https://github.com/rprichard/winpty/releases but all node unzip modules failed to parse the release zip file. This is a requirement of VS Code's build system.

[stevenvachon]

Why not use is-windows to decide whether to run npm install windows-build-tools as a post-install script?

[Tyriar]

The build is already too long imo, the better fix is to remove the build all together as it's not necessary since nothing in the winpty lib is modified.

[stevenvachon]

Can this maybe be pushed to 0.8.0 so that 0.7.0 can be released? My Travis CI builds are failing.

[Tyriar]

@stevenvachon why not run npm install windows-build-tools before installing on Travis?

[Tyriar]

Wait, Travis support Windows now? Do you mean appveyor?

[stevenvachon]

I'm running both Travis and AppVeyor. I'm using v0.6.x of this lib, which does not have #66, so my Travis builds are failing.

[Tyriar]

@stevenvachon for Linux/macOS builds like https://travis-ci.org/stevenvachon/dotenv-prompt/jobs/253712264 you can copy this config which runs install fine: https://github.com/Tyriar/node-pty/blob/master/.travis.yml

Your build is failing because of nan's compiler requirements:

../node_modules/nan/nan.h:43:3: error: #error This version of node/NAN/v8 requires a C++11 compiler

Your windows build should be fixed by installing windows-build-tools

[daviwil]

We needed prebuilt node-pty for the atom-terminal-tab package so I took @coderaiser's advice and tried using prebuild to generate prebuilt packages. Turns out that it works really well! I created another npm module called node-pty-prebuilt which serves up these prebuilt binaries in a virtually transparent way (even with the winpty bits on Windows). Here's some more info:

https://github.com/daviwil/node-pty-prebuilt/blob/prebuild/README.md

@Tyriar (hey, by the way! 😄) if you're interested in eventually making this an official part of node-pty, I'd be happy to make a PR once we've seen it work well for a while. If not, I plan to keep the prebuilt package up to date with the latest node-pty releases so that they can both be used interchangeably.

[coderaiser]

I'm using node-pty with gritty and node-pty-prebuilt reduces installation time drastically on linux:

Install using node-pty:

+ [email protected]
added 96 packages in 40.985s

Install using node-pty-prebuilt:

+ [email protected]
added 1 package in 6.931s

Would be great if node-pty use prebuilts this way :).

[Tyriar]

@daviwil 👋. This is exactly the sort of thing I'm after, the Windows build in particular takes quite a while to build due to all the extra winpty code. Some questions:

1. How do the assets get uploaded to https://github.com/daviwil/node-pty-prebuilt/releases? Do you have to build on every machine type in order to upload the assets? Seems like a lot of work 😅
2. How do you get builds for all those different v8 versions?
3. When a new Electron release happens would we need to upload new assets to get it working?

[daviwil]

1. prebuild's --upload parameter takes care of it if you provide it with a GitHub API token that has write access to the repo. You'd just need to add a call prebuild on your CI builds for all 3 platforms to get everything prebuilt and uploaded automatically. Here's how I do it for AppVeyor and TravisCI

2. prebuild --all generates builds for all "supported" Node.js and Electron versions

3. Yep, you'd have to upload a new prebuilt package for the new Electron version. You can easily upload new prebuilt packages for existing node-pty versions but it might require one-off CI runs (or running it manually on your machine). prebuild --upload will upload to the tag associated with the package.json version and will only replace the prebuilt artifacts that aren't already attached to the release

Caveat, the Windows prebuild support needs a new update of prebuild to be shipped after this PR gets merged (should happen in the next day or two).

[Tyriar]

@daviwil and for people who want to use node-pty on a brand new version of node.js for example, is there a fallback if the prebuilt assets are not present?

[daviwil]

Yep, it's built into node-pty-prebuilt's install command:

"scripts": {
    "tsc": "tsc",
    "tslint": "tslint src/**/*.ts",
    "install": "prebuild-install || node scripts/install.js",
... 

[daviwil]

The prebuild PR got merged and 7.1.0 is released, here's what the Windows prebuild command looks like when including the winpty bits:

prebuild --all --include-regex "\.(node|exe|dll|pdb)$" -u <github_token>
prebuild --all -a ia32 --include-regex "\.(node|exe|dll|pdb)$" -u <github_token>

That builds x64 and ia32 builds of winpty and node-pty for all supported Node.js and Electron versions on Windows. Works the same way on Linux and macOS (x64 only) if you get rid of the --include-regex parameter.

[Tyriar]

What I'm hearing is there's very little stuff we need to do upon release and if the desired version is not there is will fall back to regular building I say we try push this in.

Would it make sense to run prebuild --all in prepublish and prebuild --upload in postpublish?

Also this would need to be improved to work on Windows prebuild-install || node scripts/install.js (I think || breaks in cmd).

[daviwil]

Here's what happens on Windows running npm install --verbose in a package that takes a dependency on node-pty-prebuilt when a prebuilt package can't be found for the platform:

...

> [email protected] install C:\dev\atom-terminal-tab\node_modules\node-pty-prebuilt
> prebuild-install || node scripts/install.js

prebuild-install info begin Prebuild-install version 2.4.1
prebuild-install info looking for local prebuild @ prebuilds\node-pty-prebuilt-v0.7.3-electron-v53-win32-x64.tar.gz
prebuild-install info looking for cached prebuild @ C:\Users\daviwil\.atom\.apm\_prebuilds\https-github.com-daviwil-node
-pty-prebuilt-releases-download-v0.7.3-node-pty-prebuilt-v0.7.3-electron-v53-win32-x64.tar.gz
prebuild-install http request GET https://github.com/daviwil/node-pty-prebuilt/releases/download/v0.7.3/node-pty-prebuil
t-v0.7.3-electron-v53-win32-x64.tar.gz
prebuild-install http 404 https://github.com/daviwil/node-pty-prebuilt/releases/download/v0.7.3/node-pty-prebuilt-v0.7.3
-electron-v53-win32-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=1.6.15 runtime=electron arch=x64 platform=win32)

C:\dev\atom-terminal-tab\node_modules\node-pty-prebuilt>if not defined npm_config_node_gyp (node "c:\Users\daviwil\AppDa
ta\Local\atom\app-1.23.3\resources\app\apm\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.j
s" rebuild )  else (node "c:\Users\daviwil\AppData\Local\atom\app-1.23.3\resources\app\apm\bin\\..\node_modules\node-gyp
\bin\node-gyp.js" rebuild )
Building the projects in this solution one at a time. To enable parallel build, please add the "/m" switch.
  Agent.cc
  AgentCreateDesktop.cc
  ConsoleFont.cc
  ConsoleInput.cc
  ConsoleInputReencoding.cc
...

After some testing, it looks like || does work on Windows. You can test this by creating a file called test.cmd with these contents:

echo Test!
exit /b 1

Then in cmd.exe run this:

cmd /C test.cmd || notepad

When the .cmd file returns a non-zero error code, Notepad will be launched. I only have a Windows 10 machine to test on at the moment so it might be worth testing that on a Win7 machine if you have one handy.

Regarding running prebuild inside of prepublish/postpublish, that might be cool but you'd only be able to do that on one platform (since you can only publish the release once). If you want to do this for all 3 platforms you'll probably need to do it as part of a CI script.

[daviwil]

By the way, if you want to give this a shot, I'm happy to send a PR :)

[Tyriar]

After some testing, it looks like || does work on Windows. You can test this by creating a file called test.cmd with these contents:

Does this depend on the shell that's running or does it work on all? I'm up for adding gulp if that's better.

If you want to do this for all 3 platforms you'll probably need to do it as part of a CI script.

@daviwil oh ok,

By the way, if you want to give this a shot, I'm happy to send a PR :)

👍, if you add this I can add you as a repo collaborator if you want to own the publishing of these assets?

[daviwil]

I don't think the shell that's running matters, I believe it shells out to cmd.exe always. powershell.exe is the default on Windows 10 and npm is still shelling out to cmd.exe as far as I can tell.

I'll definitely give this a shot this weekend! And I'm happy to maintain the prebuild/publishing aspect if you like, it's pretty hands-off after the initial work is done.

One thing I realized when thinking about this: node-pty uses a tag format that isn't (currently) supported by prebuild. prebuild expects that release tags will be prefixed with a v while node-pty omits the prefix. Since I wouldn't want you to have to change your tag naming scheme, I filed issue prebuild/prebuild#196 to ask the prebuild folks if they're interested in providing an option for this. If they're good with it, I'll send them a PR before I start migrating the prebuild work over to node-pty.

[jianyunt]

Hi @daviwil :) glad to see you here. I have somewhat related question. I may have to modify winpty.cc to let it take user cred. After modifying it, I need to compile it, drop it to right location etc. to run node-pty. Yes this is for the cloudshell stuff. How do you compile these cc files? I am on Win10 or WS2016. Thanks!

[Tyriar]

@jianyunt you should try upstream this to https://github.com/rprichard/winpty if the change is of sufficient quality. Also see that project for more details on compilation.

[jianyunt]

Sure thanks @Tyriar . BTW, I figured it out. The easy way to build winpty stuff on windows is here.

[Tyriar]

@jianyunt do you mean windows-build-tools on npm?

[jianyunt]

Yes. Here are 4 lines to build it on my local box.

npm install -g node-gyp
npm install --global --production windows-build-tools
cd c:\<myfolder>\node_modules\node-pty
node-gyp.cmd to rebuild

[Tyriar]

Copying over my update on this from #179

I've been thinking about this a bit recently and don't want to proceed for the following reasons:

• N-API should fix the majority of the issues with prebuilds, once Electron reaches node 10 we can start publishing single binaries per platform to GitHub releases or something with much more ease
• Builds will take much longer when targeting so many platforms
• I just moved the project over to the VSTS public projects preview which renders a bunch of this PR invalid
• prebuild and prebuild-install are non-dev dependencies (ew)

In the meantime I expect @daviwil's https://github.com/daviwil/node-pty-prebuilt to stay up to date with the very few node-pty releases that will go out. @amejia1 sorry about the hassle.

[Tyriar]

Closing, this is unlikely to happen but could be setup as a third party helper.

[lydell]

Now that node-pty has switched to Node-API, I published a version of node-pty with prebuilt binaries (that work with any Node.js version that supports Node-API):

npm package: https://www.npmjs.com/package/@lydell/node-pty
repo: https://github.com/lydell/node-pty

I took some liberties with my package though:

• Removed winpty. For my use cases there is no need to support old versions of Windows.
• I support a handful of platforms (macOS x86_64, macOS ARM64, Linux x86_64 (Linux ARM missing), Windows x86_64 and Windows ARM64), and my package doesn’t try to build with node-gyp on unsupported platforms (it just fails there). Support for more platforms is welcome if people help with how to build for them!

My package uses "optionalDependencies" to tell npm to only download the binary that is needed for the current platform. (That approach is also used by esbuild for example.)

Oh! I just noticed that @parcel/watcher (3.8 M weekly downloads) uses the same approach! Prebuild in GitHub Actions, publish via "optionalDependencies", no node-gyp fallback. Their builds seem to use cross compilation. Would be cool if node-pty could adopt this too. (Update: Managed to do cross compilation to some extent myself. For macOS and Windows it seems to Just Work™️, while for Linux it’s more complicated.)

[davidmurdoch]

I think prebuilt binaries are a must now that Spectre mitigations on Windows are required.