Switch from nw.js to Electron?

[techninja]

Over the past year and a bit our dear glorious foundry of free source control hosting Github created a code editor, Atom, intended to be a desktop application for all to use, while still using web technologies to render it! Sound familiar?

Anyways, they experimented with nw.js (node-webkit), tossed it, tried some other things, then made their own thing the called "Atom-shell", which eventually became the aptly named Electron.

It claims to come stock with:

• Automatic updates
• Crash reporting
• Windows installers
• Debugging & profiling
• Native menus & notifications

Which is a lot more than nw.js seems to go for, not to mention it's got backing from Github, and a number of platforms have already jumped on the boat (Spark, Microsoft, Atom (obviously), etc.).

Now it's not all sunshine and rainbows: It does work differently, and would likely require some code changes, but other than that It's likely 99% of the codebase could remain untouched, and might even get numerous speed boosts. What's good for Atom is good for github, and in turn, good for anyone on the shell platform. Intel is apparently the one funding the show over at NW.js, and suffice to say, though they've been gaining a version a month, I've not seen stability or speed do much more than waver.

So, perhaps I put it to a vote? Is it worth it to skip out on NW for big backers, better native support and automatic updates without a whole lot of extra work? This is the discussion thread.

[techninja]

FWIW: Found this in my travels regarding node-serialport & Electron -- https://gist.github.com/jedthehumanoid/a7f8278e0a37d259adca

[techninja]

Continuing on this front:

• Bad news: I've just spent 3 days trying to compile the latest node-serialport for the latest stable nw.js, with nearly absolute lack of success. It absolutely hates me, and I have a feeling it hates my friends on the node-serialport side too, considering this incredible practical & historical breakdown of the situation by @voodootikigod himself.
• Good news: On a whim (and with unstoppable frustration at nw.js), I got the latest 5 day old version of Electron, followed Jed's somewhat cryptic instructions above, and managed to basically swap out the expected headers for node/io.js to be the ones downloaded for Electron, and have successfully compiled (basically on the first try) node-serialport for Electron.
• It seems initially unhappy and breaking on some strange things, so I'm not out of the woods yet.
• Oh and the way the app wrapper works is completely different! Some small good news: I kinda looks like the app will have control over initial window size, so we could technically save window position and restore it on load, how nice!

[techninja]

Progress:

• Success! Sort of:
  • App starts and acts just the same, after a couple tweaks.
  • jQuery & i18next think we're in node in the window context. Their detection logic never seems to consider that we could have both a window object, AND a module.exports.
  • I made single word edits to the local copies of these libs, but I suspect I may have to move things over to a different inclusion style. This will definitely need some rewrites, but the good news is it might remove the require.js code and allow modes to become something far more akin to package-able modules(!).
• The feeling I get from electron is... generally more polished. The docs certainly feel like they've been thought out a bit more, and it seems there's a a vastly more comprehensive API built out. You can even put a progress bar on the task bar icon, cross OS!
• Generally seems to start and shut-down faster, it's no wonder nw.js is working like crazy to catch-up (and yet somehow failing).
• Switching process.nextTick out with setTimeout (which used to fail miserably and slowly), quite happily allows for very quick buffer filling with approximately same speed buffer running of images that used to completely break the app. Exciting stuff!

Proof:

This single stroke image will happily kill 0.9.5, AND the current draft of 0.9.6 in decidedly different ways, yet the Electron version after a day of work surpasses them both and prints it with gusto.

[docprofsky]

For us to switch from nw.js to Electron I think we need to:

• Get building of node-serialport working
• Make sure that switching to Electron does not break anything more than is currently broken
  Additionally I think before we release the Electron version of RoboPaint we should finish:
• Implement crash reporting
• Make a Windows installer

Using Electron looks very promising and from what @techninja has said it seems much easier to develop with and has more features and better developed features than nw.js. I think switching to Electron will let us implement some features that are harder to implement using nw.js and are better implemented than nw.js.

[techninja]

Remember, you can make multi-level bullets by adding more spaces in front of them.

Responses:

• As shown, serialport is building fine on linux (that tree and this fish were done from Electron RP on linux), and likely will build fine on Mac. Working on building on PC now.
• I am fully aware of the requirements for making Electron work for us, I mean, I did create the original build system and have rolled every single release (18!) by hand. I thought the issue was stagnant and needed a kick in the butt, not to mention nw.js somehow broke my linux dev environment, and a lot of what was needed was simply looking into what's going on and trying it. Turns out things are in pretty good shape.
  • Crash reporting: Define a "crash". Application level? JS code level? Our bug sending stuff will still work, and I'd imagine github already has something for sending crash data out as it's part of their API.
  • "Windows installer"? I'm guessing you mean Linux, we've had a windows installer since release 1, and it just saves the files out to a folder, it doesn't care what they are. We may of course move to githubs suggested method that includes automated updates? Thats kind of the work left on this ticket: Once the app work, figure out everything else that goes with making releases. There's TONS of great docs on it.

[techninja]

Current Status:

• When converting the nw.js gui controls over, found a "close/closed" bug, somewhat blocking conversion, though I'd imagine I should probably just move back to a previous version, though the fix has been merged, so could likely wait for the next release.
• Apparently my windows build environment is a fussy, inconsolable baby and completely refuses to listen while spouting crazy error codes that haven't been seen online since 2007. I don't know what else it's going to take, but once I get these binaries built I'm going to hoard them like a chest of gold coins until they're ripped from my hands like useless lottery tickets off by only one number.
• I think @docprofsky offered to try to get builds working for Mac... or he was joking? That little snert.

[techninja]

Update:

• Success! I now have v44 builds for win 32/x64, Linux x64 & Darwin (Mac) x64! Technically that's the biggest annoying hurdle from getting release builds made.
• Jed's build instructions from above are for Windows, but really the gist is that node-pre-gyp can't handle the electron version headers, even though node-gyp happily makes a headers folder for em, so you just trade out the stock io.js headers for the electron headers that were downloaded by just renaming the folder and re-running npm install serialport. This is very hackish, but works great on all systems so far. The bad news is you must remember to swap these headers BACK, or all your node-gyp builds are going to be for electron, and likely not usable on the console.
• Remaining work before this can be a contender:
  • Actually use the files native built files to get electron running on the various systems.
  • Ensure there are no clear faults for any supported modes and all specific nw.js API interfaces / pain points. Catching the close event is currently broken in 0.29.2, may move backward a release, though the second latest is 10 days old, so I have this feeling a new one will be around shortly.
  • Rewrite the build to work in the "new way", whatever that requires.
  • ...actually do a full build? I'd suspect in the interest of simply "moving" to Electron, we should not run after any big feature additions this can bring (unless it's low effort and unlikely to bring bugs like automatic updates might).

After this, I'm going to place my priority on working in my polygonal shape buffering to once and for all handle smooth buffering by handing the work to another processor via web workers. And then at the same time probably bugging @docprofsky and @rogerfachini to see if they're still alive...

[oskay]

w00t!

[techninja]

Building:

• Electron is quite different from nw.js as it's only a lightweight wrapper on chromium, instead of a complete rebuild, so if we wanted to change something about how it worked that there wasn't an API for, it wouldn't take 100gbs and a day of compiling. Neat!
• Actually making our output program, there's a pre-made packager that does the nitty gritty of actually putting the files together to make the mac app, exe, etc. It'll even sign the mac app given an identity! Sure beats the pants off my crappy little scripts in the build repo.
• I'm going to attempt to remake the build system to use this and grunt. Hopefully I can work my magic and not add in the "wine" requirement for adding the icon to the windows executable 😝

Menus & Extras:

• Instead of a defunct and half forced menu for mac, we can now to almost ANYTHING we want.
  • This is great... and this is awful. Means we get to make it how we wish, but it also means we likely need to do some kind of completely new work on it. Doesn't look all that bad honestly... Any suggestions will be appreciated (and implemented).
• Not in scope of this issue, but definitely worth mentioning for later, Electron's got a great set of OS specific APIs for desktop integration. Very tasty.

Installers (It gets complicated):

• Windows: They have a grunt based installer creator called grunt-electron-installer. Basically, you hand it a folder of the completed app, and it applies the icon, install rules and everything else in a nice neat package for you! Of course I already built that...
  • The Squirrel installer looks like a really great project with tons of support and great effort put in, but, it eschews most of the "standard" practice we're used to. No wizards, no user account control (everything is installed in userspace without asking). No logic can be given to the install, except from the app itself. Meaning that basically our app manages its own options to create/update/delete things like desktop icons, etc. Hrmm.
  • Try out installing atom on windows and you'll see what I mean. The installer opens up a little box in the center of the screen with an animation, and simply opens up the app once installed. It's neat, but it's totally not what we have, or what we're used to.
  • Honestly it's refreshing, and feels a bit like a clean reinvention of the mindless "next" clicking of the past, but I'm not sure we're quite ready for it. Thoughts?
  • Best thing here really is is that it's basically a wrapper with fully automatic management of install, updates, and uninstall. Leaving any/all customization beyond the norm to the app itself which is fine as it provides event triggers to update, and can even let the old app know that it's being updated.
• Mac: Somewhat related, I don't know how the heck we missed this: https://github.com/LinusU/node-appdmg
• Linux: Haven't found much, but I'm sure there's something out there as they provide both RPM (Redhat) and DEB (Debian/Ubuntu) install files for Atom.

[techninja]

WOOF

• Everything uses Grunt for task running these days, and I've been putting it off for far too long, so the big prebuild BASH script is now Grunt-ified! (Though it should be noted, I've cut some corners and not optimized it to run on ANY OS, we're still only building from a prepped Linux machine for now)
• I've also tied into the grunt workflow the Grunt electron-packager plugin, so now with a few bits of JSON, every file that's needed is placed and edited for every operating system. Wow!
  • This doesn't 100% replace the existing OS specific build scripts yet as it's only the first step, weaving our resources into the distributed file/folder structure.
  • I had to use wine for it, but I managed to hack the icon in after the main windows package build is complete.
• I found the linux DEB and RPM build grunt scripts used by Atom! So, that's totally doable now

What the heck is left??

• Have to build out the Squirrel update triggers. I've got a good chunk of code for this already, just need to whittle it down to what makes sense for the project.
  • I feel like I need to find the definitive squirrel update trigger server... business. Have yet to find comprehensive docs on how it all works together.
• Need to get the squirrel electron windows installer build working. It apparently only builds on Windows(!?), and my build system is toasted on my main Windows box (but works on my other). Maybe I can setup a virtual machine for this... hrmmm.
• Beyond that, it's just testing them on all the systems, then moving on to the program issues and doing a release(!). My deadline is to finish this week, as I start some paying work on the 27th. Good news is it involves dealing with some of the same issues :)

[techninja]

Monday morning notes:

• Squirrel update triggers done! Copied them from Atom, and pruned out the registry and PATH addition bits we don't need.
• Upgraded to Electron 0.30.0 (a single line edit in the build script), which just so happens to include a new app.getAppPath() api call which fixes the fatal relative path issue all over the app, also it doesn't seem to need a new native build, so that's a BIG plus. Built apps now work wonderfully on Linux and Windows. I still don't have a Mac to run on as Sylvia is using the only one in the house. 😢
• I've apparently fixed my build system on the Windows box, but it doesn't help when relative paths like these keep showing up: win_delay_load_hook.c : fatal error C1083: Cannot open source file: '..\..\..\..\..\..\..\..\..\..\..\..\..\Program Files\iojs\node_modules\npm\node_modules\node-gyp\src\win_delay_load_hook.c': No such file or directory
  • I plan on looking into this, though I may first try on the machine that built the native module properly first :/
• @oskay I'm going to need an answer on the Windows install front: Do we continue to use the old-school NSIS installer? Or make the move to the Electron standard Squirrel installer? The Squirrel installer does not offer the ability to ask the user about what to install like the driver, but we can force driver install within the application on demand (or even do some kind of detection of whether it's needed?). Who knows. The good news it'll be up to us and defined in the apps code.

[oskay]

I'm very happy either way on the installer front. Please go whichever way feels (easier + better) to you.

[techninja]

Cool. Will keep running with Squirrel then. If anything it tends to aim towards some very modern trends that learn from the last 20+ years of app installation that I believe have been sorely lacking in any improvement for quite some time.

Figured out the build issue (should have Googled it), fixed in the latest node-gyp 2.0.2 released 6 days ago. Gah, this bleeding edge stuff is crazy. Glad most of the rest of the community if getting on board! Will keep you posted as things keep moving 🍔

[oskay]

Awesome. 🐈

[techninja]

Sigh, blocked again on windows installer builds, but that's fine, still lots more to do :)
See electron-archive/grunt-electron-installer#39 for progress.

[techninja]

@oskay What should the app description be in the various app stores? This is the big flop of text on the app download page next to the screenshots.

Ubuntu Software Center is likely first step. I've got it under the "Graphics" category, with the tags "Arts" & "SVG"

[oskay]

I'm not sure that we're anywhere near ready for app stores yet...

[techninja]

DEB & RPM packages have the description as a minimum recommendation. This is what I have right now:

Software for drawing robots, and your friendly painting robot kit, the WaterColorBot!


See more about the WaterColorBot @ http://watercolorbot.com or
Fork & improve the project @ https://github.com/evil-mad/robopaint

[techninja]

I've mostly given up on the Linux package builds for now. Not only is it extra for us at this point, it's not setup cleanly for our current build structure, and very very little of the process is either documented, or easy to follow. I will push my work so far

Windows installer build update:

• I'm working hard to prove that it's possible to make it work at ALL. I've setup a brand new 32bit virtual win7 machine, and am running down the steps with surgical precision, so if it doesn't work, we'll have a very nice report for reproducing the problem for devs
• If not, then.. it'll just work and.. I'll somehow manage to get the build process to something I can be happy with. I'm forced to wonder if you could control a windows machine via SSH? Hrmm.

[techninja]

Umpteenth Update:

• Builds for all systems compile and run
• Installers:
  • Linux: The electron install builder is being a butt and seems to require a bunch of uncommented scripts that volley commands back and forth. I've basically given up until after the release because it's extra at this stage anyways.
  • Windows: Most of the work is done, and it's likely I've figured out what's blocking, but I just need to work through it a bit more.
  • Mac: Everything went better than expected! [Robo Paint Mac WIP Electron_v0.9.6.dmg [47.4MB] ](http://robopaint.tn42.com/RoboPaintMacWIPElectron v0.9.6.dmg)
    • After some initial stupid, and even though it has to build on Mac, it really goes quite smoothly and doesn't have any of the wacky side-effects my old build script had.
    • Another plus: It's so simple, @oskay should be able to run this build confidently by simply running grunt build-mac from the terminal, and everything else should be taken care of.
    • What I need for signing to happen during this process, is the identity that would be passed to codesign and it can happen automatically as well, see config options here.
• App testing:
  • Most of the issues I've seen are related to relative paths used in the app. After turning the ones I can fine absolute, it appears very little else is needed to make it work.
  • My testing is not done though, and there's a subtle change in the way windows and iframes are used that actually disables direct low level node access/usage inside of iframes (though you can access parent globals through window.parent, which kinda works. Good news is we don't use node very much in modes though...
  • ...I'm considering rethinking that a bit to get rid of the require.js dependency and a number of other weirdnesses around modes that would both open them up to node awesomeness, and make them be NPM fetchable. Allowing for modes that follow the rules the community has setup for program extensions already, fitting perfectly in line with what modern JS developers are already used to, and allowing for integration into an awesome, existing & free distribution system (I know @docprofsky certainly might be interested there).
    • This is neither here nor there, and would require some big rethinking, but could seriously kickstart mode abilities and that crazy ecosystem we keep musing on.

With a bit more testing (ignore printing performance for now), getting windows install builds sussed, I'm going to call this issue closed and will submit a PR for both repos!

[techninja]

More good news: I spent maybe 30-40 minutes with @rogerfachini and we got him completely setup for builds on his Linux machine, and tested said builds with the bot plugged in on his Windows machine! Very good stuff, and proof positive we've got a good thing going.

[oskay]

I can (presumably) generate and sign code on my computer-- I'm less sure about my ability to hand off signing to other people. :P

[techninja]

You wouldn't, obviously. We would simply either put the public identity in the file and let the local signature authorize it (the codesigning would fail for anyone other than you), or we set everything up with environment variables. It kind of looks like Atom does this like that... with some other bits thrown in...

You tried the build above yet?

[oskay]

No, haven't run it. It turns out to not be simple enough for me to understand. There are additional unstated prerequisites for this to work.

Edit: I've spent ~1.5 hours trying to figure out how to build this, without hint of success. This is not the best way for me to spend my time on the project. If you seriously need me to compile this, please provide instructions.

Edit: Make that 2 hours. And whoever wrote the grunt instructions should be ashamed of themselves.

[oskay]

Hour number 3: success.

[techninja]

@oskay Oh dear, no no no. Crossed wires. I meant, I built the application for you to try ➡️ [Robo Paint Mac WIP Electron_v0.9.6.dmg [47.4MB] ](http://robopaint.tn42.com/RoboPaintMacWIPElectron v0.9.6.dmg)

Will follow up with clearer actual build instructions.

[techninja]

I admire your dedication, but you likely should have scoffed at my lack of documentation to make the build and walked away, or asked for clarity. :P

I'm not sure how required any of these things are, but this ensure a clean and proper build as far as I can tell (for Mac):

• Remove node.js. We'll be switching to its cooler, younger brother io.js as Electron has done.
• Install io.js. https://iojs.org
• Terminal, install grunt-cli, and node-gyp globally via: npm install -g grunt-cli node-gyp (shouldn't require sudo as previous versions did, saves in user space)
• Assuming the npm bin dir is in your path (it should be), you should now be able to do which grunt and it'll tell you.
• Clone and checkout the electron branch of the robopaint-build repo. git clone https://github.com/evil-mad/robopaint-build.git, then cd robopaint-build, git checkout electron
• Go to the root of the repo, and install its dependencies via npm install
• Run grunt build-mac in that same place
• Assuming everything is green, you should have the application in out/Robopaint-darwin-x64, and a (almost) final dmg in out/RoboPaintMac_v0.9.6.dmg

Pretty straightforward, I think. I mean, you did eventually figure it out? :P

[oskay]

I did figure it out eventually. However, using node not io.js.

Really not working now, after switching over, trying those instructions. npm install is throwing buckets of errors at me.

[techninja]

Feh. Removing node may NOT have been clean, I have yet to do that on a mac. Paste in some of your errors, should give me a clue as to what's up.

[oskay]

$ npm install

> [email protected] install /Users/oskay/Documents/DIY/Project_WaterColorBot/RoboPaint_Release/electron/robopaint-build/node_modules/grunt-appdmg/node_modules/appdmg/node_modules/fs-xattr
> node-gyp rebuild

gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: 404 response downloading https://nodejs.org/dist/v2.4.0/node-v2.4.0.tar.gz
gyp ERR! stack     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/install.js:251:14)
[...]

I've rolled back to node, and it's working again.

[techninja]

The 2.4.0 version is io.js, not node, which is why that's failing. My guess is it's not easy to uninstall node on mac. I guess running it on node is fine, though I've been told it's "not recommended", I suppose it doesn't matter unless you're upgrading the native modules :P

So!

• How do you like the DMG install? Quickload adjustment to show previews?
• How's general performance in startup/mode switching?
• See any obvious bugs?
• Can you sign the app after it's been built with this without having to mess with the DMG?
  • Or do we need to adjust the mac build grunt task to do that between? Note that the build-mac grunt task really just runs these three tasks in order pre-build (gets the files), electron:macbuild (packages the app) & appdmg (puts it all in the DMG with icon and background and such), so if you need to you can do you own thing in between.

[oskay]

So far as I can tell, I did manage to uninstall and unlink node, and then install and link io.js, and started with a fresh copy of the repository, too. Still, something is very screwy.

[oskay]

DMG looks nice. It launches OK, but I haven't had a chance to try much-- too busy wrestling with the other stuff. I may get a chance later tonight.

[techninja]

Hmm...
http://stackoverflow.com/questions/11177954/how-do-i-completely-uninstall-node-js-and-reinstall-from-beginning-mac-os-x

I appear to have figured out what needs to happen to make the Squirrel Windows installer actually compile. It involves some very crazy rejiggering of how node works, but it's automated and after the build, so I'm not too concerned. Will update their issue queue and move on to the final phase of this ticket: Testing for electron specific bugs!

[techninja]

It should be noted that the current electron app for Mac is currently programmed to not close the application when the window is closed, "to better act like every other mac application", but I'm beginning to feel that since this is a single window app, this doesn't quite make sense.

@oskay do you think we need something better than just a quit menu item for mac for next release? or should we just avoid the extra work?

[oskay]

A "first class" Mac app should only use the mac menu bar for its functions, never internal menu bars.

That said, it would be good to add:

• File menu: About RoboPaint (at least give version number)
• File menu: New window (which opens up the window again, if somebody closed it)
• Help menu: Online help (opens up web site location?)

Launch and context-switching are both MUCH faster in the new version.

I do not know what stage signing has to take place at. In previous builds, I've signed before packaging in the DMG.

[techninja]

I never said anything about internal menu bars, I was referencing my earlier comment about Electron's over the top menu API and it's capability to handle basically anything we need... They can even be dynamic to the given mode. I'm sure that's a feature issue of its own of course.

I'll add the minimum menus given with default shortcuts and we should be in good shape. Bonus for this build system: as I update the electron branch, you can build locally and test any code branch as long as it's pushed :)

[techninja]

@oskay Forgot to ask... do we want to do Windows code and installer signing? Big benefits so says the Atom team. Might cost money. >> https://msdn.microsoft.com/en-us/library/windows/hardware/hh801887.aspx

EDIT: In addition, new to Electron, we have 32 and 64 bit builds, meaning that my Windows builds are by default 64bit. Do we want to offer both? All new Windows machines are shipped as 64bit, though that's likely only in the last 5yrs or so. Good news it should be relatively easy to extract the native build file, and the rest is another few lines of grunt config :)

[oskay]

I'm not sure that I understand the different "dashboard service" options mean, but I'm pretty sure we're not doing LSA or UEFI, which means that a standard certificate about $90/year would work. We could do that.

From what I can tell, even windows 10 (yet to be released!) supports 32-bit. We... probably should as well. :(

[techninja]

Just rolled the native builds for 32bit (thx VirtualBox!), and integrated it into the build scripts, and successfully used the installer and ran the app from the 32bit VM! Apart from code signing, looks like it's small bug mop-up from here on out.

[techninja]

Electron bugs squashed:

• "Close catch" bug
• Save dialog API for edit mode (Saving files broken in electron builds #220)
• Minimum resize for Linux (fixed in Electron 0.30.1 released 3 days ago)
• Confirm dialog bug

Everything else that's wrong appears to simply be things that were already wrong. So that's.. good. Fills are "broken" as they use the "nextTick()" solution, that very successfully BLOCKS anything else from working while they're filling the buffer. This will be fixed first thing after Electron is done.

Automatic Updates:

• Code exists for what's needed for me to run, but because Github went with an existing system instead of building it, it doesn't rely on native Github releases to pull updates from. Poo.
• Good news is, the releases data is very easy to get to in a machine readable format: http://api.github.com/repos/evil-mad/robopaint/releases/latest
  • Bonus: With this you could have a piece of PHP or even embeddable JS that pulls and formats the latest release downloads into some pretty buttons for displaying on watercolorbot.com, could even autodetect the users OS and display accordingly. @oskay feel free to make a ticket for this and I could probably knock it out in a day.
• Considering the amount of time I have (and lack of comprehensive documentation available for the full update workflow), I'm going to Pass on auto updates for know, knowing that implementing them should be a walk in the park when it comes time.

With that, I'm going to submit the PRs, and unless anything else comes up, will merge these into master (with updated contrib readmes) sometime in the next few days! WOot.

[oskay]

Awesome-- I've scheduled some time for basic tests on Monday morning.

[oskay]

Bug in 0.9.6 (as built by me two days ago): The "park" block (in Scratch) does not seem to function.
It does work in 0.9.5, and the Park function within RoboPaint does work.

Snap shows an error when you try to run the park function (screenshot attached).

[techninja]

Wacky. 🍍
Will take a look at it after Pancake work tonight.

[techninja]

Yea, totally missed that one! I haven't written automated tests for use as a node module (as RoboPaint does) yet, it slightly changes the interaction "surface" to work better/faster, but obviously still a few bits to be tested. this should be fixed now, just rebuild and it should grab a brand new copy of cncserver and park will work.

[techninja]

Alrighty, this seems done with. We're merged and pretty happy, now on to the real problems :)