Graduate iojs out of keg-only

[bcomnes]

This is an alternative to #36343 that brings the following qualities:

This PR:

• Graduates iojs out of keg-only
• Conflicts iojs and node (this seems to be what the iojs team wants)
• installs npm from the tarball every time node or iojs is installed.
  • No patching is performed. Don't merge until compatibility is reached.

Issues that still need resolution

• Move npm install code to Library/Homebrew/language/javascript.rb and share between node and iojs formula. (Anyone feel free to hop in on this one! I don't have time right now)
• Wait for node-gyp patch to land

Why this arrangement works well

• users can either brew install node OR brew install iojs and everything "just works".
• the npm install is basically identical between the two (except for the current patch).
• nuking the npm install during updates isn't a big deal.
• npm can still update itself.
• npm updating itself wont break patch

Wackiness

• Massive DRY violation. Can the npm install code be separated out somehow?
• Patching resources is a tad funky. This would be cleaned up by Patching external resources #31508 (comment)
• Patching one formula but not the other breaks the linking/unlinking dance with both formula installed.
• npm installs on all node/iojs installs+updates and nukes whatever is in /usr/local/lib/node_modules/npm

Current iojs formula discussions

• Adds flag to patch npm install for iojs compatibility  #36357 Adds flag to patch npm install for iojs compatibility
• Node, iojs and npm formula resolution #36343 Node, iojs and npm formula resolution
• Graduate iojs out of keg-only #36369 Graduate iojs out of keg-only

working iojs taps

• https://github.com/aredridel/homebrew-iojs

Past iojs formula discussions

• add iojs #35853
• iojs v1.0.3 (new formula) #36039
• Adds iojs formula meeting requests by @mikemcquaid #36193

npm formula issues:

• npm #3762
• Homebrew installed npm can't upgrade itself #22408
• Homebrew should no longer manage npm. #27479
• node: install npm using tarball. #28075

[MikeMcQuaid]

Total dry violation. Any way to share this out of band npm install code between iojs and node?

Yeh, we could probably split this somewhere into a Library/Homebrew/language/javascript.rb file and then have various functions that are called from both formulae.

Patching one formula but not the other breaks the linking/unlinking dance with both formula installed.

Yeh, this is making me feel like we should wait until the patch is upstreamed before we include this.

Another concern with this is that we have node and iojs fighting over the /usr/local npm. This may not be a problem in reality but it'd be great to try and test installing e.g. node then iojs, iojs then node, with and without existing npm installs to make sure stuff doesn't blow up.

Thanks for the really comprehensive and considerate work here. It's much appreciated and you're massively accelerating the rate at which iojs will be a "normal" formula so well done.

[bcomnes]

Ok removing patches from the de-keged PRs (this one and, #36343) as they won't be needed for much longer.

In the mean time, can we include the patch here #36357 now that its been put behind a flag? This would make the existing iojs formula a drastically more useful in its current state while we wait for nodejs/node-gyp#564 to land and can go ahead with, most likely, this PR.

I'll start on Library/Homebrew/language/javascript.rb sharing npm install code between the two formula.

npm is quick to install/update and hard to break, so the chance of rolling back or being pushed forward during an parallel install or formula update isn't that big of a deal IME. Also the current node formula has this same behavior.

We could add the caveat to give a heads up about this:

Installing or updating {node,iojs} installs npm at its latest version.  Global npm modules and binaries are preserved.

Thank you for your feedback and support. ❤️ homebrew & node+iojs so its been a fun learning experience for me

[bcomnes]

I took a rough pass at extracting most of the npm install code, but I did't put a lot of thought into it. Feedback welcome.

[othiym23]

There are two ways to bring in npm:

1. Install the version that was released with a given version of io.js.
2. Have the formula track npm@latest, maybe with some automation.

I can't remember where @DomT4 and I last left things when we were discussing this, but it would be helpful to have some support from the formula for whichever approach you adopt. The reasons for choosing one versus the other have mostly to do with where you expect issues to be upstreamed, so as the person in charge of the npm issue tracker, I vote for 2. I'm happy to contribute work towards making this happen and will point out that there is a releases feed that could help with some of this automation.

[bcomnes]

@othiym23 is there a url we can pull in from and always get the latest version? IE https://registry.npmjs.org/npm/-/npm-latest.tgz? Or we can automate a npm install -g npm@latest after we install npm from a version we know works with the existing formula and do occasional version bumps to ensure that step keeps working.

[DomT4]

I can't remember where @DomT4 and I last left things when we were discussing this

At the moment we're serving up the latest stable npm tarball in the Node formula every time Node is released, and I manually push a PR to update it and enforce the update onto users via a formula revision if there's a major release, if we're falling too far behind, if you squash race conditions, etc etc.

Re iojs, our prior discussion was left open-ended, but we were discussing sticking it in devel-only at the time, and obviously it ended up in Homebrew/Homebrew so the maintainers here probably have to be a little more careful not to break everything than I could have gotten away with.

there a url we can pull in from and always get the latest version? IE https://registry.npmjs.org/npm/-/npm-latest.tgz?

Homebrew generally doesn't do this because it doesn't allow HB to checksum the tarball, and it doesn't allow HB to know with any certainty that x version of npm isn't causing any problems with Homebrew's x version of x. It would also rule out our ability to bottle npm. Up to the maintainers here though what we do, these are just my thoughts.

we can automate a npm install -g npm@latest

We did this before and we scrapped it. It was semi-problematic and unreliable 😢.

[bcomnes]

Darn.. ok.

I guess keeping on top of the distributed npm version, and really clear caveats seems to be a decent solution.

[othiym23]

It was semi-problematic and unreliable 😢.

I don't remember what the issues were, but if this is true (and the issues can't be worked around), then my vote switches to coupling the version of npm to match what's distributed with the corresponding version of io.js. So far, io.js has a much steadier release cadence than Joyent Node, and this means that we're less likely to see the issue where the bundled npm is significantly out of date compared to the latest release from the npm team. What I really don't want to happen is for Homebrew to in effect create a new distribution of Node, where the people who support npm have to keep an eye on which version of npm Homebrew is currently bundling. Both Homebrew and npm lose in a situation where we have to keep a close eye on what each other are doing.

[bcomnes]

I don't think a close eye is that critical. Just as long as the distributed version is pretty close, and worst case scenario, as close as it needs to successfully update to latest. Now that I have a better idea of how npm is handled in the formula, I'm willing to help keep it as close as possible along with everyone else who has. I think the following improvements to the caveats would help too:

        npm has been installed. To update run
          npm install -g npm@latest

        You can install global npm packages with
          npm install -g <package>

        They will install into the global node_modiles directory
          /usr/local/lib/node_modules

        Do NOT use the npm update command with global modules.
        The upstream-recommended way to update global modules is:
          npm install -g <package>@latest

[othiym23]

I don't think a close eye is that critical.

I disagree very strongly. Remember that npm has a firehose of incoming issues, and that end users are maybe not the best at getting all the relevant details together before filing issues (I am being extremely generous and polite here). If a Homebrew formula for Node or io.js gets out of sync in some kind of breaking way, it is absolutely material information to the people triaging incoming npm issues that the users behind a burst of incoming issues are all running Homebrew. Those people then have to come to the Homebrew issue tracker or look at the git history of the formula and try to figure out what happened. This is not a theoretical concern, as I've had to do something along those lines several times in the year that I've been working on npm full-time (see also: the npm issues you link to in the OP).

This is why I'm advocating having a definite solution one way or the other: either keep the npm version in lockstep with what's bundled with io.js (preferably using the npm that's bundled in the io.js tarball, although I understand why the current formula doesn't do that), or installing / upgrading to npm@latest at the time of brew install iojs / brew upgrade. Letting the two versions vary, if installing io.js is to install npm, is very close to unacceptable to me.

I think the following improvements to the caveats would help too

IMO putting stuff in caveats is next to useless. End users pay some attention to them, maybe (if you run brew upgrade infrequently, the amount of output from brew pretty much guarantees that only especially conscientious users are ever going to see new or changed caveats), but they aren't seeing the caveats when they are having problems with npm.

[bcomnes]

Personally, I'm +1 on automatically running npm install -g npm@latest on install and upgrades. Unlike pip or gem updates, updating npm is as common and painless as running brew update IME. In fact, I would't mind if npm just did it for me every week when I use normally ;).

[DomT4]

Darn.. ok.

FWIW, My word on this isn't anywhere near being final. The maintainers are free to override me as frequently as they like 😸. I feel personally like it negates the bottled npm in certain ways, but I very much doubt that's a show-stopper.

So far, io.js has a much steadier release cadence than Joyent Node, and this means that we're less likely to see the issue where the bundled npm is significantly out of date compared to the latest release from the npm team.

This works for me at least. I'm happier matching iojs's npm release schedule than I have been Node's. Node's reluctance to upgrade npm is not insignificant part of the reason we had so many npm problems in Homebrew.

I'll probably duck out of this issue for a while. I'm trying to avoid the whole Node/iojs situation as best I can for the sake of not annoying everyone here. Feel free to tag me back in if there's any specific issues I can be useful on.

[bcomnes]

@DomT4 Any advice on the Library/Homebrew/language/javascript.rb approach?

[MikeMcQuaid]

Personally, I'm +1 on automatically running npm install -g npm@latest on install and upgrades. Unlike pip or gem updates, updating npm is as common and painless as running brew update IME. In fact, I would't mind if npm just did it for me every week when I use normally ;).

Not in our experience. It broke for end-users relatively often. It's not reliable enough for me to be comfortable doing this for a while (maybe ever because it's not in keeping with what we do with e.g. pip).

[MikeMcQuaid]

Just to repeat myself from #36357 (comment): adding this patch to npm is a non-starter until it's merged upstream (and, ideally, released in a new stable version of npm).

[DomT4]

@DomT4 Any advice on the Library/Homebrew/language/javascript.rb approach?

I'd like to get some more upstream confirmation on what the relationship between Node and iojs is. There seems to be a lot of talk about potentially merging back together in the future, and I'm kinda worried about taking big steps only for Brew to have to undo them in 6 months or a year's time for x,y or z reason.

But I don't think that's reason to stop pondering if we can't handle npm and the Node ecosystem a little better or a little differently. The current way we handle the ecosystem is relatively stable, and it took a long time to get it to that point, but I'm not overly attached to it if we can retain or improve on the stability, and preferably, keep npm bottled. The Javascript.rb approach looks plausible, and could be relatively stable once the kinks are ironed out. I think it's an interesting route to explore.

Have you had the chance to put it through local testing yet?

[MikeMcQuaid]

I'd like to get some more upstream confirmation on what the relationship between Node and iojs is. There seems to be a lot of talk about potentially merging back together in the future, and I'm kinda worried about taking big steps only for Brew to have to undo them in 6 months or a year's time for x,y or z reason.

Personally I'm not super concerned about this.

The current way we handle the ecosystem is relatively stable, and it took a long time to get it to that point, but I'm not overly attached to it if we can retain or improve on the stability, and preferably, keep npm bottled.

Yes, this is what makes me lean towards not making changes unless we have to.

[DomT4]

Personally I'm not super concerned about this.

That's cool. It's just a personal kink, but it's not something I'm overly grumbily about, and at the end of the day, it's something you guys can override me on. No challenge from me against what you or the other maintainers feel best here. Will try to productively contribute where I can, or where I'm asked.

[bcomnes]

Not in our experience. It broke for end-users relatively often. It's not reliable enough for me to be comfortable doing this for a while (maybe ever because it's not in keeping with what we do with e.g. pip).

That was my understanding. Currently this PR does not run an update on npm. I'll try to dig up some old tickets to identify what was going wrong for my own understanding.

Does anyone happen to know more specifically what kinds of issues were going on? The state of npm has improved massively and rapidly over the last couple years. I bet it might be worth revisiting historic issues and compare them to the situation today.

Just to repeat myself from #36357 (comment): adding this patch to npm is a non-starter until it's merged upstream (and, ideally, released in a new stable version of npm).

Understand. Closed it. If people really want iojs with necessary compatibility patches from homebrew today, there are working taps available.

I'd like to get some more upstream confirmation on what the relationship between Node and iojs is.

Here are some key articles if interested:

• http://blog.izs.me/post/104685388058/io-js
• https://gist.github.com/maxogden/d96123138522c84cdb25
• https://medium.com/node-js-javascript/state-of-io-js-2b3094e6f923
• https://www.joyent.com/blog/node-js-advisory-board
• http://nodejs.org/about/advisory-board/

[bcomnes]

Rebased to capture #36518 changes. (or should that have been a merge?)

[MikeMcQuaid]

@bcomnes Rebase is good. What's the npm patch situation?

[bcomnes]

Still in the works.

[bcomnes]

Rebased and added one more additional npm test to test a small native extension install.

[DomT4]

Thanks for the rebase! I like how this is going, It's looking good. Any news from upstream on the npm patch situation yet?

[othiym23]

Still waiting on node-gyp to land whatever changes it is going to land for iojs support. You can rest assured that I will land that patch in npm as soon as it is available, and no sooner. ;)

[bcomnes]

Version bump

[DomT4]

@bcomnes Can you go into a little detail about how today's confirmation of the merge is going to change things, if at all? The convergence document seems to say that Node will continue, iojs will continue, but a third Node branch of merged Node/iojs origin will be created?

Are we looking at a situation where the iojs formula goes away in favour of the Node formula (or vice versa), or a third Node-related formula being created for the converged branch entirely, or something else?

Sorry, It's late and my head is 😫. Just trying to grab some idea as to what the end result for Homebrew is here and what, if anything, we need to do to prepare for it.

[bcomnes]

I'll have to catch up myself! TY for the heads up.

[bcomnes]

Notes:
https://twitter.com/rvagg/status/598605393636429825
https://twitter.com/mikeal/status/598595967928008705
https://github.com/jasnell/node.js-convergence
https://www.youtube.com/watch?v=UbYiFLf7MpU - Discussion starts at ~27:25 audio
nodejs/node#1664 (comment)

[DomT4]

Thanks. The amount of documentation covering everything is immense. It looks like node and iojs will be sticking around for a while even as the grand merge is occurring, which makes sense.

I guess the question is, and probably needs @MikeMcQuaid's input, if a node.js (the convergence branch) cuts a new release before node or iojs are formally deprecated, do we create a new formula for node.js and then slowly remove the node and iojs formulae, do we maintain all three long-term, do we merge the new convergence branch into either the node or iojs formula.

It looks like the node name is going to be sticking around indefinitely as the LTS version, based from this tweet so presumably we track the existing Node formula onto that line eventually. From the reading I've done I'd presume we want to do something like this long-term:

• Node LTS => Homebrew/versions
• iojs => Homebrew/versions
• New Node-iojs release => New formula in Homebrew.

But the last one could be tricky. They seem fond of the node.js name for the convergence branch, and we already have aliases set up to the existing Node formula for nodejs and node.js.

I'll read through some more of the documentation progressively. I'm already git-tracking all three branches. None of this is super-urgent, it looks like iojs and node are going to keep being a thing for a while - I'm just trying to be anal about forward-thinking 😸.

[bcomnes]

@DomT4++ I'll have to look into it more tomorrow AM most likely. Bet time for me.

Giving you collaborator access to this branch/PR in the event it becomes useful but needs some changes before merging and I'm not around.

[DomT4]

Thanks. No worries and no rush. I'm just being overly-cautious about not being caught out on a major Node ecosystem shift again, the iojs/Homebrew situation has been less smooth than pretty much anyone hoped I suspect, heh. Thanks for the quick replies here tonight, appreciated as ever.

[MikeMcQuaid]

I'd say we go for node meaning the node-iojs release and we go for adding a devel block there whenever they have a beta and update that package whenever we have something stable.

Got to LOL about this whole thing, really, given the npm patch never actually got merged but the projects themselves are merging instead 😆

[bcomnes]

Node gyp was updated in npm! Not sure what it all means yet,

https://github.com/npm/npm/releases/tag/v2.11.1

I'll post what I find out, but hopefully the patch issue was addressed.

UPDATE: Nope, header files and floating patch still an issue.

[rstacruz]

Patching is no longer an issue as of npm 2.6.1 / iojs 1.5.0.

nodejs/node#990
aredridel/homebrew-iojs#21

[MikeMcQuaid]

Cool! Happy for this to be no longer keg-only then.

[othiym23]

I'm not sure why people think the landing of nodejs/node#990 implies that the npm team is not still landing floating patches with each new version of npm when it's merged into io.js, but that isn't the case. Unless and until the changes related to process.release are landed and a new version of node-gyp is published that takes advantage of them, the npm maintainer team continues to have to cherry-pick io.js commits, as per https://github.com/npm/npm/wiki/Release-Process#for-iojs. An unpatched npm installed outside of the io.js installer will still not be able to find its patches, and the use of either a patched npm or pangyp is still required to build native dependencies unless you're using the npm distributed with io.js. Sorry!

[hmalphettes]

@othiym23 Thanks for the clarification. Sorry I got carried away and started the confusion.

[bcomnes]

Still waiting on floating patches. Taps and pangyp (paired with brew unlink node ; brew link iojs --force) available in the meantime. Both are great options, and I use them both.

[bcomnes]

Just to throw an update in here from what I can gather:

Iojs and node are merging.

The next version of node beyond 0.12 will be called node, but will be some some version of iojs.
iojs will remain around in full capacity after this merge release.
By this point there will likely be an update on the floating patch situation.

What this next node release will look like is yet to be seen.

I speculate node will be a stable branch and iojs will be more cutting edge. We shall see!

EDIT: Yes, this appears about right: https://medium.com/@nodesource/essential-steps-long-term-support-for-node-js-8ecf7514dbd

[bcomnes]

A draft schedule is available for anyone interested:

[hmalphettes]

nodejs and iojs have merged into v4.0.0
I guess we can close this one; should we even consider removing iojs all together from Homebrew's Formulas?

Here is a PR for upgrading node to v4.0.0: #43725
Let me know if something is missing.

To try it out you can use https://github.com/hmalphettes/homebrew-node:

brew remove --force node
brew remove --force iojs
brew install hmalphettes/node/node

[bcomnes]

I will close this PR for the time being. Nice work everyone. Happy to see thing start to simplify again.

[DomT4]

Thank you for all your work on this @bcomnes! Really appreciate you working with us patiently and helping clarify some of the Node/Iojs development process ❤️