PPC platforms as part of standard release

[mhdawson]

I'd like to list the things that need to get done so that we can get to the point where the PPC platforms are part of the standard release.

What I have on the list so far is

1. Release machines. Need an additional LE and BE machine to be used for releases only
2. Additional test machines. Need to have 2 LE and 2 BE machines, currently there is only 1 of each
3. Need to add testing on PPC to node-test-pull-request
4. Tests passing (they all do now except for test-tick-processor (test-tick-processor failures on PPC node#2957)

Can people chime in on what else I'll need to add to this list.

[rvagg]

sounds reasonable to me, what strings would we use in binary names for these? ppcbe and ppcle? what is standard practice for these?

[bnoordhuis]

ppc, ppc64, ppcel and ppc64el?

Does little-endian 32 bits PowerPC exist? I don't think I've ever seen it.

[mhdawson]

There is no 32bit LE version only 64 bit. For both BE and LE initially we'd only be looking to include 64 bit. 32 bit does not exist for LE and on BE it is lower priority due to the level of use.

I'd suggest we use these names ppc64 and ppcle64

[bnoordhuis]

Interesting how I made the same typo twice, I intended to write 'ppc64le'.... Anyone have data on what is more common, ppcle64 or ppc64le? I've seen both used.

[jbergstroem]

Wikipedia and some linux dists seems to point to ppc64le.

[mhdawson]

Talked to our lead from the power side and it should be ppc64le

[mhdawson]

Added 2 additional machines to the CI, one for BE and one for LE so we have the machines needed on the regression test side. They are already up and running in the CI, last step is to just add to the ansible inventory - PR here - mhdawson@e6e3747

Will need to get 2 more for releases

[mhdawson]

I now have 2 additional machines for releases. What is the process of getting them configured/added to wherever they need to go ? @rvagg Johan suggested you'd probably be the right person to answer that

[rvagg]

first step is getting them into nightly builds, I'm going to work on that this week and will ping you when I'm onto it

[mhdawson]

Is getting them into the nighties something I could help out with ?

[rvagg]

@mhdawson: first thing I'm noticing is that they're taking a very long time to clone the repo, could you put a bare mirror on them to speed it up please?

mkdir /home/iojs/git/
git clone https://github.com/nodejs/node.git --mirror /home/iojs/git/io.js.reference

the job is set up to use refs found in that location when cloning so it ends up only getting any newer commits for each build.

Next, they are missing $ARCH and $DESTCPU environment variables:

https://ci.nodejs.org/job/iojs+release/275/nodes=ppcle-ubuntu1404-release-64/console

ARCH is mainly used in determining the filename (i.e. we have x86, x64, armv6l, etc.) while DESTCPU is what configure cares about, often they are the same.

This can be done in the startup configuration for the service somewhere (@jbergstroem has stronger opinions on what the correct way to do this is than me), or even in the slave config in Jenkins were you set custom environment variables.

I see we allow ppc64 and ppc in both configure and Makefile but we have both ppcbe and ppcle machines and both are 64-bit, what is the intention here? What binaries are you wanting to produce for download and what do you think their names should be?

[mhdawson]

We only have 64 bit machines and based on the conversation that was started when I investigated getting a second set for 32 bit to be consistent with other architectures the feedback from the Power team was that 64 bit is what matters (it does not exist at all for LE). Based on that I was looking to have only the 64 bit BE and 64 LE platforms as part of the standard releases.

The names should be ppc64le and ppc64

[mhdawson]

Looking at the download links as an example I was thinking along these lines:

https://nodejs.org/dist/v4.2.2/node-v4.2.2-linux-ppc64.tar.gz
https://nodejs.org/dist/v4.2.2/node-v4.2.2-linux-ppc64le.tar.gz

Is that what you were looking for in terms of the naming. Next I'll have to see how that works out in terms of what we want for $DESTCPU and $ARCH

[mhdawson]

I think what we want is:

BE:
DESTCPU = ppc64
ARCH = ppc64

LE:
DESTCPU=ppc64
ARCH = ppc64le

I see at least ARCH is set as environment variables for Windows in jenkins so I'll so ahead and do that and we can update if Johan suggests a better way

I'll add those

[mhdawson]

Ok environment variables added on jenkins and added the bare mirror to each machine using the commands listed by Rodd run as the iojs user.

@rvagg I think that's everything you asked for

[jbergstroem]

Reckon these guys should go in the init script.

[mhdawson]

@jbergstroem is there an example of a machine where they are in the init script that I can take a look at ?

[jbergstroem]

these machines failed to build as part of 5.1:
https://ci.nodejs.org/job/iojs+release/287/nodes=ppcbe-fedora20-release-64/
https://ci.nodejs.org/job/iojs+release/287/nodes=ppcle-ubuntu1404-release-64/

Going to pull them off the release job to not delay our release any further.

@mhdawson regarding init scripts:

• fedora20: https://github.com/nodejs/build/blob/master/setup/fedora20/resources/jenkins.service
• ubuntu1404 still lacks one, so gun for start.sh.

[rvagg]

sorry @Fishrock123, they weren't meant to be included in release builds, only nightlies at this stage, we still have work to do on getting them ready!

[rvagg]

@mhdawson I'm told that you're not happy with the switch we made to Ubuntu 14.04 for ppcbe and that there's a technical reason this is a problem? Can you explain in here for me please?

I have a big concern with using a non-LTS-style distro for release builds because it means we don't get a stable platform over time. Consider how we've been building Node.js for Linux on EL5 for years and we get the same libc as a benefit. With Fedora, they are strict on deprecation so we have to upgrade every year and have a changing libc and have to scramble each time they have a new release in order to make sure that we can release any binaries (ppc holdups will be holdups for everyone once this becomes part of the official flow).

So if there's a way we can use EL (RHEL is fine by me fwiw, doesn't have to be CentOS), Debian or Ubuntu LTS then that would be ideal.

[rvagg]

@jbergstroem https://ci.nodejs.org/job/iojs+release/350/nodes=ppcbe-fedora20-release-64/console

Bad owner or permissions on /home/iojs/.ssh/config

Is this from a script now or was that manual?

FYI everyone else, these machines are active for nightly builds https://nodejs.org/download/nightly/ but we're still ironing out wrinkles so they are not reliably showing up in each new nightly yet.

[mhdawson]

The reason I chose Fedora 20 was because it was the earliest OS for BE that was supported by osuosl (they don't have RHEL available). If we use ubuntu 14.04 instead then it means that the binaries will only support the later OS levels. The issues we have seen are usually related to the default libc/libstdc++. Having said that, some of these issues were seen before the move of v8 up to the 4.8.X level of the compiler forced our hand in terms of default support for some platforms so the reasonable levels to support have moved up a bit.

At this point my main concern is if the whether binaries built on ubuntu 14.04 will run on RHEL 7. Given your concern I suggest we create an ubuntu 14.04 release machine. I can then test whether the binaries run on our internal RHEL 7 machines.

[jbergstroem]

@rvagg inconsistencies when I manually set it up. Fixed it a while ago.

[rvagg]

@mhdawson Fedora 20 had an EOL on 2015-06-23, so we'd be building on an unsupported platform which is probably not a great idea. RHEL7 is libc 2.17 while Ubuntu 14.04 is 2.19 which is not a great mix I suppose. Is there any chance of getting an EL variant onto osuosl?

[jbergstroem]

available os'es:

..perhaps Michael can contact their helpdesk and see if we can get more options on there.

[rvagg]

Oh, and the currently oldest supported Fedora, 23, uses libc 2.21, which is obviously far from ideal and 23 is EOL in 6 months I think anyway. This is the problem I have with using Fedora in general, it moves too fast and doesn't have a long-term strategy.

[rvagg]

Debian 7 is a bit of a sweet-spot in terms of having an old enough libc. Compiler problems are the cause for concern there however it's something we've had to deal with before: https://github.com/nodejs/build/tree/master/setup/debian-wheezy-gcc and I'm currently dealing with it for ARM: nodejs/node#4531 (I have new gcc 4.9 packages built on a new machine, just need transfer them to the test & release machines and store them somewhere for future use).

[rvagg]

ping @mhdawson

[jbergstroem]

@rvagg he's currently on vacation. I can query osuosl about getting debian7 on there?

[rvagg]

it can wait till he gets back, debian7 gets complicated with gcc and I'd rather avoid that complication if we can manage it .. somehow .. centos6 / rhel6?

[jbergstroem]

Yeah, I'd like to avoid gcc juggling as well.

CentOS is a no go: https://wiki.centos.org/About/Product
rhel6 I can query about.

[jbergstroem]

Seeing how its not very complicated to crossdev -- should we perhaps take that into consideration?

[mhdawson]

Can you expand on what you mean by crossdev ? Something like compiling on x86 targetting PPC ?

The action I had before I left on vacation was to see if a binary build on Ubuntu 14.04 runs on RHEL 7. I think it does for LE so I just need to check for BE. What I need to do is spin up an Ubuntu 14.04 BE machine compile and then test on one of our RHEL 7 machines. (I don't have a ubuntu 14.04 BE machine handy). I think we need to at least support RHEL 7 and if we can do that with an ubuntu 14.04 build then its probably the right answer unless somebody cries out for RHEL 6 support.

I'll still catching up but I'll try to out the ubuntu 14.04 build/run on RHEL 7 this week.

[jbergstroem]

Regarding crossdev I was mostly referring to setting up a toolchain of our own on a host, but it'd probably be simpler using debian7 as Rod did above.

[mhdawson]

I tried building on BE unbuntu 14.04 and it seems to be missing some compiler defines present on RHEL/Fedora machines (PPC64). I also spun up a debian7 machine and it also had the same problem. I tried hacking the detection in Node to work around it but it looks like it is also used in the dependencies. I'll have to chase down what's going on in this respect.

@rvagg @jbergstroem what I'd like to do for now is to separate the efforts for LE and BE. On the LE front is there anything else we need before it becomes part of the standard releases ?

[jbergstroem]

I think we've successfully built a few LE nightlies; shouldn't be a problem. @rvagg?

[rvagg]

Yeah, I think we can move forward with LE builds, probably not for the security releases but perhaps the ones after that. Where do we have support @mhdawson? Just v5 or is v4 gtg as far as you know too?

[mhdawson]

The release after the security one sounds good.

v4 and later should be good to go.

[mhdawson]

Seems like 4.3.1 went out without the LE builds. Can we confirm they will be part of the 4.4 planned as the next release ?

[rvagg]

This is currently commented out in iojs+release:

# if [[ $NODE_LABELS =~ ppc && $disttype =~ release|custom ]]; then
#   echo "Not building $disttype on PPC slave"
#   exit 0
# fi

But https://ci.nodejs.org/job/iojs+release/404/nodes=ppcle-ubuntu1404-release-64/console seems to be the job for 4.3.1 and it went out with:

Not building release on PPC slave

So did someone update this in the meantime?

@mhdawson v5 is likely next Tuesday and that'll be our next chance to test, I'll try and remember to watch for this but you might need to follow-up. Even if it fails we can add a build after the fact, it's just a tiny bit messy and I'm not sure it's a great idea on an LTS in case people are relying on the exact SHASUMS* files as they are when we announced it (who knows how people use these things in the wild).

[MylesBorins]

@rvagg I believe that @jbergstroem updated that just today so we can try to get ppc up and running with the v4.4.0-rc

[jbergstroem]

@rvagg that'd be me; I did post-release after talking to @thealphanerd and seeing @mhdawson's update here.

[mhdawson]

Sniff tested the v4.4.0-rc by starting repl and installing/building native module. Looks ok.
In terms of the download page do we need to co-ordinate with anybody else to have a row added for ppc le linux ?

[jbergstroem]

I've tested it on one of the ppcle machines.

[rvagg]

need to bring @nodejs/website into this discussion

tbh I'm not convinced we should be bothering to advertise this on the downloads page, it's already kind of cluttered as it is, even the "SunOS" option is questionable. https://nodejs.org/metrics/summaries/os.png - only 200,000 downloads of those binaries since April 2014 and most of those would be mirrors.

On one hand it's nice to show how many platforms and architectures we support, that's a great positive message, on the other hand, it complicates the options and muddies the water for people who just want to download Node.js! "I've got a Mac, should I be using the PPC binaries or is that not a thing now?". "Hey, I have an ancient Mac, can I get Node running on it with these!?".

Maybe this is just about reorganising that downloads widget but I'm a -0 on expanding it atm.

[jbergstroem]

Fwiw I'm with @rvagg. Exotic os'es doesn't need to be part of the full listing (I guess we could define what exotic means as well).

[mhdawson]

I can see having the most common ones up front but I think we need links to the others which are reasonably easy to find. Out of curiosity what are the download stats for arm ? It doesn't seem right to have the PPC Linux downloads to be the only ones excluded from the main list.

I think confusion for MAC owners should be covered by making sure the name has Linux in it. Something like PPC Linux

[jasnell]

I'd like to see us include all the download options but put the less common ones under a "More options" section or some such.

[rvagg]

I suspect we're all generally on the same page here and this is just a matter of design, or organisation. More common options should be super easy to select, advanced options should be separated from them in some way that lets advanced users select those.

ARM stats can be found here: https://nodejs.org/metrics/summaries/arch.png, I'm surprised that armv8 is as high as it is but they are all still pretty low and I'd +1 moving them down to an advanced section too until they start to become a more common download option (stick a .csv on the end of that URL to look at actual data, I can't see arm trends from the graph view). I'm not sure how to deal with them as a block though, if taken as a whole the numbers are decent but if it's only armv7 that trends upward do we still keep them together as a group?

@nodejs/website I think we need some help on coming up with a download table solution that provides us with a common grouping and an advanced grouping.

[mhdawson]

Just to confirm I agree with @rvagg's last comment, I'm not sure I'd use "advanced" but the two tier approach makes sense to me. In terms of Arm I think I'd want to keep them together until that causes problems visually. The same would go with power once we get BE included. In both cases if somebody is looking for a particular architecture I think finding them in the same place makes sense as long as it does not compromise how the page looks.

[mhdawson]

PPC LE is now in the standard releases. Going to close this in favor of #377 where we will capture progress on doing the same for BE