Add arm32v7 builder

[gleich]

Create a new file called arm.Dockerfile that allows you to run it on arm32v7 CPUs such as the Raspberry Pi's CPU.

[nhooyr]

Could you elaborate on what the current problem is and how this helps?

[gleich]

At the moment the docker image for code-server only works on x86 processors (desktop processor) and returns the following error when you try to run on an arm32v7 processor (such as the Raspberry Pi):

standard_init_linux.go:211: exec user process caused "exec format error"

This is due to the fact that the docker image is built on an x86 processor. In this case, we don't even need to change the base image thanks to the fact that node has an arm image already available. So, in theory, this adds support for running code-server on a Raspberry Pi!

One problem with this is that in order for it to work on a arm32v7 processor it needs to be built on an arm32v7 processor. Sadly github-actions and travis-ci both don't support you to run your CI on arm32 CPUs. There are a few solutions to this tough:

1. Run a Jenkins server on a Raspberry Pi that you or someone on the team buys. I made a docker image that comes with Jenkins and docker installed https://hub.docker.com/repository/docker/mattgleich/arm32v7jenkins-docker
2. Try building it on an x86 with a CPU arm emulation tool such as QEMU

I am going to work to get this working and will submit a PR once done

[sr229]

@nhooyr we tried building it over at Drone but it keeps OOMing, we're still investigating what we can do with the build toolchain since it should work under normal circumstances since we run with 128GB RAM in Drone Cloud

[nhooyr]

That is unfortunate news. Not sure what we can do.

[nhooyr]

https://discourse.drone.io/t/drone-cloud-arm32-ooms-when-building/6622

[nhooyr]

Ah so we need to compile for ARM32 on ARM64.

[sr229]

I'll see on how to compile the project using 32-bit libs but AFAIK documentation to do it for Node.js is quite scarce but I'll keep looking.

[sr229]

@nhooyr AFAIK we should be compiling this on a 32-bit capable container, preferrably Raspbian, in a ARM64 environment (Raspbian can run 32-bit userland against a 64 bit kernel so it shouldn't be an issue).

[aptalca]

@sr229 I'm with linuxserver.io and we publish a code-server image. We use native builders with our jenkins and have a bunch of arm32v7 devices. If you point me to your specific build instructions, I'll try to build it and let you know.

[aptalca]

I tried a build on an odroid xu4, running armbian buster (arm32v7), kernel 4.14.165.
Ran in an ubuntu bionic docker container with node 12.15.0 and yarn 1.21.1
Followed the steps from https://github.com/cdr/code-server#development
yarn watch resulted in the following error:

root@3d29582a8c4c:/tmp/vscode/src/vs/server# yarn watch
yarn run v1.21.1
$ cd ../../../ && yarn watch
$ gulp watch --max_old_space_size=4095
[15:29:46] Node flags detected: --max_old_space_size=4095
[15:29:46] Respawned to PID: 8612

<--- Last few GCs --->

[8612:0x1e47950]     2561 ms: Mark-sweep 20.7 (29.5) -> 20.7 (29.5) MB, 52.8 / 0.0 ms  (average mu = 0.606, current mu = 0.001) allocation failure GC in old space requested
[8612:0x1e47950]     2614 ms: Mark-sweep 20.7 (29.5) -> 20.7 (22.0) MB, 53.4 / 0.0 ms  (average mu = 0.437, current mu = 0.001) last resort GC in old space requested
[8612:0x1e47950]     2669 ms: Mark-sweep 20.7 (22.0) -> 20.7 (22.0) MB, 54.6 / 0.0 ms  (average mu = 0.278, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35b0e235 <JSObject>
    0: builtin exit frame: compileFunction(this=0x2b042315 <JSGlobal Object>,0x2fd0a595 <JSArray[5]>,0x2fd0a585 <JSArray[0]>,0x25340275 <undefined>,0x253403bd <false>,0x25340275 <undefined>,0,0,0x2fd1f5e9 <String[#53]: /tmp/vscode/node_modules/typescript/lib/typescript.js>,0xb0580089 <Very long string[7797825]>,0x2b042315 <JSGlobal Object>)

    1: _compile [0x4407861d] [internal/modules/cjs/loader....

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[nhooyr]

@aptalca yea thats the same error @sr229 was getting. We need to cross compile on 64bit arm for 32bit arm.

[aptalca]

Oh I see now, my mistake. I hadn't read the linked thread on drone and incorrectly assumed drone was using arm64 builders for arm32 work and that cross compile was the issue. That's why I tried on native arm32.

[nhooyr]

Some discussion at #35

[deftdawg]

My process for cobbling together builds is as follows:

1. Build Code Server on aarch64 docker container on a X86_64 host using qemu-aarch64-static via binfmt (linking needs ~12GB of RAM; impossible on a 32-bit arch which can address a max of 4GB per process)
2. Shipping the compiled directory to a native arm6 system (steps are the same for arm32v7); in my case an RPi Zero W / RPi 2
3. Going into code-server-*-built and running npm rebuild on everything (node arm6 unofficial build needed)

That produces something that can be run via the node arm6 unofficial build via node ./build/code-server*/out/vs/server/main.js --auth none

The packing can't be done on a RPi2(1GB), because build.ts is trying to pass the entire packed binary to fs.writeFile as a buffer which easily exceeds the 1GB; from what I've read it should be possible to use streams to reduce that footprint to something more sane (streams -> https://medium.com/dev-bits/writing-memory-efficient-software-applications-in-node-js-5575f646b67f)

I'm not sure if streams could also help with the 12GB ram footprint of the build phase; I don't even know where to start investigating that one as build has way more going on.

[sr229]

I'm going to assume the build script is passing the entire thing as a buffer hence the OOM errors. I think this is something @code-asher would likely be happy to investigate to optimize the builds.

[code-asher]

The next version will be out soon and it no longer packages into a single binary so that should simplify building a bit.

[jsco2t]

Question (and if this needs to be a new issue just let me know):

I noticed that some arm binaries are being produced as part of the official releases. Ex: https://github.com/cdr/code-server/releases/tag/2.1698

I grabbed the ARM64 versions and deployed them onto a RaspberryPi just to see if they would work. I'm getting a bad image format style error:

code-server2.1698-vsc1.41.1-linux-arm64 $ ./code-server
-bash: ./code-server: cannot execute binary file: Exec format error

This leads me to believe that the current ARM binaries aren't compiled for the version of ARM the raspberryPi I am using runs (I have access to a Pi Zero W, as well as a Pi 4). Is there any chance we can get full build instructions for ARM if we want to try to compile and run the server on a raspberry pi.

• I know that running the server on a RaspberryPi device, especially the 2.x code versions, will result in an OOM error - even if I managed to get the binary to run. I also understand the experience will be substandard. That being said - I still love the idea of being able to run this on ARM devices.

[code-asher]

Hopefully with v3 we'll be able to distribute arm64 releases that work a bit better but in the meantime check out https://github.com/cdr/code-server/blob/master/doc/CONTRIBUTING.md#build for how to build.

[skabber]

I tried building on my RaspberryPi 4 according to the CONTRIBUTING.md file but ran into this error.

pi@raspberrypi:~/Projects/code-server $ yarn build
yarn run v1.22.4
$ yarn runner build
$ cd ./ci && NODE_OPTIONS=--max_old_space_size=32384 ts-node ./build.ts build
[default] rootPath is "/home/pi/Projects/code-server"
[default] codeServerVersion is "3.0.1"
[build] cleaning up old build...took 8ms
[build] building vs code...failed
Command failed: yarn gulp compile-build
/usr/bin/node[16083]: ../src/node_platform.cc:414:std::shared_ptr<node::PerIsolatePlatformData> node::NodePlatform::ForIsolate(v8::Isolate*): Assertion `data' failed.
Aborted
error Command failed with exit code 134.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 

[code-asher]

That's a new one. Are you building with Node 12?

[soyflourbread]

That's a new one. Are you building with Node 12?

Same problem, node 10.17.0, yarn 1.22.4, debian buster armhf

[skabber]

Once I switched from node 10 to 12, I get a JavaScript heap out of memory error.

[b00gizm]

Once I switched from node 10 to 12, I get a JavaScript heap out of memory error.

Same for me.

Node: v12.16.1
Yarn: 1.22.4

[deftdawg]

Gents, kindly read the entire thread before "me too" posting... Per previous posts (#1337 (comment)), it requires 12GB of RAM to build.

[code-asher]

I'm not super familiar with Raspbian but a cursory search seems to suggest it doesn't have a firewall enabled by default. Have you enabled iptables or ufw or something?

[sr229]

I think we should still supply prebuilt binaries and have postinstall download them (fallback to compilation if cannot find a prebuilt) to save time for them.

[7HE-W0R1D]

My process for cobbling together builds is as follows:

1. Build Code Server on aarch64 docker container on a X86_64 host using qemu-aarch64-static via binfmt (linking needs ~12GB of RAM; impossible on a 32-bit arch which can address a max of 4GB per process)
2. Shipping the compiled directory to a native arm6 system (steps are the same for arm32v7); in my case an RPi Zero W / RPi 2
3. Going into code-server-*-built and running npm rebuild on everything (node arm6 unofficial build needed)

That produces something that can be run via the node arm6 unofficial build via node ./build/code-server*/out/vs/server/main.js --auth none

The packing can't be done on a RPi2(1GB), because build.ts is trying to pass the entire packed binary to fs.writeFile as a buffer which easily exceeds the 1GB; from what I've read it should be possible to use streams to reduce that footprint to something more sane (streams -> https://medium.com/dev-bits/writing-memory-efficient-software-applications-in-node-js-5575f646b67f)

I'm not sure if streams could also help with the 12GB ram footprint of the build phase; I don't even know where to start investigating that one as build has way more going on.

Thanks for your package @deftdawg! In fact it is the only one that runs on my pi zero w. However, I'd like to try the latest build of code-server but I'm totally new to docker and binfmt. I tried googling for results but cannot find the right way to go. Could you elaborate a bit more on how to "Build Code Server on aarch64 docker container on a X86_64 host using qemu-aarch64-static via binfmt"? I'm on a windows 10 64bit laptop with docker and qemu installed, but I cannot figure out what to do next :(
Thanks in advance!

[deftdawg]

@7HE-W0R1D - Going from memory as my machine died; waiting for a NUC11 replacement to come in a few weeks hopefully.

You need a Linux machine (or VM) running X86_64 intel kernel with lots of RAM. Trying to do this with a docker/VM on Windows is probably going to be painfully slow (though it should be possible if you are able to allocate tonnes of memory to docker).

On your linux machine/VM, you need to install qemu-user-static and possibily qemu-user-binfmt if it doesn't install as a dep.

What it does is essentially register some scripts with the binfmt kernel module that when it sees an ARM executable (by looking at the magic line of the file) it automatically emulates arm64.

Then run an ARM64 distro using qemu-aarch64-static and then follow the build steps to compile code-server.

It's not worth the effort tbqh... Like what would you plan to do (development wise) on a Pi Zero that a newer version of VS Code is going to make any difference?

[7HE-W0R1D]

@7HE-W0R1D - Going from memory as my machine died; waiting for a NUC11 replacement to come in a few weeks hopefully.

You need a Linux machine (or VM) running X86_64 intel kernel with lots of RAM. Trying to do this with a docker/VM on Windows is probably going to be painfully slow (though it should be possible if you are able to allocate tonnes of memory to docker).

On your linux machine/VM, you need to install qemu-user-static and possibily qemu-user-binfmt if it doesn't install as a dep.

What it does is essentially register some scripts with the binfmt kernel module that when it sees an ARM executable (by looking at the magic line of the file) it automatically emulates arm64.

Then run an ARM64 distro using qemu-aarch64-static and then follow the build steps to compile code-server.

It's not worth the effort tbqh... Like what would you plan to do (development wise) on a Pi Zero that a newer version of VS Code is going to make any difference?

Thanks for the timely reply! Well the last line is real good question, I guess I will give the VM/Docker a shot but if it fails(probably will) I'll stick to your prebuild package.

[7HE-W0R1D]

@deftdawg also I saw someone saying #1337 (comment) this yarn add command can do the thing. I tried it and it does build the package successfully, like for 2000 or 3000 seconds. But when I run code-server, the welcome password screen appears and then there will be a 500 error with no extra messages. Does that mean its impossible to use the yarn install (or yarn add now) on rpi or it is something that is fixable?

[RobertCNelson]

I've kinda just jumped into this project for beagleboard.org , we had been shipping "cloud9" for the last 5 or 6 years as our default ide.. It's been un-maintined for last few years with me just hacking on it keep it building on debian's default version of nodejs.

So i started looking at this project last week.. I've been successful on building it a Raspberry Pi 4 8GB model running in armhf mode. Currently 3.8.1/3.9.0 run great on our BeagleBone Blacks' (AM335x) Cortex-A8 with 512MB of ram..

Only 2 patches are required..

First there is an arfifcal stop in ci/lib.sh:

https://github.com/cdr/code-server/blob/v3.9.0/ci/lib.sh#L37-L50

arch() {
  case "$(uname -m)" in
  aarch64)
    echo arm64
    ;;
  x86_64 | amd64)
    echo amd64
    ;;
  *)
    echo "unknown architecture $(uname -a)"
    exit 1
    ;;
  esac
}

I just added an armhf option, it's not really used anywhere, more then likely to just stop the build with an exit 1..

From 51644af7dbe139d3f8db4170c7a6da18e2072051 Mon Sep 17 00:00:00 2001
From: Robert Nelson <[email protected]>
Date: Wed, 10 Feb 2021 10:20:51 -0600
Subject: [PATCH] bbb.io: ci/lib.sh lets not stop when detecting armv7l

Signed-off-by: Robert Nelson <[email protected]>
---
 ci/lib.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ci/lib.sh b/ci/lib.sh
index 96a413f1c..179c27688 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -42,6 +42,9 @@ arch() {
   x86_64 | amd64)
     echo amd64
     ;;
+  armv7l)
+    echo armhf
+    ;;
   *)
     echo "unknown architecture $(uname -a)"
     exit 1

Second, there is a gulp call in lib/vscode/package.json that requires 8GB of memory for JavaScript, i backed this down to 6GB which allows it to run on the 8GB Pi 4.. While watching a build in TOP, i'm really wondering what the point of this 8GB setting was..

https://github.com/cdr/code-server/blob/v3.9.0/lib/vscode/package.json#L32

"gulp": "gulp --max_old_space_size=8192",

From 00cf8a3d8a5d95183e7fd251d65cbd368e8c0c3e Mon Sep 17 00:00:00 2001
From: Robert Nelson <[email protected]>
Date: Wed, 10 Feb 2021 12:35:22 -0600
Subject: [PATCH] bbb.io: lib/vscode/package.json: limit memory from 8gb to 6gb

Signed-off-by: Robert Nelson <[email protected]>
---
 lib/vscode/package.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/vscode/package.json b/lib/vscode/package.json
index d4efccba3..4e4e5060f 100644
--- a/lib/vscode/package.json
+++ b/lib/vscode/package.json
@@ -29,7 +29,7 @@
     "kill-watch-extensionsd": "deemon --kill yarn watch-extensions",
     "mocha": "mocha test/unit/node/all.js --delay",
     "precommit": "node build/hygiene.js",
-    "gulp": "gulp --max_old_space_size=8192",
+    "gulp": "gulp --max_old_space_size=6144",
     "electron": "node build/lib/electron",
     "7z": "7z",
     "update-grammars": "node build/npm/update-all-grammars.js",

anyways to build i grabbed nodejs' armv7l build from: https://nodejs.org/dist/latest-v12.x/ (as debian buster's nodejs is v10...)

Installed yarn 1.22.5 with nodejs's npm/nodejs build, then just ran (and waited about an hour..)

yarn
yarn build
yarn build:vscode
yarn release
yarn release:standalone

Here are my packaged build i'm testing for BeagleBoard.org, they do require a pure Debian userspace for armhf (ARMV7-A..)

https://rcn-ee.net/repos/debian/pool/main/b/bb-code-server/

These can be run as:

/opt/code-server/app/bin/code-server --auth none --bind-addr 0.0.0.0:3000

I'm still working on our *.deb package ^, as I'm hacking on a systemd socket startup.. (only 512MB of ram on our boards..) and tuning it for our default first user (debian: user 1000), along with a list of pre-installed extensions..

For reference here is my full tree: https://github.com/rcn-ee/code-server/tree/v3.9.0-bbb.io (i nuked a few git calls as i'm building it in a debian sbuild with no git installed..)

Regards,

[eh8]

Are there still any plans to release the armhf binaries to accompany x86 and arm64 in the future via GitHub? The linuxserver docker images are neat but I'm hoping for a bare metal installation if possible.

[sr229]

Are there still any plans to release the armhf binaries to accompany x86 and arm64 in the future via GitHub? The linuxserver docker images are neat but I'm hoping for a bare metal installation if possible.

There isn't that large demand if you ask me, plus most of the development happens in ARM64 or x86_64 hosts, so there won't be any demand for this.

[eh8]

There isn't that large demand if you ask me, plus most of the development happens in ARM64 or x86_64 hosts, so there won't be any demand for this.

I can't speak beyond my own circumstances, but as I understand there many use Raspberry Pi/ODroid/etc either as standalone units or in k8'ed configuration as build servers which would suit code-server. It would be a presupposition to discount these platforms for lack of computing power/memory simply because there are small Raspberry Pis that exist.

It also looks like @RobertCNelson was able to tweak the build scripts trivially to produce viable 32-bit ARM builds, so I would hope that code-server can reproduce the builds officially.

[jsjoeio]

If it's trivial to add + maintain, then we'd be happy to support this.

@eh8 we'd happily review a PR if you submitted one (same goes to you @RobertCNelson). You two probably more than we do (or me at least).

Otherwise, we can try and take a look at this in the near future.

[sr229]

There isn't that large demand if you ask me, plus most of the development happens in ARM64 or x86_64 hosts, so there won't be any demand for this.

I can't speak beyond my own circumstances, but as I understand there many use Raspberry Pi/ODroid/etc either as standalone units or in k8'ed configuration as build servers which would suit code-server. It would be a presupposition to discount these platforms for lack of computing power/memory simply because there are small Raspberry Pis that exist.

It also looks like @RobertCNelson was able to tweak the build scripts trivially to produce viable 32-bit ARM builds, so I would hope that code-server can reproduce the builds officially.

This is a valid reason though it's pretty hard to compile for ARMv32 considering there aren't any more public builders offering this, and the fact it's a little hard to configure the CI to do the compile steps for it (only for it to break really horribly).

[oxy]

Now that we have arm64-on-x64 cross compile, it should be fairly simple to also do armv7-on-x64 builds; if there's no PRs for this, I'll look into it in the future, but there are currently other issues that are higher priority.