-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide an official ARM docker image and/or provide documentation for building on ARM #1861
Comments
@gizmochief7 it's been a little while, have you made progress? It sounds like you're building for 32-bit ARMv7; my particular interest is in arm64 builds. |
@vielmetti It has been and I almost forgot about posting this. We did not have any further luck with this attempt and without any community help we decided to move past Envoy. We tried out another reverse proxy/side car project for micro services and it just worked out of the box on both 32-bit and 64-bit. So we haven't wasted any more time here. Sorry I couldn't be more help. |
Sorry, didn't see this bug: my cmake files are here:
https://github.com/costinm/istio-build (tested on pi and android),
I'm using them with repo to get the deps ( start from
https://github.com/costinm/istio-repo )
There is another PR needed to compile head ( another option with 64bit int
) - will send it when I get back from vacation.
I've been running envoy on my home pi (for a personal server) for few
weeks, it's pretty stable and light, but I haven't
made any progress in automating the build, CircleCI doesn't seem to like my
build.
…On Mon, Nov 20, 2017 at 8:44 PM, gizmochief7 ***@***.***> wrote:
@vielmetti <https://github.com/vielmetti> It has been and I almost forgot
about posting this. We did not have any further luck with this attempt and
without any community help we decided to move past Envoy. We tried out
another reverse proxy/side car project for micro services and it just
worked out of the box on both 32-bit and 64-bit. So we haven't wasted any
more time here. Sorry I couldn't be more help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1861 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAFI6jvr_r5AMUuCa4cAUnkmziR4N3x5ks5s4lUkgaJpZM4P5-S8>
.
|
fwiw, I tried to build bazel on my pi (yes, it took several hours), and then built envoy on pi for armv7l 64bit. While envoy started up and was able to talk to http services, it had difficulty talking to certain https services (esp twitter). There seemed to be some boringssl issue that prevented ssl connections from working. My hunch was that it didn't compile properly. At that point, I gave up, because I had spent 4 solid days fighting bazel on pi. |
I think the cross-compilation option is faster and less painful - and easy
to expand to other platforms.
Boringssl and most of the dependencies have native support for cmake (or
can be easily adapted) - one of the problems with building with
'bazel' on pi is that deps are not really built with bazel, so even if
bazel can cross-compile c++ code, it would still not help us, most of the
code comes from the libraries compiled with autoconf. Of course autoconf
also supports cross - but then I also wanted to run it on android,
and cmake is supported very nicely by the NDK.
It requires some periodic tweaking - adding the new envoy files and fixing
ocasional 32-bit regressions (so far only the flags package seems to be a
problem - not clear why), but it's a pretty fast build.
But I appreciate the pain and effort in straight-compiling and fighting
bazel :-)
…On Tue, Nov 21, 2017 at 6:35 AM, Shriram Rajagopalan < ***@***.***> wrote:
fwiw, I tried to build bazel on my pi (yes, it took several hours), and
then built envoy on pi for armv7l 64bit. While envoy started up and was
able to talk to http services, it had difficulty talking to certain https
services (esp twitter). There seemed to be some boringssl issue that
prevented ssl connections from working. My hunch was that it didn't compile
properly. At that point, I gave up, because I had spent 4 solid days
fighting bazel on pi.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1861 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAFI6mUOLKwygzX6Xwx9LXYCxrEc7_oEks5s4t-fgaJpZM4P5-S8>
.
|
https://github.com/costinm/istio-build/releases if you want to try some
(older) binaries, matching ~ Istio 0.2
…On Tue, Nov 21, 2017 at 9:23 AM, Costin Manolache ***@***.***> wrote:
I think the cross-compilation option is faster and less painful - and easy
to expand to other platforms.
Boringssl and most of the dependencies have native support for cmake (or
can be easily adapted) - one of the problems with building with
'bazel' on pi is that deps are not really built with bazel, so even if
bazel can cross-compile c++ code, it would still not help us, most of the
code comes from the libraries compiled with autoconf. Of course autoconf
also supports cross - but then I also wanted to run it on android,
and cmake is supported very nicely by the NDK.
It requires some periodic tweaking - adding the new envoy files and fixing
ocasional 32-bit regressions (so far only the flags package seems to be a
problem - not clear why), but it's a pretty fast build.
But I appreciate the pain and effort in straight-compiling and fighting
bazel :-)
On Tue, Nov 21, 2017 at 6:35 AM, Shriram Rajagopalan <
***@***.***> wrote:
> fwiw, I tried to build bazel on my pi (yes, it took several hours), and
> then built envoy on pi for armv7l 64bit. While envoy started up and was
> able to talk to http services, it had difficulty talking to certain https
> services (esp twitter). There seemed to be some boringssl issue that
> prevented ssl connections from working. My hunch was that it didn't compile
> properly. At that point, I gave up, because I had spent 4 solid days
> fighting bazel on pi.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#1861 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAFI6mUOLKwygzX6Xwx9LXYCxrEc7_oEks5s4t-fgaJpZM4P5-S8>
> .
>
|
I've secured a Pine64 Rock64 SOC which is ARM8 / aarch64 and has 4GB of RAM. I've successfully I've leveraged @clnperez way of removing the Lua JIT. I can't get her trick to use the locally installed Go to work I've modified I'm using
This prevents the build automatically downloading Go for The build then fails with
My suspicion is that the lyft/protoc-gen-validate is hard coded for linux_amd64 (or considering compilation works on MacOS / Darwin maybe it just doesn't support ARM). Any help from Bazel wizards like @jmillikin-stripe or from people that have tried non amd64 compilation like @clnperez @costinm @rshriram would be great. |
Issue noted in bufbuild/protoc-gen-validate#75 - there are definitely some hard-coded dependencies in lyft/protoc-gen-validate to amd64 binaries. |
I'll try and help. I don't have much ARM experience but a coworker (@tophj-ibm) does. So we'll try to put our heads together on this one. Any progress we can make would be great. We'd also like to have a ppc64le docker image. wonder twins unite? My first guess is that something in cc_configure works differently for ARM. I ran into some things where, when using certain flags, gcc would behave differently for power than it did for x86. envoy, for the last release, seems to have come to rely a lot more on how bazel configured the gcc toolchain and did a little less customization. That got rid of almost all the issues I was seeing, but maybe there is some more tweaking needed for ARM. For your WORKSPACE file, what's the logic behind including the call to `go_download_sdk'? |
Wonder twins unite @clnperez! Without adding the specific |
The issue of Bazel mis-identifying the aarch64 system is noted at bazel-contrib/rules_go#1506 - thanks for the diagnosis. |
Before resuming work on this I think we should prioritize moving to rules_go 0.12.0. At the moment a build with 0.12.0 fails with a |
So, I'm hitting this now as well on ppc64le. :D |
@clnperez I've had success this week building on aarch64 / arm64 using these patches from @lubinsz #3681 (comment). @htuch unblocked protobuf 3.6.0 with this patch to lyft/protoc-gen-validate bufbuild/protoc-gen-validate@345b6b4. rules_go 0.12.0 is still blocked. |
@moderation i just took a look at those patches, and they look like they're all adding arm support, whereas we already have power support. i'm actually hitting a different error today than I did last night, and that is the gazelle compile (at least i think it's the gazelle compile) isn't finding gcc in my PATH. I might open a new issue for that. |
@clnperez We were having the same issue on arm64. The patch for rules_go at lubinsz/rules_go@497b488 fixed that issue. Not sure how portable that patch is but might give you some ideas. |
@moderation using it as-is seems to break compatibility with lyft/protoc-gen-validate as the patch for rules_go is based off of a branch that deprecated there are so many moving parts to trying out one thing, though, i could have missed something. but i think i have it all lined up correctly. i'll dig some more and see what i find. edit: link the link-only patch based off of rules_go 0.11.1: clnperez/rules_go@2f55b46 |
@moderation it turns out this was what i needed: bazel-contrib/bazel-gazelle#242 |
@clnperez this is similar to this patch that enabled the arm64 build - lubinsz/rules_go@497b488#diff-b88d1275b3ee8565f7821ba59b402139. Thanks @lubinsz. It would be good to go back and have these changes included upstream. |
Several upstream issues in |
Would be super interested in an arm64 docker image! |
Put this together but it does not build envoy, something about FROM alpine:edge AS bazel
ARG BAZEL_VERSION='3.3.1'
RUN mkdir /bazel
WORKDIR /bazel
RUN wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip
RUN unzip bazel-${BAZEL_VERSION}-dist.zip
# script needs bash, build needs rest
RUN apk add bash=5.0.17-r0 \
openjdk8=8.242.08-r2 \
build-base=0.5-r2 \
linux-headers=5.4.5-r1 \
zip=3.0-r8 \
python3=3.8.3-r0
RUN ln -s /usr/bin/python3 /usr/local/bin/python && \
env JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
FROM alpine:edge AS envoy
RUN apk add git=2.27.0-r0 \
openjdk8=8.242.08-r2 \
autoconf=2.69-r2 \
automake=1.16.2-r0 \
libtool=2.4.6-r7 \
samurai=1.1-r0 \
cmake=3.17.3-r0 \
python3=3.8.3-r0 \
build-base=0.5-r2 \
llvm10-dev=10.0.0-r2 \
go=1.14.4-r1 \
clang=10.0.0-r3 \
linux-headers=5.4.5-r1 \
libc6-compat=1.1.24-r9 \
musl-dev=1.1.24-r9
RUN go get -u github.com/bazelbuild/buildtools/buildifier
RUN go get -u github.com/bazelbuild/buildtools/buildozer
RUN git clone https://github.com/envoyproxy/envoy.git /envoy
WORKDIR /envoy
COPY --from=bazel /bazel/output/bazel /usr/local/bin/bazel
RUN ln -sf /usr/lib/llvm10/ /opt/llvm && \
ln -sf /usr/bin/python3 /usr/local/bin/python && \
env JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk \
bazel build //source/exe:envoy-static --sandbox_debug
|
|
@Jingzhao123 ^^ |
@trilom It is not recommended to build a musl (alpine) based binary, you have to disable signal_actions and gperftools. Disabling latter will impact performance significantly (around 30~40% on x64 on a simple benchmark) |
I did read that at one point, I'm not versed in building with musl but I did read similar, and that is for the most part why I stopped with the alpine approach. Thank you for the information. My ideal goal is to use something similar to |
Realistically speaking, how far are we from an arm64 image that could be considered stable? Also (though, this one is my fault for not knowing), how long does it take to build an arm64 image with that container? Hours, minutes? Days? I have a RPi4 available that I could use to build Envoy. Also, I'm getting an error trying to pull the images:
EDIT: Nevermind the error.. I forgot I had a 32bit OS on this Pi |
@CelsoSantos I ran the build just a couple of days ago on my RPi4. It produced a static non-containerized binary that I was able to drop onto my target device and run. The build time was around 12 hours, though I did notice that it was just using a single core. |
@carmiac the single core issue you saw began with a recent release of Bazel. Here is what I posted in the Bazel Slack.
Setting this flag will allow you to use more than one core. I had a 4G RAM RPi4 and I had to restrict the build using Lastly check out https://stevesloka.com/compile-envoy-on-raspberry-pi4/. |
@CelsoSantos It's been a while, but IIRC the command I ran back on the 18th of May worked and generated a Docker image (which I uploaded to Docker Hub as al45tair/envoy-arm64). It seems to work. @moderation @carmiac It's probably worth getting some time on an ARM64 server machine rather than building on a Pi. Even if you don't have access to one at work (I'm lucky in that regard), you can spin them up in AWS — there are Graviton and Graviton2 instances available, I believe — and they'll build it a lot faster than a Pi will. |
@al45tair I'm pretty happy with my local RPi4 dev environment. After the initial build the subsequent incremental builds are relatively quick and compiling out all the extensions you don't need works well. I'm in a very small minority of people who don't think Docker helps with anything so I'm building Bazel and Envoy straight on the RPi4 |
I wouldn't be comfortable saying it is stable until it passes most of our test suite on arm64. |
A note that Bazel 3.4.0 now ships with official arm64 binaries - bazelbuild/bazel#8833 (comment) That may save you a build step, @moderation , if you can adapt to any changes associated with the Bazel point release at 3.4.0. |
Thanks @vielmetti. Looks like they need to work on their automation a bit as the latest release 3.4.1 isn't yet available. But definitely looks promising |
Risk Level: Low Testing: CI Docs Changes: N/A Release Notes: N/A (should be added when docker build is enabled) Part of #1861 Signed-off-by: Lizan Zhou <[email protected]>
Risk Level: Low Testing: CI Docs Changes: N/A Release Notes: N/A (should be added when docker build is enabled) Part of envoyproxy#1861 Signed-off-by: Lizan Zhou <[email protected]> Signed-off-by: Kevin Baichoo <[email protected]>
In this patch, it will enable the envoyproxy/envoy arm image to build in community arm CI environments. 1. Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the buildx tool to build the envoyproxy/envoy arm images on x86 platform. 2. Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then invoke the docker_ci.sh scripts to generate images. Risk Level: Medium (of breaking images) Testing: CI Docs Changes: N/A Release Notes: Added Fixes envoyproxy#1861 Signed-off-by: Jingzhao.Ni <[email protected]>
In this patch, it will enable the envoyproxy/envoy arm image to build in community arm CI environments. 1. Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the buildx tool to build the envoyproxy/envoy arm images on x86 platform. 2. Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then invoke the docker_ci.sh scripts to generate images. Risk Level: Medium (of breaking images) Testing: CI Docs Changes: N/A Release Notes: Added Fixes envoyproxy#1861 Signed-off-by: Jingzhao.Ni <[email protected]> Signed-off-by: chaoqinli <[email protected]>
In this patch, it will enable the envoyproxy/envoy arm image to build in community arm CI environments. 1. Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the buildx tool to build the envoyproxy/envoy arm images on x86 platform. 2. Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then invoke the docker_ci.sh scripts to generate images. Risk Level: Medium (of breaking images) Testing: CI Docs Changes: N/A Release Notes: Added Fixes envoyproxy#1861 Signed-off-by: Jingzhao.Ni <[email protected]> Signed-off-by: chaoqinli <[email protected]>
Would it be possible to get an official Docker image of Envoy on ARM or at least some documentation on how to build it?
My team and I have tried off and on to build Envoy on ARM using the Bazel build process without any luck. Its challenging to say the least. I just saw a PR #1795 by @costinm dated a couple weeks ago based on the issue #1781 that talked about building it on ARM successfully after a few fixes. This is great news but we aren't sure how to duplicate it. @costinm mentioned that he used a custom CMAKE and a pi-cross-compiler to make it work. Documentation on this would be very helpful. Also, if there exist an image I could use for testing, then that would be very helpful too. Thanks again for such an awesome product!
The text was updated successfully, but these errors were encountered: