Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline setup for aarch64-linux #9326

Merged
merged 55 commits into from
Nov 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
73fcf21
build hab on aarch64-linux
sajjaphani Jul 23, 2024
ba51756
try with arm64-linux
sajjaphani Jul 23, 2024
36dd3ed
try with arm-linux
sajjaphani Jul 23, 2024
d60ff0e
with docker-linux-arm64 agent
sajjaphani Jul 23, 2024
e66c553
another try
sajjaphani Jul 23, 2024
82843de
fix target
sajjaphani Jul 23, 2024
562c30b
temporary channel for arm64
sajjaphani Sep 6, 2024
f01ea3d
adjust rustc_target from pkg_target
sajjaphani Sep 6, 2024
1e36f37
build pkgs on aarch64 in ci
sajjaphani Sep 9, 2024
e44cd28
try with default-privileged-aarch64
sajjaphani Sep 9, 2024
6ffe273
target default-privileged-aarch64 queue
sajjaphani Sep 9, 2024
4ac55a7
try varied config
sajjaphani Sep 9, 2024
efc61d0
default-arm64
sajjaphani Sep 9, 2024
1ca7adc
try without privileged
sajjaphani Sep 9, 2024
e8234bf
revert privileged
sajjaphani Sep 9, 2024
eeac7ce
fix shellcheck
sajjaphani Sep 9, 2024
dea96d8
trigger pipeline
sajjaphani Sep 11, 2024
7da0dd3
try without quotes
sajjaphani Sep 11, 2024
5727d66
add missing space in cmd aqrgs
sajjaphani Sep 11, 2024
047bb91
try with docker-linux-amazon-2023-arm64
sajjaphani Sep 19, 2024
5f90c51
try sudo
sajjaphani Sep 19, 2024
dc2dcc2
target docker-linux-amazon-2023-arm64 with sudo
sajjaphani Sep 19, 2024
b479d1d
fix component name
sajjaphani Sep 23, 2024
19cc606
disable shellcheck SC2086
sajjaphani Sep 25, 2024
05ba30d
first cargo test
sajjaphani Oct 4, 2024
31f473f
fix command
sajjaphani Oct 4, 2024
aceebb1
add deps
sajjaphani Oct 4, 2024
e2e3a50
add all unit tests
sajjaphani Oct 4, 2024
0e99a63
fix shellcheck
sajjaphani Oct 4, 2024
e3e3225
adjust script, fix test
sajjaphani Oct 7, 2024
229bb41
install hab
sajjaphani Oct 7, 2024
319cf48
install hab with curl
sajjaphani Oct 7, 2024
cf3f5c2
install hab
sajjaphani Oct 7, 2024
3959879
fix cfg
sajjaphani Oct 7, 2024
761a0e1
fix fmt
sajjaphani Oct 7, 2024
d0133f1
updage agent
sajjaphani Oct 8, 2024
3d8c8cf
adds space
sajjaphani Oct 8, 2024
ff3f02f
should be an error
sajjaphani Oct 17, 2024
e8da464
aarch64-linux has standard pkgs
sajjaphani Oct 17, 2024
798661f
fix unused import
sajjaphani Oct 17, 2024
973758c
remove cfize, mesomise and fix container plan
sajjaphani Oct 17, 2024
4b121e8
set interpreter for aarch64 tests
sajjaphani Oct 18, 2024
aaa5bdd
fix rustfmt
sajjaphani Oct 18, 2024
2a26e83
aarch64 packages are updated against LTS-2024
sajjaphani Oct 25, 2024
a53ce9c
fix shellcheck
sajjaphani Oct 25, 2024
339d7e9
set missing fallback channel
sajjaphani Oct 28, 2024
d4e011b
add aarch64 to the release pipeline
sajjaphani Oct 28, 2024
f5eb005
temporarily prefer local hab* pkgs
sajjaphani Oct 28, 2024
1385d03
use core origin in verity
sajjaphani Oct 29, 2024
53dff91
prefer local installed
sajjaphani Oct 29, 2024
f934eb7
another try
sajjaphani Oct 29, 2024
c102d24
set studio fallback channel
sajjaphani Oct 30, 2024
82fc262
remove unneeded target
sajjaphani Oct 30, 2024
c9befb7
correct fallback channel
sajjaphani Oct 30, 2024
5866c8c
update fallback channel and remove arm binary upload
sajjaphani Nov 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 82 additions & 6 deletions .expeditor/release_habitat.pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,6 @@ steps:
# Release!
#######################################################################

- label: "[:linux: build hab (ARM)]"
command:
- .expeditor/scripts/release_habitat/package_and_upload_arm_binary.sh
agents:
queue: docker-linux-arm64

- label: "[:linux: build hab]"
command:
- .expeditor/scripts/release_habitat/build_component.sh hab
Expand All @@ -38,6 +32,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we remove the [:linux: build hab (ARM)] build above?

- label: "[:linux: build aarch64 hab]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh hab

- label: "[:linux: :two: build hab]"
command:
- .expeditor/scripts/release_habitat/build_component.sh hab
Expand Down Expand Up @@ -111,6 +114,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 hab-plan-build]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh plan-build

- label: "[:linux: :two: build hab-plan-build]"
command:
- .expeditor/scripts/release_habitat/build_component.sh plan-build
Expand Down Expand Up @@ -146,6 +158,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 hab-backline]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh backline

- label: "[:linux: :two: build hab-backline]"
command:
- .expeditor/scripts/release_habitat/build_component.sh backline
Expand All @@ -170,6 +191,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 hab-studio]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh studio

- label: "[:linux: :two: build hab-studio]"
command:
- .expeditor/scripts/release_habitat/build_component.sh studio
Expand Down Expand Up @@ -208,6 +238,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 launcher]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh launcher

- label: "[:linux: :two: build launcher]"
command:
- .expeditor/scripts/release_habitat/build_component.sh launcher
Expand Down Expand Up @@ -241,6 +280,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 hab-sup]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh sup

- label: "[:linux: :two: build hab-sup]"
command:
- .expeditor/scripts/release_habitat/build_component.sh sup
Expand Down Expand Up @@ -274,6 +322,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 hab-pkg-export-container]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh pkg-export-container

- label: "[:windows: build hab-pkg-export-container]"
command:
- powershell .expeditor/scripts/release_habitat/build_component.ps1 pkg-export-container
Expand All @@ -295,6 +352,15 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: build aarch64 hab-pkg-export-tar]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/build_component.sh pkg-export-tar

- label: "[:linux: :two: build hab-pkg-export-tar]"
command:
- .expeditor/scripts/release_habitat/build_component.sh pkg-export-tar
Expand Down Expand Up @@ -352,6 +418,16 @@ steps:
environment:
- BUILD_PKG_TARGET=x86_64-linux

- label: "[:linux: upload hab aarch64 binary]"
agents:
queue: default-privileged-aarch64
env:
HAB_FALLBACK_CHANNEL: "LTS-2024"
BUILD_PKG_TARGET: "aarch64-linux"
command:
- sudo -E .expeditor/scripts/release_habitat/package_and_upload_binary.sh
if: build.creator.name == 'Chef Expeditor' || build.env("UPLOAD_AND_PROMOTE") == 'true'

- label: "[:linux: :two: upload hab binary]"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need a ARM :docker: Upload containers to Docker Hub too right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Will do that in a seperate PR!

command:
- .expeditor/scripts/release_habitat/package_and_upload_binary.sh
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ channel=$(get_release_channel)
echo "--- Channel: $channel - bldr url: $HAB_BLDR_URL"

declare -g hab_binary
install_release_channel_hab_binary "x86_64-linux"
install_release_channel_hab_binary "$BUILD_PKG_TARGET"

import_gpg_keys

Expand Down
3 changes: 2 additions & 1 deletion .expeditor/scripts/shared.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ curlbash_hab() {
sudo rm -rf /hab/pkgs/core/hab/0.82.0 || \
rm -rf /hab/pkgs/core/hab/0.82.0
fi

curl https://raw.githubusercontent.com/habitat-sh/habitat/main/components/hab/install.sh | sudo bash -s -- -t "$pkg_target"
case "${pkg_target}" in
x86_64-linux | x86_64-linux-kernel2)
x86_64-linux | aarch64-linux | x86_64-linux-kernel2)
hab_binary="/bin/hab"
;;
x86_64-darwin)
Expand Down
30 changes: 30 additions & 0 deletions .expeditor/scripts/verify/build_package-aarch64.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash

set -eou pipefail

package_path=${1?package_path argument required}

# Install hab from a temporarily uploaded aarch64 package
curl https://raw.githubusercontent.com/habitat-sh/habitat/main/components/hab/install.sh | sudo bash -s -- -t "$BUILD_PKG_TARGET" -c "$HAB_FALLBACK_CHANNEL" -v 1.6.1178
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we just put this package in stable and not need to rely on a specific build number? I don't see a problem with that since we have not publicized arm support and it wouldn't be possible for customers to see it on the official downloads site.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once this is merged, I am thinking of updating the channel to dev to build the latest hab and update the pipeline. If that doesn’t work, I can promote the build created by the release pipeline and then update it here.


# Since we are only verifying we don't have build failures, make everything
# temp!
export HAB_ORIGIN
HAB_ORIGIN=core
# let's make a selfcontained tempdir for this job
export JOB_TEMP_ROOT
JOB_TEMP_ROOT=$(mktemp -d /tmp/job-root-XXXXXX)
export HAB_CACHE_KEY_PATH
HAB_CACHE_KEY_PATH="$JOB_TEMP_ROOT/keys"

export HAB_STUDIO_SECRET_HAB_FALLBACK_CHANNEL
HAB_STUDIO_SECRET_HAB_FALLBACK_CHANNEL="$HAB_FALLBACK_CHANNEL"
export HAB_PREFER_LOCAL_CHEF_DEPS="true"

echo "--- :key: Generating temporary origin key"
hab origin key generate "$HAB_ORIGIN"
echo "--- :hab: Running hab pkg build for $package_path"
# Install the temporarily built hab-studio.
# Once hab is released in the LTS channel, this step may no longer be required.
hab pkg install core/hab-studio -c "$HAB_FALLBACK_CHANNEL"
hab pkg build "$package_path"
78 changes: 78 additions & 0 deletions .expeditor/scripts/verify/run_cargo_test-aarch64.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
#!/bin/bash

set -eou pipefail

# Install hab from a temporarily uploaded aarch64 package
curl https://raw.githubusercontent.com/habitat-sh/habitat/main/components/hab/install.sh | sudo bash -s -- -t aarch64-linux -c lts24-aarch64-linux -v 1.6.1178

# shellcheck source=.expeditor/scripts/shared.sh
source .expeditor/scripts/verify/shared.sh

if [[ ${1:-"--"} = "--" ]]; then
scope="habitat workspace"
else
component="$1"
shift
scope="$component"
fi

toolchain=$(get_toolchain)

sudo -E hab pkg install core/zeromq
sudo -E hab pkg install core/protobuf
sudo -E hab pkg install core/rust/"$toolchain"
sudo -E hab pkg install core/xz
sudo -E hab pkg install core/coreutils
sudo -E hab pkg install core/openssl
sudo -E hab pkg install core/perl
sudo -E hab pkg install core/make

export OPENSSL_DIR
OPENSSL_DIR="$(hab pkg path core/openssl)/bin"
export OPENSSL_LIB_DIR
OPENSSL_LIB_DIR="$(hab pkg path core/openssl)/lib"

export LIBZMQ_PREFIX
LIBZMQ_PREFIX=$(hab pkg path core/zeromq)
# now include zeromq and gcc so they exist in the runtime library path when cargo test is run
export LD_LIBRARY_PATH
LD_LIBRARY_PATH="$(hab pkg path core/gcc-base)/lib:$(hab pkg path core/zeromq)/lib:$(hab pkg path core/xz)/lib:$(hab pkg path core/openssl)/lib"

export LIBRARY_PATH
LIBRARY_PATH="$(hab pkg path core/xz)/lib"

export PROTOC_NO_VENDOR=1
export PROTOC
PROTOC=$(hab pkg path core/protobuf)/bin/protoc

_oldPth=$PATH
_pth="$(hab pkg path core/coreutils)/bin:$(hab pkg path core/openssl)/bin:$(hab pkg path core/perl)/bin:$(hab pkg path core/make)/bin"
eval "$(hab pkg env core/rust/"$toolchain"):$PATH"
export PATH="$PATH:$_pth:$_oldPth"

# Set testing filesystem root
export FS_ROOT
FS_ROOT=$(mktemp -d /tmp/testing-fs-root-XXXXXX)

export RUST_BACKTRACE=1

# Build the all the hab binaries so that we can run integration tests
if [[ "$scope" == "sup" ]]; then
cargo build
fi

echo "--- Running cargo test with scope '$scope' and args '$*'"

if [[ -n ${component:-} ]]; then
cd "components/$component"
fi

# We do not have any packages in the stable channel for aarch64 and probably never will.
# Set the HAB_INTERPRETER_IDENT to point to LTS-2024 to proceed with the tests.
export HAB_INTERPRETER_IDENT="core/busybox-static/1.36.1/20240805133911"

# Always add `--quiet` to avoid the noise of compilation in test output.
# The invocation to this script can add `--format pretty` to the test runner
# args (that is, after --, like --nocapture and --test-threads) if the names
# of the tests being run is desired in the output.
cargo test --quiet "$@"
Loading
Loading