Bookworm Base-image Upgrade #17234
Merged
Bookworm Base-image Upgrade #17234
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
11 tasks
saiarcot895
force-pushed
the
bookworm
branch
from
57d4e13
to
8272d37
Compare
FIPS packages for Bookworm are not yet available. Disable FIPS until those packages are ready. Signed-off-by: Saikrishna Arcot <[email protected]>
Changes from Bullseye slave container: * Python 2 is no longer available at all * Python 3.11 (instead of Python 3.9) * GCC 12 (instead of GCC 10) * Python ipaddr package is no longer available * OpenJDK 17 (instead of OpenJDK 11) * Remove doxygen armhf manual compilation (no longer needed) * Disable FIPS, as the FIPS binaries are currently not yet available * Install Python setuptools through Debian instead of pip * Install Python wheel through Debian instead of pip * Install Python nose through Debian instead of pip * Install Python j2cli through Debian instead of pip * Install Python pexpect through Debian instead of pip * Install Python parameterized through Debian instead of pip * Install Python pyyaml through Debian instead of pip * Install Python pyfakefs through Debian instead of pip * Install Python m2crypto through Debian instead of pip * Python pympler 1.0 (instead of 0.8) * Install Python build (as a replacement to setup.py) Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
In Bookworm's version of setuptools, direct calls to setup.py are deprecated and no longer guaranteed to work. One of the recommended commands is to use the `build` python package to build packages, and call it with `python -m build`. This, by default, builds the packages in a virtualenv to ensure that only the specified dependencies in setup.py are needed to build the package. This also extends to running tests, where directly calling `setup.py test` may not work, and the recommended alternatives are to either call `pytest` directly, or call `tox` or `nox.` More details are available at [1]. For SONiC's use case, for building python packages, we cannot build all Python packages in a virtualenv since there are dependencies that we would have built earlier, and these packages are not pushed to pypi or any package registry. (There may be a cleaner approach to this, though, but I'm not aware of it.) For this reason, the `-n` flag is added to not build the package in a virtualenv. For testing, `pytest` is now called instead of `setup.py test`. [1] https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html Signed-off-by: Saikrishna Arcot <[email protected]>
Update tabulate to 0.9.0 and ijson to 3.2.3 Signed-off-by: Saikrishna Arcot <[email protected]>
Newer versions of pip/setuptools don't support test_requires, and the current standard is to specify any extra dependencies (such as those required for testing) under extra_requires. Therefore, specify the testing dependencies under extra_requires. These can be installed via pip using `pip install '.[testing]'`. Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]> Co-authored-by: Aravind Mani <[email protected]>
This tells the build infra that they need to be built as part of Bullseye and not Bookworm. Signed-off-by: Saikrishna Arcot <[email protected]>
This ordering dependency causes FRR to get built for Bookworm, which we don't need currently. Skip this by having it apply only to Bookworm. Signed-off-by: Saikrishna Arcot <[email protected]>
Debian Bookworm no longer uses NTP, and instead uses NTPsec. Modify our files to update/replace the NTPsec files instead. Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
The help text printed for sonic-yang-mgmt has slight differences depending on the package versions. Loosen this check to only check the options themselves, rather than the surrounding text. Signed-off-by: Saikrishna Arcot <[email protected]>
Update test_cfggen_from_yang.py and test_yang_data.json to the current config_db format, and allow tests for sonic-config-engine to run for Bookworm. Also update pyangbind to 0.8.2 for Bookworm to fix an issue with some classes being moved into a different package. Signed-off-by: Saikrishna Arcot <[email protected]>
This fixes 4 issues: * Update tabulate to 0.9.0 and deepdiff to 6.2.2 * Specify test dependencies under extra_requires * Add check_output parameter to the setup function due to the patch * Fix error about having a mutable default for field headers in dataclass Signed-off-by: Saikrishna Arcot <[email protected]>
This fixes 3 issues: * Specify test dependencies under extra_requires * Update the PAM configuration for Bookworm * Break a cyclical dependency between sonic-host-services and sonic-buildimage by moving the contents of src/sonic-host-services-data into sonic-host-services submodule Signed-off-by: Saikrishna Arcot <[email protected]>
Notable changes: * Use j2cli from Debian repos instead of pip * Use setuptools from Debian repos instead of pip * Use wheel from Debian repos instead of pip * Update grpcio and grpcio-tools python packages to match version in Bookworm * Use m2crypto from Debian repos instead of pip Signed-off-by: Saikrishna Arcot <[email protected]>
Starting with Bookworm, Debian moved the non-free Linux firmware blobs into a new non-free-firmware component, since they are frequently needed by users and since they need to be updated frequently. Since the only thing we currently install from the non-free component (that I can think of) is the Linux firmware, have Bookworm use non-free-firmware instead of non-free. Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
…kernel 6.1 and bookworm (sonic-net#16954) * sonic-platform-modules-cel: broadcom: adapt for kernel 6.1 and bookworm The i2c_driver->remove API declaration has been updated to return void instead of int, as part of cleanup patches in 6.1. More details can be referred from here: [1]. Update the remove API definition in the modules accordingly and cleanup variables that go unused from the remove API. Update python build commands for bookworm. The packaging based on calling setup.py is deprecated and using build module/pip utility is the recommended method for python packaging/installation. Further details can be referred to from here: [2], [3]. The build module is picky about the package information file, which needs to be either setup.py or pyproject.toml. Additionally, fix formatting inconsistencies in debian/changelog reported by `dh_installchangelogs` during the build. Tested the changes by compiling the changes as below: make sonic-slave-bash NOBUSTER=1 NOBULLSEYE=1 sudo dpkg -i target/debs/bookworm/linux-headers-6.1.0-11-2-*.deb cd platform/broadcom/sonic-platform-modules-cel KVERSION=6.1.0-11-2-amd64 dpkg-buildpackage Also verified the python scripts under the sonic-platform-modules-cel with pyflakes to ensure no new errors are flagged (with exception of unused modules). References: [1] - torvalds/linux@ed5c2f5f [2] - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.htm [3] - 0b20a48 (Update Python build commands for Bookworm, 2023-09-07) Signed-off-by: Ramasamy Chandramouli <[email protected]> * platform/pddf: i2c: adapt for kernel 6.1 and bookworm * Fixup i2c_driver->remove API due to changes in the function prototype (ref: [1]). * Cleanup `MODULE_SUPPORTED_DEVICE` macros that were cleaned up in the upstream (ref: [2]). * Sanitize python packaging and installation using the `build` module instead of calling the setup.py directly (ref: [3]. [4]). Tested the changes by compiling pddf module as below: make sonic-slave-bash NOBUSTER=1 NOBULLSEYE=1 sudo dpkg -i target/debs/bookworm/linux-headers-6.1.0-11-2-*.deb cd platform/pddf/i2c KVERSION=6.1.0-11-2-amd64 dpkg-buildpackage References: [1] - torvalds/linux@ed5c2f5f [2] - torvalds/linux@6417f031 [2] - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.htm [3] - 0b20a48 (Update Python build commands for Bookworm, 2023-09-07) Signed-off-by: Ramasamy Chandramouli <[email protected]> * platform/broadcom: include platform-modules-cel in builds With pddf modules patched for 6.1, platform-modules-cel can be compiled and included in the final image. Testing by building sonic-broadcom.bin/sonic-broadcom-dnx.bin. Signed-off-by: Ramasamy Chandramouli <[email protected]> * pddf/i2c: revert correct rootdir for pip install The pip install directory has been set to test-pkg1/ for testing the build and incorrectly retained as is. Revert this to the correct path $(PACKAGE_PRE_NAME). Signed-off-by: Ramasamy Chandramouli <[email protected]> * platform/broadcom: include pddf/modules-cel in the base package Without this change, the modules were built but not packaged in the final .bin. The final sonic-broadcom.bin has been tested for bootup on Celestica's Silverstone platform. admin@sonic:~$ uname -a Linux sonic 6.1.0-11-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4 (2023-08-08) x86_64 GNU/Linux admin@sonic:~$ show platform summary Platform: x86_64-cel_silverstone-r0 HwSKU: Silverstone ASIC: broadcom ASIC Count: 1 Serial Number: R4009B2F062504LK200024 Model Number: N/A Hardware Revision: N/A admin@sonic:~$ show version | head SONiC Software Version: SONiC.g0aad6c67c-rachandr SONiC OS Version: 12 Distribution: Debian 12.2 Kernel: 6.1.0-11-2-amd64 Build commit: 0aad6c67c Build date: Thu Oct 26 07:13:47 UTC 2023 Built by: rachandr@AZUHPS14 Platform: x86_64-cel_silverstone-r0 Signed-off-by: Ramasamy Chandramouli <[email protected]> --------- Signed-off-by: Ramasamy Chandramouli <[email protected]>
…orm (sonic-net#16982) * [Edgecore][sonic-platform-modules-accton]Support kernel 6.1 and bookworm * Modify pddf drv code for i2c_remove_callback function fail
Signed-off-by: Saikrishna Arcot <[email protected]>
This patch enables compiling of Marvell platform module and fixes sonic-platform-nokia compilation issues for bookworm.
Rasdaemon is not installed on armhf or arm64 Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Vivek Reddy <[email protected]> [Nvidia] Enable iproute2 & fix mft build (#16) * Enable iproute2 as the SDK is also built Signed-off-by: Vivek Reddy <[email protected]> * [Nvidia] Dont use mkbmdeb method of dkms to build the package Signed-off-by: Vivek Reddy <[email protected]> * Added linux image to the Depends section of mft Signed-off-by: Vivek Reddy <[email protected]> [Nvidia] [Bookworm] Separate KERNEL_MFT into a new target (sonic-net#16782) * [Nvidia] Seperate KERNEL_MFT into a new target because of kernel header dependency Signed-off-by: Vivek Reddy <[email protected]> * Update linux-kernel submodule Signed-off-by: Vivek Reddy <[email protected]> * Fix paralell build problem Signed-off-by: Vivek Reddy <[email protected]> --------- Signed-off-by: Vivek Reddy <[email protected]>
pam-auth-update doesn't store local configuration, and it's meant to be used by packages only. Because libpam-systemd was getting uninstalled afterwards, this caused tacplus to get re-enabled. Signed-off-by: Saikrishna Arcot <[email protected]>
This fixes lldpcli hitting some error related to being unable to get a lock on /var/run/lldpd.socket. This version is the version in Debian Bookworm, even though lldpd is in the lldp container, and that is on Bullseye. This is because there is a change that went into 1.0.12 that uses a separate lock file for the lldpd socket instead of locking the socket file itself. This appears to cause problems in an unprivileged docker container for unknown reasons (privileged docker container is fine). Bullseye is on 1.0.11, which isn't new enough to have this change. I can't see any specific system capability that might address this. Rather than debugging this further, just upgrade to the Bookworm version. Signed-off-by: Saikrishna Arcot <[email protected]>
systemd changed the log message syntax for a container going down. Update the regex for the new format. Signed-off-by: Saikrishna Arcot <[email protected]>
Debian changed the defaults of the sudo package to never lecture the user when using an unauthorized sudo command, which breaks our use case of lecturing once. Add a line to lecture once, which is the old defaults. Signed-off-by: Saikrishna Arcot <[email protected]>
Signed-off-by: Saikrishna Arcot <[email protected]>
Add a note saying if running on a recent kernel, then Docker 20.10.10 or newer needs to be used. This is because in Bookworm, glibc will use the `clone3` syscall, which is not properly handled by Docker's seccomp filter in versions older than 20.10.10. Signed-off-by: Saikrishna Arcot <[email protected]>
…sonic-net#17015) extract-certs was moved from scripts/ to certs/ from kernel 5.17 gregkh/linux@340a025 Signed-off-by: Vivek Reddy <[email protected]>
* Enable SDK modules for Bookworm * Update SAI deb to 1.13.0-1 Signed-off-by: Pavan Naregundi <[email protected]>
* [bookworm] Fix docker gid mismatch with host * Use group-add arg instead of update sonic-slave user Signed-off-by: Vivek Reddy <[email protected]>
…nic-net#17134) Why I did it Add platform support for Debian 12 (Bookworm) on Mellanox Platform How I did it Update hw-management to v7.0030.2008 Deprecate the sfp_count == module_count approach in favour of asic init completion Ref: Mellanox/hw-mgmt@bf4f593 Add xxd package to base image which is required by hw-management scripts Add the non-upstream flag into linux kernel cache options Update the thermalctl logic based on new sysfs attributes Fix the integrate-mlnx-hw-mgmt script to not populate the arm64 Kconfig How to verify it Build kernel and run platform tests Signed-off-by: Vivek Reddy <[email protected]> Co-authored-by: Junchao-Mellanox <[email protected]> Co-authored-by: Junchao-Mellanox <[email protected]>
…2.2104 (sonic-net#17131) Why I did it Update SDK/SAI and FW for Mellanox Platform How I did it Update SDK/FW to v4.6.2104/v2012.2104 Fixed Issues: Some of the Warmboot related files which were created by SDK during switch create are now generated during pre shutdown flow New Features: Debian 12 and kernel 6.1 support Update SAI New Features: Auto Fec Support FDB entries are now restored after warmboot to prevent temporary system flooding. Minor Enhancement and Bug Fix in integrate-mlnx-sdk How to verify it Build Image and run tests Signed-off-by: Vivek Reddy <[email protected]>
Why I did it [Bookworm] Update platform-modules-dell for Bookworm sonic-net#16735 How I did it Modified platform driver to comply with bookworm kernel. Removed MODULE_SUPPORTED_DEVICE wherever used. Modified python build commands for building whl packages. How to verify it Verify whether all the platform bookworm debs are built. make target/debs/bookworm/platform-modules-z9100_1.1_amd64.deb Load the platform debian into the device and install it in bookworm image. Verify the platform related CLI and the functionality
saiarcot895
force-pushed
the
bookworm
branch
from
8272d37
to
9ffd1af
Compare
saiarcot895
requested review from
StormLiangMS,
qiluo-msft,
xumia and
lguohan
as code owners
|
@xumia @liushilongbuaa FYI, in case of build failures or increased build times |
|
@saiarcot895 , the HEAD of sonic-utilities is set to your own repo. diff --git a/src/sonic-utilities b/src/sonic-utilities |
|
Hi @liushilongbuaa, I opened #17297 to fix it, but it looks like some other commit that is being brought in is breaking tests (this is based on the fact that #17069 is also failing PR tests, but doesn't have my commits). Specifically, |
qiluo-msft
reviewed
Feb 21, 2024
| @@ -256,24 +256,19 @@ sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install apparmor | |||
| sudo cp files/image_config/ntp/ntp-apparmor $FILESYSTEM_ROOT/etc/apparmor.d/local/usr.sbin.ntpd | |||
| sudo LANG=C chroot $FILESYSTEM_ROOT apt-get -y install apt-transport-https \ | |||
| ca-certificates \ | |||
| curl \ | |||
| gnupg2 \ | |||
There was a problem hiding this comment.
It was being removed in line 275 below, after docker-ce was installed. This package is technically not needed for the purpose of installing docker-ce.
Gfrom2016
added a commit
to Gfrom2016/sonic-buildimage
that referenced
this pull request
Aug 28, 2024
Signed-off-by: zitingguo-ms <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
This PR upgrades SONiC's base image from Debian Bullseye to Debian Bookworm. Bookworm brings in the following notable changes:
Work item tracking
How I did it
How to verify it
KVM tests all pass, basic bootup done on a few platforms to make sure interfaces come up and BGP comes up.
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)