Bookworm migration for sonic-mgmt-framework and sonic-gnmi dockers #18548

ranjinidn · 2024-04-03T00:53:51Z

Why I did it

The base OS image has already migrated to bookwork. The current sonic-mgmt-framework is still on buster and sonic-gnmi on bullseye. The changes are made to migrate the sonic-mgmt-framework and sonic-gnmi dockers to bookworm build.

Work item tracking

Microsoft ADO (number only):

How I did it

Changed the docker manifest files to do the build for bookwork distro. The dependency packages are updated accordingly.

How to verify it

Built the mgmt-framework and gnmi dockers and executed the azure pipeline unit tests locally.

Note: The sonic-mgmt-framework build for bookworm has a dependency on the the below PR for the build to go through
sonic-net/sonic-mgmt-common#133
sonic-net/sonic-mgmt-framework#132

The sonic-gnmi build for bookworm has a dependency on the the below PR for the build to go through
sonic-net/sonic-gnmi#216

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

k-v1 · 2024-04-03T14:23:14Z

dockers/docker-sonic-mgmt-framework/Dockerfile.j2

                 grpcio==1.58.0 \
-                 grpcio-tools==1.20.0 \
+                 grpcio-tools==1.58.0 \
                 certifi==2017.4.17 \
                 python-dateutil==2.6.0 \
                 six==1.11.0 \


Do we still need these packages for docker-sonic-mgmt-framework?

connexion python-dateutil six outdated version of certifi

Addressed. Removed unnecessary packages.

…manifest file

…ub.com:ranjinidn/sonic-buildimage into bookworm_migration_for_mgmt_framework_and_gnmi

…backend

…gnmi

…ub.com:ranjinidn/sonic-buildimage into bookworm_migration_for_mgmt_framework_and_gnmi

…gnmi

saiarcot895 · 2024-04-24T03:17:57Z

Would you be able to handle the telemetry docker in this PR as well?

ranjinidn · 2024-04-24T21:02:13Z

Would you be able to handle the telemetry docker in this PR as well?

Yes we will be able to migrate both gnmi and mgmt-framework together as sonic-mgmt-common repo is common library used by both and the bookworm migration changes are already merged to master. We are waiting on the sonic-gnmi repo changes for bookworm migration to be merged. (sonic-net/sonic-gnmi#216)

Regarding the build failure in this PR we find a few open PRs have the same build failure reason. Hence we think the build failure is not related to the changes in this PR. How do we get past this so that this PR can be merged?

…gnmi

saiarcot895 · 2024-04-29T20:09:53Z

That build failure is now fixed. Looks like there's a possibly new failure for marvell-armhf:

python scripts/klish_preproc_cmdtree.py /sonic/src/sonic-mgmt-framework/build/target/command-tree /sonic/src/sonic-mgmt-framework/build/target/cli-macro 6 
make[4]: python: No such file or directory
make[4]: *** [Makefile:38: /sonic/src/sonic-mgmt-framework/build/target/command-tree/.done] Error 127
make[4]: Leaving directory '/sonic/src/sonic-mgmt-framework/CLI/clitree'
make[3]: *** [Makefile:39: all] Error 2
make[3]: Leaving directory '/sonic/src/sonic-mgmt-framework/CLI'
make[2]: *** [Makefile:50: cli] Error 2
make[2]: *** Waiting for unfinished jobs....

This might need a change in sonic-slave-bookworm to add the python-is-python3 for all architectures, instead of just amd64 and arm64.

ranjinidn · 2024-05-02T17:29:58Z

@saiarcot895 we are facing some build failures and this is blocking us to test further.
The following PRs are opened to resolve the build dependencies on mgmt-framework and gnmi.
sonic-net/sonic-mgmt-framework#141
sonic-net/sonic-gnmi#224
However for the gnmi PR apart from build failure some pipeline Test failures are also seen. This is in discussion with the appropriate teams to resolve. We are not sure the timeline for the above PRs to be resolved.

In the meanwhile if the submodule update for mgmt-common is reverted (PR below) we can resolve the build issue for sonic-buildimage.
#18825

Please can you do the needful so we can get this PR merged for May release.

Also we tried using the python-is-python3 both in dockers/docker-sonic-mgmt-framework/Dockerfile.j2 and sonic-slave-bookworm/Dockerfile.j2 But it was still failing for armhf with the same python error. Any pointers will be helpful.

amrutasali · 2024-05-03T00:55:21Z

@saiarcot895 an update on above comment

sonic-net/sonic-gnmi#224 has been updated to take care of pipeline test cases too.

So instead of reverting the submodule update for mgmt-common (PR) the build issue for sonic-buildimage can be resolved by merging the following and making submodule update for these 2 repos :
sonic-net/sonic-mgmt-framework#141
sonic-net/sonic-gnmi#224

saiarcot895 · 2024-05-03T01:10:04Z

Got it, thanks @ranjinidn and @amrutasali!

Reverts of submodule updates are a bit difficult to do so given current automation, so it's better to make new commits in the submodules to fix the issue.

saiarcot895 · 2024-05-03T01:14:16Z

As for the armhf build error, it looks like the errors are in sonic-mgmt-framework and sonic-gnmi, both of which are related to go.mod and go.sum, so I'm thinking they would get handled by the above commits.

…gnmi

ranjinidn · 2024-05-09T00:32:51Z

@saiarcot895 The builds are passing but see some failures in the kvmtests. Not very clear what is the error. Will you please help retrigger this / understand these errors.

ranjinidn · 2024-05-09T18:38:53Z

Thanks Sai for retriggering the tests, we still see a couple of kvmtest timeouts.
If you think this is not an issue and are random failures seen due to the setup environment, can you merge this PR?
The below 3 PRs for the individual repos are already merged for bookworm migration.
sonic-net/sonic-mgmt-common#133
sonic-net/sonic-mgmt-framework#132
sonic-net/sonic-gnmi#216

saiarcot895 · 2024-05-09T18:42:45Z

I can't approve/request a merge of this PR until all the checks are passing (unless there's a justifiable reason for the checks to fail).

It looks like the current timeouts are because it failed to lock a testbed in time, rather than any specific failure during the test execution. I've requested another rerun of those two test suites.

…gnmi

ranjinidn · 2024-05-10T17:26:08Z

@saiarcot895 we have seen the kvm tests failing due to "Acquire testbed timeout" a few times and have been rebasing our branch to master to trigger a new build. Please can you let us know if there is a way in which we can retrigger the failing cases only. Looks like we don't have permission to retrigger only the failing test. Else each retrigger we will have to reach out to you.
It is important to push these changes for the May release as the mgmt-framwork and gnmi repos are already migrated for bookworm as mentioned earlier. Without this change the docker build for mgmt-framework and gnmi is broken currently.

Update: @saiarcot895 All builds and tests have passed. Now this can be merged so the build works for the above mentioned dockers.

lguohan · 2024-05-12T09:27:01Z

sonic-slave-bookworm/Dockerfile.j2

 # Install Bazel build system (amd64 and arm64 architectures are supported using this method)
 # TODO(PINS): Remove once pre-build Bazel binaries are available for armhf (armv7l)
 {%- if CONFIGURED_ARCH == "amd64" or CONFIGURED_ARCH == "arm64" %}
 ARG bazelisk_url=https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-{{ CONFIGURED_ARCH }}
 RUN curl -fsSL -o /usr/local/bin/bazel ${bazelisk_url} && chmod 755 /usr/local/bin/bazel
 # Bazel requires "python"
 # TODO(PINS): remove when Bazel is okay with "python3" binary name
-RUN apt install -y python-is-python3
+#RUN apt install -y python-is-python3


please remove it.

…nt since its to be included unconditionally

ranjinidn · 2024-05-13T17:26:53Z

/azpw ms_conflict

…gnmi

ranjinidn · 2024-05-14T23:28:45Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2024-05-14T23:28:48Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2024-05-14T23:29:01Z

Azure Pipelines successfully started running 1 pipeline(s).

ranjinidn · 2024-05-15T06:09:09Z

@saiarcot895 we have a successful build for this PR. Please do merge this PR.

ranjinidn · 2024-05-15T19:12:55Z

@lguohan please review and merge this PR.

amrutasali and others added 2 commits April 2, 2024 23:56

Bookworm migration for mgmt-framework container

00aa63e

Bookworm migration for sonic-gnmi container

04c9ee9

ranjinidn requested review from qiluo-msft, xumia and lguohan as code owners April 3, 2024 00:53

k-v1 reviewed Apr 3, 2024

View reviewed changes

saiarcot895 added the Bookworm label Apr 3, 2024

amrutasali added 2 commits April 4, 2024 00:52

removed unnecessary python packages from sonic-mgmt-framework docker …

c6d7973

…manifest file

Merge branch 'bookworm_migration_for_mgmt_framework_and_gnmi' of gith…

adc3db3

…ub.com:ranjinidn/sonic-buildimage into bookworm_migration_for_mgmt_framework_and_gnmi

ranjinidn mentioned this pull request Apr 4, 2024

<WIP>Bookworm migration for all dependency packages in azurepipeline for sonic-gnmi sonic-net/sonic-gnmi#210

Closed

6 tasks

This was referenced Apr 4, 2024

Bookworm migration for all dependency packages in azurepipeline for sonic-mgmt-framework sonic-net/sonic-mgmt-framework#132

Merged

Bookworm migration for all dependency packages in azurepipeline for sonic-mgmt-common repo sonic-net/sonic-mgmt-common#133

Merged

ranjinidn mentioned this pull request Apr 11, 2024

Bookworm migration for all dependency packages in azurepipeline for sonic-gnmi sonic-net/sonic-gnmi#216

Merged

6 tasks

use curl openssl for mgmt-framework same as what is used in bookworm …

d26573a

…backend

ranjinidn closed this Apr 18, 2024

ranjinidn reopened this Apr 18, 2024

ranjinidn closed this Apr 19, 2024

ranjinidn reopened this Apr 19, 2024

ranjinidn added 2 commits April 19, 2024 17:38

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

8b248b4

…gnmi

Merge branch 'bookworm_migration_for_mgmt_framework_and_gnmi' of gith…

0650b83

…ub.com:ranjinidn/sonic-buildimage into bookworm_migration_for_mgmt_framework_and_gnmi

kwangsuk approved these changes Apr 19, 2024

View reviewed changes

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

bff4efa

…gnmi

ranjinidn force-pushed the bookworm_migration_for_mgmt_framework_and_gnmi branch from 4a79131 to bff4efa Compare April 23, 2024 23:34

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

4fbd5b7

…gnmi

saiarcot895 mentioned this pull request May 1, 2024

SONiC container upgrade to Bookworm (Debian 12) sonic-net/SONiC#1541

Closed

ranjinidn force-pushed the bookworm_migration_for_mgmt_framework_and_gnmi branch 2 times, most recently from 2a9caa8 to 4fbd5b7 Compare May 1, 2024 19:23

install python-is-python3 for all archs for bookworm

ecc1d02

ranjinidn force-pushed the bookworm_migration_for_mgmt_framework_and_gnmi branch from ded4126 to a689b08 Compare May 7, 2024 18:34

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

99ec35e

…gnmi

ranjinidn force-pushed the bookworm_migration_for_mgmt_framework_and_gnmi branch from a689b08 to 99ec35e Compare May 8, 2024 15:16

ranjinidn added 2 commits May 9, 2024 23:25

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

0f5d463

…gnmi

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

453f6c3

…gnmi

lguohan reviewed May 12, 2024

View reviewed changes

Address review comment to remove bazel 's python-is-python3 requireme…

dd83fc1

…nt since its to be included unconditionally

ranjinidn added 2 commits May 13, 2024 20:58

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

9209c7e

…gnmi

Merge branch 'master' into bookworm_migration_for_mgmt_framework_and_…

8992555

…gnmi

saiarcot895 approved these changes May 15, 2024

View reviewed changes

yxieca merged commit f4fc6ce into sonic-net:master May 16, 2024
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bookworm migration for sonic-mgmt-framework and sonic-gnmi dockers #18548

Bookworm migration for sonic-mgmt-framework and sonic-gnmi dockers #18548

ranjinidn commented Apr 3, 2024 •

edited

Loading

k-v1 Apr 3, 2024

amrutasali Apr 4, 2024

saiarcot895 commented Apr 24, 2024

ranjinidn commented Apr 24, 2024

saiarcot895 commented Apr 29, 2024

ranjinidn commented May 2, 2024

amrutasali commented May 3, 2024

saiarcot895 commented May 3, 2024

saiarcot895 commented May 3, 2024

ranjinidn commented May 9, 2024

ranjinidn commented May 9, 2024

saiarcot895 commented May 9, 2024

ranjinidn commented May 10, 2024 •

edited

Loading

lguohan May 12, 2024

amrutasali May 13, 2024

ranjinidn commented May 13, 2024

ranjinidn commented May 14, 2024

mssonicbld commented May 14, 2024

azure-pipelines bot commented May 14, 2024

ranjinidn commented May 15, 2024

ranjinidn commented May 15, 2024

Bookworm migration for sonic-mgmt-framework and sonic-gnmi dockers #18548

Bookworm migration for sonic-mgmt-framework and sonic-gnmi dockers #18548

Conversation

ranjinidn commented Apr 3, 2024 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

k-v1 Apr 3, 2024

Choose a reason for hiding this comment

amrutasali Apr 4, 2024

Choose a reason for hiding this comment

saiarcot895 commented Apr 24, 2024

ranjinidn commented Apr 24, 2024

saiarcot895 commented Apr 29, 2024

ranjinidn commented May 2, 2024

amrutasali commented May 3, 2024

saiarcot895 commented May 3, 2024

saiarcot895 commented May 3, 2024

ranjinidn commented May 9, 2024

ranjinidn commented May 9, 2024

saiarcot895 commented May 9, 2024

ranjinidn commented May 10, 2024 • edited Loading

lguohan May 12, 2024

Choose a reason for hiding this comment

amrutasali May 13, 2024

Choose a reason for hiding this comment

ranjinidn commented May 13, 2024

ranjinidn commented May 14, 2024

mssonicbld commented May 14, 2024

azure-pipelines bot commented May 14, 2024

ranjinidn commented May 15, 2024

ranjinidn commented May 15, 2024

ranjinidn commented Apr 3, 2024 •

edited

Loading

ranjinidn commented May 10, 2024 •

edited

Loading