How to upgrade os-kernel with source code from kernel.org?

[wonleing]

BurmillaOS Version: (ros os version)
v4.14.x
Where are you running BurmillaOS? (docker-machine, AWS, GCE, baremetal, etc.)
KVM, baremetal, iot
Which processor architecture you are using?
x86
Do you use some extra hardware? (GPU, etc)?
no

I am tring to replace 4.14.x kernel with 4.19 kernel, I use os-kernel build a new one and replace the KERNEL_URL in this 'os' project. I succeeded to build the iso, but it can not start properly. The worse thing is I can not see any error log, I don't know how to debug it...

Can you give me a proper instruction on how to do it, or some effecient way of debug?

Thank you very much,
Leon

[olljanat]

You can find our kernel build files from https://github.com/burmilla/os-kernel

However notice that we have made decision that v1.9.x versions will stay on 4.14.x kernels. If you need newer version then I highly recommend that you test our 2.x latest beta version https://github.com/burmilla/os/releases/tag/v2.0.0-beta4

[wonleing]

I don't quite understand the version policy.
So If I insist to do Kernel 4.19, both v1.9.x and v2.x can support, but v2.x is the better choice. right?

[olljanat]

We are trying to use standard software versioning where those numbers indicate about how big change is.

Also we have some targets written to https://github.com/burmilla/os/projects

You can find our original kernel / os versioning discussion from #5

[newhuangchuan]

Hello, when I use the kernel code in the public network kernel.org to compile, use the public network kernel to compile the .config file to make the kernel, and then the compiled burmillaOS fails to start. May I ask what is in your code? The difference between kurnel-config and the .config file compiled by menuconfg in the kernel code

[olljanat]

@wonleing @newhuangchuan you can see our kernel configs and all build scripts on os-kernel and from releases view you can find links to exact commits.

However, can you plz explain that what you are trying to achieve by building custom kernel?

[wonleing]

@wonleing @newhuangchuan you can see our kernel configs and all build scripts on os-kernel and from releases view you can find links to exact commits.

We compared os-kernel/config/x86/kernel-config and .config that generated by 'make menuconfig' (source code is from kernel.org, same version as describe in os-kernel/config/x86/kernel-config). There are 2000+ differences.

However, can you plz explain that what you are trying to achieve by building custom kernel?

We are maintaining our own kernel source, added patchs drivers and more arches support base on kernel.org kernel.
So enventually we hope to build an iso base on our own kernel source and rootfs.

[olljanat]

We compared os-kernel/config/x86/kernel-config and .config that generated by 'make menuconfig' (source code is from kernel.org, same version as describe in os-kernel/config/x86/kernel-config). There are 2000+ differences.

ah, true because we have heavily disabled non-used features.

Anyway, I think that setting which you are missing is CONFIG_KERNEL_XZ=y which looks to be disabled by default and which is needed because BurmillaOS uses XZ to compressed base Docker images inside of initrd:

initrd_extract/usr/share/ros$ file *
images-init.tar:   XZ compressed data
images-system.tar: XZ compressed data

We are maintaining our own kernel source, added patchs drivers and more arches support base on kernel.org kernel.
So enventually we hope to build an iso base on our own kernel source and rootfs.

OK. Are those patched kernel sources available somewhere where I can see them? Also do you have patches version of QEMU which can be used to emulate those archs (if they are not included to it by default)?

We have one similar case open on #23 where we noticed that porting BurmillaOS to another archs is quite tricky on current architecture which why I have been doing prototyping about new one on https://github.com/burmilla/os-base-new but currently that one uses rootfs from Debian so if you are looking for archs which are not supported by it that need to be solved first.

[wonleing]

Are those patched kernel sources available somewhere where I can see them?

unfortunatelly no... it is not controlled by me. And we use real machine instead of QEMU for other arches.

because we have heavily disabled non-used features

So the question is how to generate a new config/x86/kernel-config file when I doing kernel upgrade to 4.19.x or higher? any policy or step or tool? I suppose the 4.14.x kernel-config does not quite suite.

#23 where we noticed that porting BurmillaOS to another archs.

I know the asker of that thread :) Indeed we are trying to do similar thing. And we have already had rootfs of Debian to support that arch.
About system and user docker, can we just use the binary that built by official docker-ce source code (of course added arch support)? Or is there any other tricky change?
We already built out all V19.03.8 docker-ce/docker-ce-cli for all arches.

https://github.com/burmilla/os-base-new

This is awesome. We are trying to use it.

[wonleing]

I found the kernel source and qemu for mips64.
http://ftp.loongnix.org/toolchain/qemu/2015-OS/qemu-1.4.tgz
http://ftp.loongnix.org/toolchain/qemu/2015-OS/linux-3.10.84.tgz
http://ftp.loongnix.org/toolchain/qemu/2015-OS/run-linux.tgz

They are quite old, don't know if it makes any help...

[olljanat]

So the question is how to generate a new config/x86/kernel-config file when I doing kernel upgrade to 4.19.x or higher? any policy or step or tool?

On theory this is only setting which you need to update to default kernel config:

CONFIG_KERNEL_XZ=y

Alternatively you can change this line

os/scripts/tar-images

Line 53 in 5a13f03

ARCHIVE_CMD="xz -4 -e"

to

ARCHIVE_CMD="gzip"

before you generate ISO file. Then standard config from kernel.org should works.

[olljanat]

@wonleing btw. Is It "LoongArch" support which you are trying to achieve or some other arch?

[wonleing]

@wonleing btw. Is It "LoongArch" support which you are trying to achieve or some other arch?

we focus on these arches: amd64, arm64, mips64, loongarch64 and sw64(Alpha)

[wonleing]

Now we are able to support standard kernel.org 4.19.90 kernel and buildroot-2021.02.1
Can we join as burmillar project maintainer/developer, so we can sync these update to the community?

[olljanat]

Can we join as burmillar project maintainer/developer, so we can sync these update to the community?

Yes we are looking for maintainers for this project #2 and there is still room for more persons. Also it would be nice if you can share more details about your use case(s) on #6 Especially about that which kind of workloads you are planning to run on top of BurmillaOS after you get in working on all those archs? Also it would be interesting to know that why you want to do it with BurmillaOS instead of some other OS?

[wonleing]

Also it would be interesting to know that why you want to do it with BurmillaOS instead of some other OS

ContainerOS is much smaller and lighter than common linux distribution while still support containers runtime. So this is paticular fit for VM guestOS and IOT devices when they want to use containerized micro services.
Compare to Fedora CoreOS, BurmillarOS is much smaller and easier to build and maintain. Also I like the idea of 'containerized system process', I think this could be a future trend somehow.

[olljanat]

Compare to Fedora CoreOS, BurmillarOS is much smaller and easier to build and maintain.

Agreed, that why I ended up to fork RancherOS.

Also I like the idea of 'containerized system process', I think this could be a future trend somehow.

Yes on theory it is good idea but current system-docker is far from optimal and we need figure out some alternative for it #28 (comment)

[olljanat]

we focus on these arches: amd64, arm64, mips64, loongarch64 and sw64(Alpha)

@wonleing btw. Which bootloader you use on mips64, loongarch64 and sw64? Is grub available on there or is there need to use some special one?

Also one thing which came to my mind is that you might want to look how our Raspberry Pi build process works as it is probably more near of your use case. For those we build kernel using scrips on https://github.com/burmilla/os-rpi-kernel and actual media gets created by scripts on https://github.com/burmilla/os/tree/master/scripts/images/raspberry-pi-hypriot64

[wonleing]

Which bootloader you use on mips64, loongarch64 and sw64?

grub. no special things here
For now we mainly build and test burmillaOS on x86_64 KVM. Trying to find the right way to build and upgrade kernel/rootfs/services with our source code (from generic OS source code). Still have many problems to resolve...
hardware related issue could be handled later

[newhuangchuan]

Have you ever encountered os-docker in the system-docker restarting state, and executing the docker command directly shows that it has not been started. Is the docker command not related to os-docker in system-docker?

[olljanat]

os-docker container run Docker engine/daemon (dockerd). docker command is Docker CLI which connects to it.

Why it is on restarting state can be investigated with sudo system-docker logs docker or looking for "/var/log/docker.log"

[newhuangchuan]

When looking at /var/log/docker.log, the display is as follows, which seems to be caused by the iptables rules, and iptables cannot create the DOCKER chain of the nat table. But there is only one iptables-xml in the system.
The error is shown in the figure below：

[olljanat]

Which version of code you are using? Error says nf_tables so you are most probably missing at least this one #84

[newhuangchuan]

The version I use is v2.0.0-beta2.
When checking the rules of iptables, there is only one iptables-xml command that can be used. After installing through apt-get, it is still an error.

[olljanat]

Hmm. Beta2 is very early draft. It would be better to take beta4 which contains a lot of corrections done after that.

[newhuangchuan]

Okay, let me switch, thank you very much.

[newhuangchuan]

Hello, we have used the branch of v1.9.x. Is the firewall he uses by default nf_tables?

[wonleing]

Hello, we have used the branch of v1.9.x. Is the firewall he uses by default nf_tables?

Please merge this patch to your code
bcaa4a1

[newhuangchuan]

Hello, how do you build user docker and system-docker in BurmillaOS?

[ToeiRei]

@newhuangchuan Look at https://github.com/burmilla/os-kernel - the documentation there should help you

[olljanat]

Hello, how do you build user docker and system-docker in BurmillaOS?

@newhuangchuan user docker uses official static binaries from https://download.docker.com/linux/static/stable/ those as packed inside of docker images using these https://github.com/burmilla/os-services/tree/master/images/10-docker-20.10.5

System-docker sources are on https://github.com/burmilla/os-system-docker and repositories referenced by it but we have not actually ever build it. I just copied binaries from https://github.com/rancher/os-system-docker/releases/tag/17.06-ros6 to our repo.

What I know for sure is that system-docker is based on Docker 17.06 and contain these customizations on engine burmilla/docker@4119920...6368e2f and these on cli burmilla/docker-cli@f6b3234...fa7c1da

Some more info about it can be found from discussion on #28

[newhuangchuan]

Hello, I would like to ask if you have ever reported a trash error when building os-system-docker here:
trash: error: no such option: -k
However, there is a trash command in the local system. which trash >> /usr/bin/trash

Usage: trash [OPTION]... FILE...

Put files in trash

Options:
  --version            show program's version number and exit
  -h, --help           show this help message and exit
  -d, --directory      ignored (for GNU rm compatibility)
  -f, --force          ignored (for GNU rm compatibility)
  -i, --interactive    ignored (for GNU rm compatibility)
  -r, -R, --recursive  ignored (for GNU rm compatibility)
  -v, --verbose        explain what is being done

To remove a file whose name starts with a '-', for example '-foo',
use one of these commands:

    trash -- -foo

    trash ./-foo

Report bugs to https://github.com/andreafrancia/trash-cli/issues
It seems that there is no -k parameter in the parameters.

There is also whether there is a way to make os-docker mirroring

[newhuangchuan]

Hello, I have built the os-services project here. May I ask, after the image under this project is built, is it used as a burmillaOS system? For example, starting an nginx service in burmillaOS should be directly accessible like other systems, right? Or must the services project be used for construction?
`Images ready to push:

rancher/os-alpineconsole:f0e2da8-dirty
rancher/os-centosconsole:f0e2da8-dirty
rancher/os-debianconsole:f0e2da8-dirty
rancher/os-fedoraconsole:f0e2da8-dirty
rancher/os-hypervvmtools:f0e2da8-dirty
rancher/os-modemmanager:f0e2da8-dirty
rancher/os-openvmtools:f0e2da8-dirty
rancher/os-pingan:f0e2da8-dirty
rancher/os-qemuguestagent:f0e2da8-dirty
rancher/os-selinuxtools:f0e2da8-dirty
rancher/os-ubuntuconsole:f0e2da8-dirty
rancher/os-waagent:f0e2da8-dirty
rancher/os-amazonmetadata:f0e2da8-dirty
rancher/os-iscsi:f0e2da8-dirty
rancher/os-nvidiadriver:f0e2da8-dirty
rancher/os-volumenetshare:f0e2da8-dirty
rancher/os-zfs:f0e2da8-dirty`

[olljanat]

Those are Docker images. os-config.tpl.yml defines services included to OS which uses those images like this:

os/os-config.tpl.yml

Lines 112 to 398 in 76b4a14

	services:
	command-volumes:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: echo
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	log_driver: json-file
	net: none
	privileged: true
	read_only: true
	volumes:
	- /usr/bin/ros:/usr/bin/ros:ro
	- /usr/bin/system-docker:/usr/bin/system-docker:ro
	- /usr/bin/system-docker-runc:/usr/bin/system-docker-runc:ro
	system-volumes:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: echo
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	log_driver: json-file
	net: none
	privileged: true
	read_only: true
	volumes:
	- /dev:/host/dev
	- /etc/docker:/etc/docker
	- /etc/hosts:/etc/hosts
	- /etc/logrotate.d:/etc/logrotate.d
	- /etc/resolv.conf:/etc/resolv.conf
	- /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt.rancher
	- /etc/selinux:/etc/selinux
	- /lib/firmware:/lib/firmware
	- /lib/modules:/lib/modules
	- /run:/run
	- /usr/share/ros:/usr/share/ros
	- /var/lib/boot2docker:/var/lib/boot2docker
	- /var/lib/rancher/cache:/var/lib/rancher/cache
	- /var/lib/rancher/conf:/var/lib/rancher/conf
	- /var/lib/rancher:/var/lib/rancher
	- /var/lib/waagent:/var/lib/waagent
	- /var/log:/var/log
	- /var/run:/var/run
	container-data-volumes:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: echo
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	log_driver: json-file
	net: none
	privileged: true
	read_only: true
	volumes:
	- /var/lib/user-docker:/var/lib/docker
	- /var/lib/m-user-docker:/var/lib/m-user-docker
	user-volumes:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: echo
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	log_driver: json-file
	net: none
	privileged: true
	read_only: true
	volumes:
	- /home:/home
	- /opt:/opt
	- /var/lib/kubelet:/var/lib/kubelet
	media-volumes:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: echo
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	log_driver: json-file
	net: none
	privileged: true
	read_only: true
	volumes:
	- /media:/media:shared
	- /mnt:/mnt:shared
	all-volumes:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: echo
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	io.docker.compose.rebuild: always
	log_driver: json-file
	net: none
	privileged: true
	read_only: true
	volumes_from:
	- container-data-volumes
	- command-volumes
	- media-volumes
	- user-volumes
	- system-volumes
	{{if eq "amd64" .ARCH -}}
	acpid:
	image: {{.OS_REPO}}/os-acpid:{{.VERSION}}{{.SUFFIX}}
	command: /usr/sbin/acpid -f
	labels:
	io.rancher.os.scope: system
	net: host
	uts: host
	privileged: true
	volumes_from:
	- command-volumes
	- system-volumes
	{{end -}}
	cloud-init-execute:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: cloud-init-execute -pre-console
	labels:
	io.rancher.os.detach: "false"
	io.rancher.os.scope: system
	io.rancher.os.after: ntp
	net: host
	uts: host
	privileged: true
	volumes_from:
	- system-volumes
	volumes:
	- /usr/bin/ros:/usr/bin/ros:ro
	console:
	image: {{.OS_REPO}}/os-console:{{.VERSION}}{{.SUFFIX}}
	command: ros console-init
	labels:
	io.rancher.os.scope: system
	io.rancher.os.after: network
	io.docker.compose.rebuild: "false"
	io.rancher.os.console: default
	environment:
	- HTTP_PROXY
	- HTTPS_PROXY
	- NO_PROXY
	net: host
	uts: host
	pid: host
	ipc: host
	privileged: true
	oom_score_adj: -100
	restart: always
	volumes_from:
	- all-volumes
	volumes:
	- /media:/media:shared
	- /mnt:/mnt:shared
	logrotate:
	image: {{.OS_REPO}}/os-logrotate:{{.VERSION}}{{.SUFFIX}}
	command: /usr/sbin/logrotate -v /etc/logrotate.conf
	labels:
	io.rancher.os.createonly: "true"
	io.rancher.os.scope: system
	io.rancher.os.before: system-cron
	cron.schedule: "@hourly"
	uts: host
	net: none
	privileged: true
	volumes_from:
	- command-volumes
	- system-volumes
	network:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: netconf
	labels:
	io.rancher.os.scope: system
	io.rancher.os.after: udev
	io.rancher.os.reloadconfig: "true"
	net: host
	uts: host
	pid: host
	privileged: true
	volumes_from:
	- system-volumes
	- command-volumes
	volumes:
	- /usr/bin/iptables:/sbin/iptables:ro
	ntp:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: /bin/start_ntp.sh
	labels:
	io.rancher.os.scope: system
	io.rancher.os.after: network
	net: host
	uts: host
	privileged: true
	restart: always
	volumes_from:
	- command-volumes
	- system-volumes
	preload-user-images:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: ros preload-images
	net: host
	labels:
	io.rancher.os.detach: "false"
	io.rancher.os.scope: system
	io.rancher.os.after: console
	privileged: true
	volumes_from:
	- command-volumes
	- system-volumes
	syslog:
	image: {{.OS_REPO}}/os-syslog:{{.VERSION}}{{.SUFFIX}}
	command: rsyslogd -n
	labels:
	io.rancher.os.scope: system
	log_driver: json-file
	net: host
	uts: host
	privileged: true
	restart: always
	volumes_from:
	- command-volumes
	- system-volumes
	system-cron:
	{{if eq "amd64" .ARCH -}}
	image: burmilla/container-crontab:v0.5.0
	{{else -}}
	image: burmilla/container-crontab:v0.5.0{{.SUFFIX}}
	{{end -}}
	labels:
	io.rancher.os.scope: system
	uts: host
	net: none
	privileged: true
	restart: always
	volumes:
	- /var/run/system-docker.sock:/var/run/docker.sock
	environment:
	DOCKER_API_VERSION: "1.22"
	udev-cold:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: ros udev-settle
	labels:
	io.rancher.os.detach: "false"
	io.rancher.os.scope: system
	net: host
	uts: host
	privileged: true
	volumes_from:
	- command-volumes
	- system-volumes
	udev:
	image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
	command: udevd
	labels:
	io.rancher.os.detach: "true"
	io.rancher.os.scope: system
	io.rancher.os.after: udev-cold
	net: host
	uts: host
	privileged: true
	restart: always
	volumes_from:
	- command-volumes
	- system-volumes
	docker:
	{{if eq "amd64" .ARCH -}}
	image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}}
	{{else -}}
	image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}}
	{{end -}}
	command: ros user-docker
	environment:
	- HTTP_PROXY
	- HTTPS_PROXY
	- NO_PROXY
	labels:
	io.rancher.os.scope: system
	io.rancher.os.after: console
	net: host
	pid: host
	ipc: host
	uts: host
	privileged: true
	restart: always
	volumes_from:
	- all-volumes
	volumes:
	- /sys:/host/sys
	- /var/lib/system-docker:/var/lib/system-docker:shared

That is same syntax than on docker-compose files but that file also contains some other OS configurations.
Extra services which can be enabled with sudo ros service ... are defined on those compose files on os-services repo.
Look: https://burmillaos.org/docs/system-services/custom-system-services/

[newhuangchuan]

Will the construction of burmillaOS depend on os-services?

[olljanat]

Only user docker image is used from os-services by default. All other system services are build from https://github.com/burmilla/os/tree/master/images

[newhuangchuan]

Hello, on the arm64 system, burmillaOS has been built, and the selected code branch v2.0.0-beta4 and feat/uefi-support are two. After the build is completed, the display is shown when starting through qemu, you know What caused this situation? Is it caused by the absence of efi files in the ISO file during the ISO building process? I mounted the image on the system and compared it with other ISO files.

[olljanat]

At least you need duplicate these modifications to arm64 file too e42fef0#diff-f05bf44a1c5840775c97074e1e62efa65149d8f6f0d0654b9b80a1b6b65b52e5

[newhuangchuan]

Hello, I have modified this part in the v2.0.0-beta4 branch, but the built one still has the problem of UEFI not being able to boot. I doubt it is caused by the absence of files required by UEFI in the image.
this is arm64 ubuntu:
boot casper dists efi install md5sum.txt pool ubuntu ubuntu-ports
this is burmillaOS:
boot rancheros

[olljanat]

Try with this guide https://github.com/mkinney/myranch/blob/master/readme.MD#create-uefi-bootable-iso-from-rancher-os-iso-from-mac and btw after you figure out needed configuration it would be better to comment it to #8

[wonleing]

Does 2.0.0-beta4 tag support uefi? I saw an uefi-support branch, we tried it, cannot boot up either.
Another question is: Does arm64 have to use uefi? Anyway to swith it to bios in arm64 KVM?

[olljanat]

No 2.0.0-beta4 does not have UEFI support yet. That is biggest reason why it is still marked as beta instead of release candidate. Arm64 can stay with BIOS only unless there is users who need UEFI support for it.

[newhuangchuan]

If some users need UEFI to boot the system for installation, how to support it?
All burmillaOS ISOs can be installed to bare metal through a USB flash drive, right?

[olljanat]

Yes all BurmillaOS ISOs can be installed from USB. UEFI installation need special partitioning which is not currently handled by ros so those need to be done manually until UEFI support is implemented to installer.