Burmilla v1.9.1 ISO's console boot up takes a very long time

[h8liu]

Burmilla v1.9.1 ISO's console boot up takes a very long time, about 5+ minutes.

The console stops at message "BurmillaOS v1.9.1 started.", and silently it will be stuck there for about 5+ minutes.

I am not sure what it is waiting on.

This can be reproduced by using a VirtualBox VM that boots the ISO. non-ISO based booting (e.g. boot from hard drive after installation) does not seem to have this same issue.

The next message on dmesg is "random: crng init done". maybe it is initializing the random number generator? It feels an unexpected long time for that.

Another guess would be it is downloading something, but even for that 5+ minutes seems too long.

[h8liu]

okay, I tried it again. It actually took about 1 min or so, but one has to press some key to call out the console prompt.

[ec-c]

Why is this issue closed? It's still an issue for me.

[h8liu]

reopening then.

it seems that I need to press an "Enter" or something, and eventually the prompt will show up. sometimes it is rather fast, sometimes it is not.

if there is no key press, the prompt does not come out, and it is unclear when it starts accepting key press.

[olljanat]

Like commented on #50 (comment) boot process is asynchronous which why it depends on platform which order those gets loaded. Especially when booting from ISO.

However as BurmillaOS is based on RancherOS so I would like understand if there is measurable difference between those too? Especially compared to RancherOS with Debian console.

[h8liu]

rancheros v1.5.6 's official iso does not have this issue. neither does v1.5.8

I do not think those are using the debian console though. maybe the console is the issue?

and for this issue, I do not see the burmilla splash (the BurmillaOS ascii art). it is more like the console takes a much longer time than rancher to get ready for use.

[olljanat]

RancherOS defaults to that super tiny and completely useless console called for default. To simplify maintenance and having to more useful toolset on console we decided to use Debian as baseline and drop other on #9

If you want test RancherOS boot with Debian console it should be possible by giving kernel parameter

rancher.console=debian

on boot menu.

And to test exactly same way you need generate custom iso with cached Debian console https://rancher.com/docs/os/v1.x/en/installation/custom-builds/custom-rancheros-iso/#building-with-a-different-console

[h8liu]

hmm... from my point of view, the current console in the burmilla's iso no longer has utilities like parted, mkdosfs, which can be important for hacking around the install procedure.. so it is even more useless in this context at least for me..

also note that the current console has a fairly non-trivial .bashrc, but does not have a .bash_profile file, so the .bashrc is not executed in most cases..

[olljanat]

hmm... from my point of view, the current console in the burmilla's iso no longer has utilities like parted, mkdosfs, which can be important for hacking around the install procedure.. so it is even more useless in this context at least for me..

Yea this is the second reason why I wanted to lock to just one console. On RancherOS there is six different console options and each of those have different tooling/features/issues.

Now when we have just one we can fix those just once one. Feel free to open up PR about adding those.

also note that the current console has a fairly non-trivial .bashrc, but does not have a .bash_profile file, so the .bashrc is not executed in most cases..

Those was added on #31 to enable colors to console. At least on my envs those works fine no matter if I login from console or using SSH. Improvements are welcome.

[h8liu]

oh, there is the .profile file, so my claim was wrong. sorry.

will look into possible PRs.

still needs more investigation on why the long delay on showing the prompt though.

[olljanat]

Sure but would be good if someone can investigate if that was issue already on RancherOS with Debian console or if that was created by switching to Debian buster.

[olljanat]

I have been playing a lot with QEMU now and I haven't seen this issue on there so looks like virtual box related issue.

Can you try boot on virtual box using debug option from boot menu to see if that will give us more details about where it is stuck?

[h8liu]

this issue also occurs on bare metal. it is not only virtual box related.

[olljanat]

Hmm. Looks that I managed to duplicate it also iPXE boot with rancher.debug=true kernel parameter:

However it would be nice if you can check if debug messages are same on virtual box/bare metal when it gets stuck?

[h8liu]

I see that stuff is stuck at Calling GET /containers/console/json I think it is trying to start the console container?

[wonleing]

I used rancher.debug=true. And that's all related output in /var/log/message:
May 27 07:11:39 rancher kernel: [ 1.817617] usb 1-1: SerialNumber: 42
May 27 07:11:39 rancher kernel: [ 38.647389] random: crng init done
May 27 07:11:39 rancher kernel: [ 39.116214] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
May 27 07:11:39 rancher kernel: [ 41.143686] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
May 27 07:11:39 rancher kernel: [ 41.145063] Bridge firewalling registered
May 27 07:11:39 rancher kernel: [ 41.202602] Initializing XFRM netlink socket
May 27 07:11:39 rancher kernel: [ 41.214327] IPv6: ADDRCONF(NETDEV_UP): docker-sys: link is not ready
May 27 07:11:39 rancher kernel: [ 60.545162] cgroup: system-docker-r (354) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
May 27 07:11:39 rancher kernel: [ 60.545819] cgroup: "memory" requires setting use_hierarchy to 1 on the root
May 27 07:11:39 rancher acpid: starting up with netlink and the input layer
May 27 07:11:39 rancher acpid: 2 rules loaded
May 27 07:11:39 rancher acpid: waiting for events: event logging is off
crng init and cgroup take too much time?

[olljanat]

crng init and cgroup take too much time?

crng is part of os-base which is build using buildroot. v1.9.x versions are based on very old buildroot 2018.02.11 and it cannot be updated because newer versions of it does not support 4.14.x kernels.

However if that it the case then problem is most probably already fixed on v2.0.0-beta4 which is based on buildroot 2020.02.8 and most probably contains crng corrections too.

[olljanat]

Btw. I eventually figured out how to build kernel 4.14.x compatible OS base with latest Buildroot (https://github.com/burmilla/os-base/tree/kernel-4.14.x and https://github.com/burmilla/os-initrd-base/tree/kernel-4.14.x) so v1.9.2-rc3 is now based on Buildroot 2021.02.2. How it looks is that any better? (and especially not worse than earlier)?