Yet more debloating

[MaxKellermann]

With a few tricks, this eliminates the target build of Perl and Python completely. They were built, but not actually installed, because unused secondary packages of some recipes used them. This saves a lot of build time.
Lots of libraries are also omitted from the image, shrinking it again by a few megabytes: GnuTLS (everything's OpenSSL now), libxml2, libbz2, liblzo

[kedder]

I need to make sure ovshell/compan still builds and runs after this. I feel there's quite a lot of borderline changes that need actual testing on actual hardware. I'll try to do some tests on my testing rig.

[MaxKellermann]

What is compan?
I don't think requiring ovshell compatibility for PRs is a good policy, until ovshell is actually integrated (there isn't even a recipe, so nobody but you can test). If anything breaks ovshell, this one change can always be reverted as soon as ovshell gets integrated.

[linuxianer99]

I agree with @MaxKellermann
@kedder : What about bringing your tools officially to Openvario ?
via recipe etc ... and replace the "dusty" ovmenu-ng ??

[kedder]

What is compan?

A competition manager - the app to help with downloading tasks and competition files.

I don't think requiring ovshell compatibility for PRs is a good policy

It is not a policy. What I'd like to ensure that it is still possible to write custom software and run it on OV after it is stripped down. The ovshell/compman is a kind of test rig for that. I'm ok to adapt to these changes and make more packages to be built as part of my PPA build (including the python itself), but I want to make sure it is still possible to do.

IOW, I don't want to completely shut down the possibility to write and run programs for OV.

[kedder]

What about bringing your tools officially to Openvario

I know at least one missing feature that a some users request - simpler way to transfer files to/from OV. So I wanted to implement that, but I have troubles dedicating enough time for this.

There's also some opposition from @MaxKellermann now, because python and friends is a pretty fat dependency that will make an image significantly bigger.

[MaxKellermann]

There's also some opposition from @MaxKellermann now, because python and friends is a pretty fat dependency that will make an image significantly bigger.

My opinion doesn't count much here, I'll never fly with an OpenVario, hell, most likely I'll never fly a glider again, so why would I care. It's your project, you can do anything you want.

But I think writing the UI in Python is the worst possible choice, and I think using a text console UI toolkit is the worst possible choice. Using the text console for that ugly ovmenu shell script was a sane choice to solve the problem very very quickly with the least effort, and while switching from Shell to Python makes development somewhat better at the next stage, it solves no user-facing problem. Terminal-based UIs look horrible and their usability is horrible. This makes no sense.

About Python, yes, enabling it adds roughly 50% to the OpenVario image size. That alone would be reason enough for not using it. This is not some server or desktop machine which needs Python anyway, where using Python adds nothing to the installation requirements, and you can do some quick Python hacks.

Not only is Python a huge dependency, it also requires installing another UI toolkit. As I said above, you chose a text console toolkit, which "only" weighs another 1 MB; a GUI toolkit like Qt would weigh tens of megabytes. Not acceptable for me.

Also, doing stuff in Python doesn't allow sharing code between XCSoar and that frontend. Look at KoboMenu: it can reuse lots of code from XCSoar, it even uses XCSoar's GUI toolkit so the user sees a consistent graphical user interface with a consistent look & feel.

Now look at competition manager - why is this not a feature of XCSoar? There is nothing OpenVario-specific about compman - if you had developed this in C++ as part of XCSoar, all XCSoar users would benefit, not just OpenVario users. That would explode your audience from a few dozen (how many OpenVarios are there in the world?) to more than a hundred thousand. Think about ROI - how much usefulness you'd get in return for your noteworthy (free) time investment.

Keeping it in Python would severely restrict its reusability - being part of XCSoar would be a huge gain for both projects.

Same goes for ovshell - if it were developed in C++, you could reuse some KoboMenu code, and at the same time, KoboMenu would benefit from ovshell improvements. Maybe some parts could be integrated in XCSoar, so XCSoar gets a frontend for configuring Wifi, so you can manage Wifi access without having to quit XCSoar. Let the code be vibrant and float between all (frontend) components of XCSoar.

With ovshell and compman, you seem ot repeat the mistakes that were made when a lot of time was wasted with developing variod, which reimplemented XCSoar's audio vario code (without knowing it existed). Getting OpenVario to a point where variod can be replaced by XCSoar requires yet more work, and a lot of care needs to be taken to avoid regressions for existing users - wasting more time unnecessarily.

Of course, that's just my opinion, and you can have yours, and only you can decide how you spend your free time. You can waste time as much as you want, or you can decide that I'm completely wrong and ovshell/compman with your architecture are the right way to go, and the best choice for OpenVario. I'll disagree but ... you can just ignore that.

[MaxKellermann]

Check #199 for my idea of how a main menu written in C++ looks like. Source code: https://github.com/XCSoar/XCSoar/blob/master/src/OV/OpenVarioMenu.cpp

With this program, the image gets smaller because ovmenu-ng was the only package depending on bash, and so bash will be omitted - and bash is much larger than OpenVarioMenu! (even though OpenVarioMenu contains a static copy of the XCSoar GUI toolkit):

-rwxr-xr-x    1 root     root       1095960 Jan 19 08:00 /bin/bash.bash
-rwxr-xr-x    1 root     root        578876 Jan 20 22:41 /usr/bin/OpenVarioMenu

[linuxianer99]

@kedder I suggest merging this PR, so we can go further with the repo structure change.

You have to install your menu via IPK anyway, so you can also install the required dependencies, right ?
Regardless of the decision python-based menu or XCSoar GUI toolkit based, we should add the dependencies to the according menu recipe ...

[August2111]

I totally agree with Max's suggestion: I don't like the approach of having another 'smart' UI next to XCSoar, I think we're getting too bogged down here. Many of the things that Kedder covers with his Python programs would be much better off in XCSoar, and to be honest I have to admit that that's exactly why I haven't recommended the new shell to anyone in my club - beautiful as it is. In my opinion only minimal work should be necessary in a shell - and not the daily one.

[kedder]

@MaxKellermann just for the sake of discussion, not trying to convince anyone.

I chose python and text UI for these reasons:

1. I wanted quick results and quick turnaround time
2. Make it easily extensible by me or someone else
3. Ability to have responsive, friendly UI, that can be controlled with cursor keys
4. I'm familiar with Python and not with C++.

In retrospect, if I would start from scratch today, I'd pick python and text-based UI again. C++ is definitely out of the question because of 1, 2 and 3. Graphical UI brings a lot of unnecessary complexity for little benefit in case of OV (i.e. in conflict with 1 and 2 again).

I didn't know anything about KoboMenu, but even if I did, I would not seriously consider it. All I can find about it is a .cpp file in XCSoar repo. The idea of having to convince XCSoar devs that the code base needs a WIFI management module doesn't entertain me at all :)

The last thing I care about is the some extra hundred MB that would be taken on the SDcard. There is no shortage of space on todays sd cards and sd cards are useful to hold information, not free space.

[kedder]

@kedder I suggest merging this PR

@linuxianer99, unfortunately I can't run the current image (#204). The current hardknott branch is basically broken and doesn't boot. Do you think we should fix that before accepting more changes?

[MaxKellermann]

Do you think we should fix that

Yes.

before accepting more changes?

No. What would be the point of that?

[MaxKellermann]

Graphical UI brings a lot of unnecessary complexity

What really brings a lot of unnecessary complexity is Python.

A GUI menu just adds "more of the same" of what's already there (i.e. XCSoar) - no additional complexity.

The idea of having to convince XCSoar devs that the code base needs a WIFI management module

Already exists. https://github.com/XCSoar/XCSoar/blob/master/src/Kobo/WifiDialog.cpp

[Scumi]

Graphical UI brings a lot of unnecessary complexity

What really brings a lot of unnecessary complexity is Python.

A GUI menu just adds "more of the same" of what's already there (i.e. XCSoar) - no additional complexity.

The idea of having to convince XCSoar devs that the code base needs a WIFI management module

Already exists. https://github.com/XCSoar/XCSoar/blob/master/src/Kobo/WifiDialog.cpp

That's what I was just about to comment. From the code it appears that there is the option to connect to Wifi already and it potentially can use XCSoars Keyboard or HighScore Style (very useful for rotary encoders) to enter passwords and such. So you could omit using a Keyboard altogether with this menu. OV specific things could still be added.

Maybe it is a "flavor"-thing. I love Kedders menu implementation but having a fully integrated XCSoar-style menu would be nice as well, especially for users that have little background in OS other than just clicking buttons.

[kedder]

before accepting more changes?

No. What would be the point of that?

The point is to be able to test the proposed changes.

[DanD222]

The way I look at things, I consider no. 4 as the most important issue - who’s going to do it? Is @kedder going to re-implement the features currently afforded by OV shell/compman in C++? I’m not sure. Is kedder going to do further development of new features (I would love an easy way to copy files from a USB stick to the XCSoar home folder) in a C++ -based solution, if he’s not familiar with the language? I’m not so sure either.

[August2111]

I think @Scumi summed up my points quite well!
However, a lot of our users have already switched to @kedder's Python scripts - for this reason I would still install these shell scripts (and of course Python) for 2022 - but clearly indicate them as deprecated, so there would also be some time to fully integrate this functionality in XCSoar to install! Maybe with a splitting branch...

[kedder]

Just to make it clear: python doesn't strictly have to be part of the base image to make ovshell installable. So don't need to mark anything "deprecated". What I'm worried about is reconfiguration of lots of packages to remove various features from them. That would be much more annoying to fix, if ovshell depends on these removed features in some way.

[MaxKellermann]

What I'm worried about is reconfiguration of lots of packages to remove various features from them. That would be much more annoying to fix, if ovshell depends on these removed features in some way.

Agree, that shouldn't happen. And if it happens, we can fix it easily.

Most of this PR is about reducing the OpenVario build time, not the resulting image size. This doesn't remove Perl and Python from the image; they weren't part of the image previously. But they were built, to satisfy dependencies of packages which were not installed. Save the planet, don't waste electricity for compiling stuff you'll never use!

[kedder]

Hm, I actually can't build this branch. The build fails on "configure" stage of "itstool-native" package with error boiling down to

| checking whether /workdir/build/tmp/work/x86_64-linux/itstool-native/2.0.6-r0/recipe-sysroot-native/usr/bin/python3-native/python3 version is >= 2.6... yes
| checking for /workdir/build/tmp/work/x86_64-linux/itstool-native/2.0.6-r0/recipe-sysroot-native/usr/bin/python3-native/python3 version... 3.9
| checking for /workdir/build/tmp/work/x86_64-linux/itstool-native/2.0.6-r0/recipe-sysroot-native/usr/bin/python3-native/python3 platform... linux
| checking for /workdir/build/tmp/work/x86_64-linux/itstool-native/2.0.6-r0/recipe-sysroot-native/usr/bin/python3-native/python3 script directory... ${libdir}/python3.9/site-packages
| checking for /workdir/build/tmp/work/x86_64-linux/itstool-native/2.0.6-r0/recipe-sysroot-native/usr/bin/python3-native/python3 extension module directory... ${libdir}/python3.9/site-packages
| checking for python module libxml2... not found
| configure: error: Python module libxml2 is needed to run this package

[MaxKellermann]

Here, itstool isn't built at all, because it isn't needed. Can you see which recipe depends on it?

[MaxKellermann]

If I had to guess, I think it's shared-mime-info which depends on itstool. But I disabled the build of shared-mime-info - so what are you really building, OpenVario or are you building your private OpenVario fork?

[kedder]

@MaxKellermann I'm not sure I'm doing it correctly, but this is what I've got: I'm generating a dependency graph with

bitbake -g openvario-image

Then grepping for dependencies of itstool:

$ cat build/task-depends.dot | grep '\-> "itstool'
"itstool-native.do_build" -> "itstool-native.do_populate_lic"
"itstool-native.do_build" -> "itstool-native.do_populate_sysroot"
"itstool-native.do_compile" -> "itstool-native.do_configure"
"itstool-native.do_configure" -> "itstool-native.do_patch"
"itstool-native.do_configure" -> "itstool-native.do_prepare_recipe_sysroot"
"itstool-native.do_install" -> "itstool-native.do_compile"
"itstool-native.do_patch" -> "itstool-native.do_unpack"
"itstool-native.do_populate_lic" -> "itstool-native.do_patch"
"itstool-native.do_populate_sysroot" -> "itstool-native.do_install"
"itstool-native.do_prepare_recipe_sysroot" -> "itstool-native.do_fetch"
"itstool-native.do_unpack" -> "itstool-native.do_fetch"
"openvario-image.do_build" -> "itstool-native.do_build"
"openvario-image.do_populate_lic_deploy" -> "itstool-native.do_populate_lic"
"openvario-image.do_rootfs" -> "itstool-native.do_populate_lic"
"shared-mime-info-native.do_prepare_recipe_sysroot" -> "itstool-native.do_populate_sysroot"

Not sure how openvario-image directly depends on it.

I disabled my ppa layer for this build, here are my bblayers.conf:

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /workdir/poky/meta \
  /workdir/poky/meta-poky \
  /workdir/poky/meta-yocto-bsp \
  /workdir/poky/meta-sunxi \
  /workdir/poky/meta-openembedded/meta-oe \
  /workdir/poky/meta-openembedded/meta-python \
  /workdir/poky/meta-openembedded/meta-networking \
  /workdir/poky/meta-openvario \
  "

[MaxKellermann]

Not sure how openvario-image directly depends on it.

It does not - this is a quirk of BitBake's dependency graph feature which makes it is very cumbersome to use.
It shows indirect dependencies, but only because the "license" image needs to include the license of all recipes which were used during the build, even if no part of those recipes is actually included in theimage.

Your output shows that it's indeed shared-mime-info-native. Now who needs that?

[kedder]

Dependencies go back to psplash in this chain:

itstool-native <- shared-mime-info-native <- gdk-pixbuf-native <- psplash

I think this makes sense, because I see psplash on boot, so it must be legitimately used in the image

[MaxKellermann]

itstool-native <- shared-mime-info-native <- gdk-pixbuf-native <- psplash

D'oh! Of course! That's my mistake, sorry. You were using this PR branch, but after my last push, I added another commit to fix exactly that problem, but did not push it. Will do right away.

[MaxKellermann]

I just tested openvario-7-PQ070 on my OpenVario, and it appears to work.

[kedder]

Ok, I've finally got the image built.

One regression that I noticed (not sure if this is result of this MR or previous ones), but opkg update doesn't work anymore. It fails with this:

# opkg update
Downloading https://ftp.openvario.org/feed-hardknott/all/Packages.gz.
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer
 * opkg_download_backend: Failed to download https://ftp.openvario.org/feed-hardknott/all/Packages.gz, wget returned 1.
...

Even though this url returns 404 right now, wget can't read other https urls from ftp.openvario.org:

# wget https://ftp.openvario.org/ug_ov.pdf
Connecting to ftp.openvario.org (144.76.75.243:443)
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 40): handshake failure
wget: error getting response: Connection reset by peer

[MaxKellermann]

That's caused by ac2058d - I'll try building busybox with OpenSSL.

[August2111]

Unfortunataly up to now feed-hardknott is non-existing directory, only feed-warrior is existing - and last filled in May 2020 ;-(

[MaxKellermann]

With CONFIG_FEATURE_WGET_OPENSSL, it works:

root@ov-cubieboard:~# wget https://ftp.openvario.org/ug_ov.pdf       
Connecting to ftp.openvario.org (144.76.75.243:443)
saving to 'ug_ov.pdf'
'ug_ov.pdf' saved

[August2111]

FTP update directory feed-hardknott:
I have no write access to this FTP, otherwise I would have tried to keep it up to date. Will this be filled automatically? But then it hasn't worked for almost 2 years...

[kedder]

Ok, rebuilt the image and tried again - everything seems to work! Python packages were built as part of my PPA build, no problems with that. Didn't notice any further regressions, I'll just need to make ovshell to be a systemd service now.

[kedder]

@August2111 I believe the ftp archive was last published on the last official image we released. AFAIK ftp.openvario.org is not updated between image releases, but people who used openvario-shell are subscribed to my PPA feed, that gets updates regularly, so they received new releases of xcsoar and ovshell/compan through it.

[August2111]

@kedder The last 'official' release was the 21118 in May 2021 (see https://github.com/Openvario/meta-openvario/releases/tag/21118) but the newest update at ftp.openvario.org was at 2020-04-18!
Nice that this worked with ovshell/compman - but for the 'normal' user with ovmenu-ng this didn't work!

[August2111]

Hopefully all this stuff is working at the end of this story - and for that #197 was an additional big step, I think!

[mihu-ov]

@linuxianer99 and probably myself have write access to ftp.openvario.org
If we need additional access I am sure Andi (openvario.org admin) can arrange it. Just sent him an invitation over github.

[linuxianer99]

@kedder The last 'official' release was the 21118 in May 2021 (see https://github.com/Openvario/meta-openvario/releases/tag/21118) but the newest update at ftp.openvario.org was at 2020-04-18! Nice that this worked with ovshell/compman - but for the 'normal' user with ovmenu-ng this didn't work!

Please keep in mind, that we do this all in out spare time !!!!!!!!!!!!!