sysext: Add python sysext

[JeWe37]

sysext: Add python sysext

Minimal sysext containing dev-lang/python and dev-python/pip. Not wanting to include python in the base image makes sense, but as an optional sysext I do believe it makes sense, especially considering that a lot of configuration tooling is python based.

How to use

Inside the SDK container

./build_packages
./build_image sysext

Testing done

Installed a package via pip and checked that it could be used from python REPL.

• Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
• Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

[pothos]

It kind of makes sense because one otherwise needs to use a pypy build or similar as Ansible bootstrap command.
The Azure OEM extension already uses Python, and that means this new Python extension would result in overlay conflicts. However, I think they should not matter because we built both from the same Python version. Would be good to test that things are ok, though.

[github-actions]

Build action triggered: https://github.com/flatcar/scripts/actions/runs/9448288856

[pothos]

A problem with pip which has to run in a venv but venv creation fails:

python3 -m venv  myenv
Error: Command '['/home/core/myenv/bin/python3', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Note: Tested this not on the built image because of slow download, used an older nightly and systemd-sysext refresh --force but it should still be ok.

[JeWe37]

I haven't tested this myself but I'm guessing I know where it comes from: ensurepip isn't installed. I did notice but didn't think it would matter seeing as pip is already installed. Fixing it would be easy, it's just a USE flag that s been disabled. I'll do that later.

As for the Azure extension, the nicest and most reliable solution would likely be to have it depend on this sysext, but there is no scheme for that. Building that also might not be so easy. I can't really test the Azure case myself though.

[pothos]

Thanks for looking into this. I don't expect that we support every python dev flow but with venv and pip most things should be covered for running simple things.

As for the Azure extension, the nicest and most reliable solution would likely be to have it depend on this sysext, but there is no scheme for that. Building that also might not be so easy. I can't really test the Azure case myself though.

We could do that by detecting that we are on Azure and then synthesizing this extension enablement entry in the initrd, update postinst action, and flatcar-update but we don't yet have a nice way to prepopulate the Azure image. We could place it on the rootfs as stop-gap solution but that means it gets downloaded when one reformats the rootfs. Okay for Azure but not for airgapped envs. We had the idea to repurpose the empty /usr B partition for that. Anyway, nothing for this PR.

[JeWe37]

That's a strange place for the tests to fail given what this changes. Do I perhaps have to update the version number or something along those lines?

[pothos]

The two failed tests are a problem with the CI setup, sorry for the confusion they cause.

Would be nice if someone could finish the review here (Test pip venv, check if the pip enablement makes a difference for the Azure or GCE OEMs - I guess not?), CC: @flatcar/flatcar-maintainers

[JeWe37]

I'd love to see this make it in before the next alpha, such that it can easily be used, anything I can help with to that end still?

[tormath1]

I'd love to see this make it in before the next alpha, such that it can easily be used, anything I can help with to that end still?

Sorry I missed that, I was AFK until the beginning of this week. Will have a look / test things! Thanks for this contribution. 💪

[tormath1]

A problem with pip which has to run in a venv but venv creation fails:

python3 -m venv  myenv
Error: Command '['/home/core/myenv/bin/python3', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Note: Tested this not on the built image because of slow download, used an older nightly and systemd-sysext refresh --force but it should still be ok.

I tested on the built image: no issue.

core@localhost ~ $ python3 -m venv  myenv
core@localhost ~ $ source myenv/bin/activate
(myenv) core@localhost ~ $ which python
/home/core/myenv/bin/python
(myenv) core@localhost ~ $ pip install requests
Collecting requests
  Using cached requests-2.32.2-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests)
  Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests)
  Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests)
  Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests)
  Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Using cached requests-2.32.2-py3-none-any.whl (63 kB)
Using cached certifi-2024.2.2-py3-none-any.whl (163 kB)
Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB)
Using cached idna-3.7-py3-none-any.whl (66 kB)
Using cached urllib3-2.2.1-py3-none-any.whl (121 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2024.2.2 charset-normalizer-3.3.2 idna-3.7 requests-2.32.2 urllib3-2.2.1
(myenv) core@localhost ~ $ python
Python 3.11.9 (main, May 23 2024, 11:02:40) [GCC 13.2.1 20240210] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>>

[tormath1]

Great thanks, overall it looks good now. Any chance to rebase before I kick off a last CI to run more some tests on other providers?

[JeWe37]

Great thanks, overall it looks good now. Any chance to rebase before I kick off a last CI to run more some tests on other providers?

Done, rebased onto current main

[JeWe37]

Not entirely sure what those errors are about, but I don't think they have anything to do with the PR again.

[tormath1]

@JeWe37 thanks a lot for your contribution. As a final step, can I ask you to provide a small entry in this table to reference the podman and python extensions: https://www.flatcar.org/docs/latest/provisioning/sysext/#flatcar-release-extensions (https://github.com/flatcar/flatcar-website/blob/d92622d131661e8562df0c90df401b4a13ecdd13/content/docs/latest/provisioning/sysext/_index.md?plain=1#L27) ?