Add cibuildwheel support

[ShadowJonathan]

This PR adds cibuildwheel support for python wheels: allowing for CI-powered reproducible python builds into self-contained wheels.

(Python wheels are distribution packages containing everything a python library needs to run, including shared libraries. They're tagged with compatibility markers (OS, Architecture, and for linux: GLIB version), so that pip can download the correct variant from a package index like https://pypi.org)

This is achieved with enscons, which puppets scons to achieve the right builds and environments.

This names the output library as soar-sml, and renames Python_sml_ClientInterface.py to __init__.py, so that import soar_sml will have the same effect as import Python_sml_ClientInterface.

As this then becomes a pip-installable library, sys.path.append is not required anymore, import soar_sml will do the trick from anywhere on the system.

---

TODOs

• Test more python versions.
  • (Python 3.8 - 3.12 have been successfully built for)
  • (PyPy appears to build for successfully, but running a simple import and echo world test reveals it does not work properly.)
• Test more operating systems.
  • (Linux x86_64 have been successfully built for (and tested))
    • (Linux ARM64 was built for on my dev machine, but this hasn't been added to the CI)
  • (Windows x86_64 has been successfully built for, x86 (win32) does not work properly due to inability to tell scons to build for x86 only)
  • (Mac x86 and ARM64 have been successfully built for (and tested))
• Add .github/workflows file for doing cibuildwheel
• Add proper metadata to pyproject.toml
  • authors, keywords, classifiers, urls, and probably another few fields in the pyproject.toml file needs to be updated/added with the right data.

---

My goal with this is to make Soar ready to be packaged for https://pypi.org distribution (likely in a future PR, with proper discussion), and to then create a library that will make accessing soar even more pythonic. (Name pending, also, soar was taken, *shakes fist*)

I hope that both of these efforts will lower the barrier for entry and experimentation with Soar, especially as SML allows anyone to create a "body" for Soar agents to "interact" with the world.

[ShadowJonathan]

(Converting to draft while I try to figure out Windows builds)

[garfieldnate]

This is very exciting! Thank you so much for looking into this. I would love for users to be able to simply pip install soar-cog-arch==9.6.3 to get Soar (seriously, "snakes on a robot" took the soar name?! 🤣).

• I would like the package version to match Soar's version (if it doesn't already); this isn't defined in any one place, but you could put in 9.6.2 for now and it'll get replaced before the next release.
• Ideally I think we need to include a script that gives users the directory where Soar is installed ($SOAR_HOME). This is important when, for example, using Soar with Java, where you have to run with -Djava.library.path="$SOAR_HOME". Or maybe it's not necessary after the wheel is installed? 🤔
• I would prefer to keep the Python_sml_ClientInterface name, since changing it will break existing clients. Was there a technical reason you needed to change it, or was it just that the name is ugly (which it definitely is)?

For future work (it's a bigger change), we may want this to be usable for general users that want to run VisualSoar, the debugger, etc. from the launch scripts. We would need to include the scripts/jars for running these into the wheel, and we would need to notify the user of where Soar was installed (previous point). Right now the final Soar distribution is built using a separate project: https://github.com/SoarGroup/Release-Support.

Regarding more Pythonic access for Soar, there is a popular (in our tiny community) project that does something similar called pysoarlib. I have a PR open for making it pip-installable: amininger/pysoarlib#7. I actually ended up forking it for our current internal project because I wanted to make a lot of changes to it. I don't want to discourage you from working on a new Python library, as a fresh take would probably be very nice, but I wanted to let you know that it's there. The design pattern with SoarClient using AgentConnector implementations and exposing only the input, output and init events has been especially useful.

[ShadowJonathan]

I would like the package version to match Soar's version

Already intending to, so that soar-sml can have its versioning decoupled, and simply define a range of soar versions for which it supports its python/sml interface.

Or maybe it's not necessary after the wheel is installed?

Not necessary, as Soar gets built and statically linked against the SML interface library (or at the very least, included in the wheel), meaning that its essentially bundling a distribution of Soar with the wheel itself as well.

Was there a technical reason you needed to change it, or was it just that the name is ugly?

Well, multiple reasons;

• it certainly commits multiple crimes against the zen of python
• it violates python packaging naming standards, combining snake_case and camelCase
• the wheel would have to package a python file "at root", which is something that's very much not recommended, as python's entire packaging standard revolves around importable modules, and a Python_sml_ClientInterface.py and _ Python_sml_ClientInterface.so sitting at the root of site-packages is not considered very "clean", and possibly may cause problems even (when some tools try to look up packages/directories and/or when importing/auto-importing)

So part technical reason and social/cleanliness reason.

There could possibly be an extra package which installs that Python_sml_ClientInterface.py at site-packages root and just does from soar_raw import *, maybe call it soar-compat (installable via soar-raw[compat]), but I'm really against having it there by default.

we may want this to be usable for general users that want to run VisualSoar, the debugger, etc. from the launch scripts

I... would have to take a closer look at that, at the moment I'm really just focusing on getting the SML part up and ready. I don't have any thoughts on this at the moment, and the decision on wether to bundle the VisualSoar parts (if in includes java; no), making it easy to make VisualSoar/debugger connectable (probably), or something else, is something I don't have enough info for.

This library is just intended to start the soar kernel, load agent code, and setup the IO link via python. Debugging and whatnot should then be pluggable similar to how someone who usually runs Soar would expect it to work, imo.

For visualsoar or the debugger to connect, the programmer would have to first fire off python with a script that loads the soar agent (with its connector), and only then they could connect to it, methinks. This would miss out the connector part of debugging, which would need to be done on the python side in the IDE's native debugger.

(pysoarlib) [...] but I wanted to let you know that it's there.

I saw, though thanks for mentioning it for if I hadn't :)

I'm going to take a look at it for inspiration, but largely the reason I want to make an independent library is because I then both learn the internals of SML while thinking of a pythonic interface, hopefully with that fresh perspective not being bogged down by rigidity to conform to something similar to Soar's internals, etc.

Taking a look again, the sub-classable connector interface does strike a fancy; I think I'd like more parts of that to be as modular, such as entire input or output trees, "interpreters", so that data gets transformed into WM elements efficiently, and such interpreters/trees can be modularised and shared around, iterated.

(plus having nice formatted value trees on repr() and the likes)

That library design looks poll-based instead of push-based; I want to play around with that as well, where instead of the WM being polled for every update, the programmer can opt to instead explicitly push data at "off" intervals (decoupled from the decision cycle). But I have to think how such a library design would work.

One nice thing about decoupling this from the "raw" library is that I can do alpha/beta versioning while I experiment with this :)

Edit: An additional thought; I can make it work with asyncio, instead of requiring synchronous code, which would then help introduce tons of libraries which do asynchronous IO off the device (such as driving motors, or APIs)

[garfieldnate]

the wheel would have to package a python file "at root"

What do you mean by this? Do you mean that you plan for the import to be from something more nested, like from soargroup.soar import soar_raw or something?

BTW I think calling it soar_sml instead of soar_raw would suffice for clarity; the SML API is as raw as it gets from the Python side. Then for your pythonic wrapper you'd be free to be as creative as you want, from the mundane "soar_pythonic" to the more fun "glider" or "eagle" (or whatever else soars).

I think we can punt the question of more general usage down the road for now. Having any form of pip-installable Soar would already be very cool. This is also related to the point where you say Not necessary, as Soar gets built and statically linked against the SML interface library; a general installation of Soar has SML bindings for Python, Java, Tcl, C++ and C#. It's a lot to manage.

the programmer can opt to instead explicitly push data at "off" intervals

A nice role-reversal! Give highest-level control to the external application instead of to Soar. This programming pattern is less common for Soar because it is more complex, but I would love to see it done. I could be mistaken, but I think you'd want to create the kernel in the current thread using sml::Kernel::CreateKernelInCurrentThread, and you'll need to take note of the usage comment here: https://github.com/SoarGroup/Soar/blob/development/Core/ClientSML/src/sml_ClientKernel.h#L318. You have to call CheckForIncomingCommands() periodically if you want the kernel to be reactive to any other clients (such as the debugger). (If I recall correctly this doesn't apply to the client in the thread that created the kernel, just other clients remotely connecting to it).

I think your ideas for a Pythonic Soar package sound really cool, so I'm excited to see what you come up with!

[ShadowJonathan]

Felt a bit inspired and started drawing up the different ways that soar and the environment could asynchronously run alongside eachother, apologies for the digression. (Click to open)

I thought about how soar-sml would do asynchronous IO for a second, and I figured there'd at least be three configurations possible;

Here's a simple loop; Soar (grey) runs one decision cycle, and then waits before the output (red) has been completed on the connector, then waits before the input (blue) is completed on the connector, and then runs another cycle.

However, due to asyncio, it is possible to say "Hey, run the output code together with the input code, and only wait for the input to complete";

This would have the input and output code fight for the state of the world, but this may be possible in some scenarios (or some sub-connectors), and a programmer might want to enable this for optimisation purposes.

Then there's a final method, which runs the input collector every time, and then feeds soar the last result (encircled), while also asynchronously running the output;

Again, this would have implications, but could be something the programmer wants to weigh off.

All of these have tradeoffs, but supporting all of them is possible, and its good to provide options.

[ShadowJonathan]

What do you mean by this? Do you mean that you plan for the import to be from something more nested, like from soargroup.soar import soar_raw or something?

Currently, the import would be import soar_raw, soar_raw would be installed as a directory under python's site-packages folder, and putting anything but a namespace directory there (or installing to someone else's namespace directory) is frowned upon.

BTW I think calling it soar_sml instead of soar_raw would suffice for clarity

That's fair, then I think I'll indeed swap it around for that (making soar-raw into soar-sml; I'll update this PR with that at some point), and find a good name for my pythonic package :)

A general installation of Soar has SML bindings for Python, Java, Tcl, C++ and C#. It's a lot to manage.

To be honest, I think pivoting to packaging the python binding for soar to a wheel file for users to install would be more user-friendly in the long-term, since instructing them to manually pivot $PATH to that directory is awkward at best, and fragile and non-portable at worst.

(wrt CreateKernelInCurrentThread)

I'll have to see what is most optimal, what I'm seeing here is that doing CreateKernelInCurrentThread basically defers a lot of housekeeping to the developer, right? If so, that'd have more control (and could play nicely with asynchronous IO), but would be a bit more complexity. I'll play around with that later :)

[garfieldnate]

All sounds good :) One last note on concurrency: core Soar does not support it. It is not thread-safe. The SML client library uses locking to ensure that only one command can get sent at a time, and on the kernel side there is a queue for receiving commands over the network socket (as when the debugger connects).

So there are a couple safety measures in place, but if you're not careful you can run into issues like #430. This is also related to the actions one might take in a handler; when control returns to Soar after a handler is complete, it doesn't do any checking to make sure that, e.g. the agent list hasn't changed. Unfortunately these edge cases aren't well-documented, but if you stick to actions that were definitely intended and designed for, such as creating WME's on the input link in the input handler and reading command WME's on the output link in the output handler, everything will work fine.

[moschmdt]

Felt a bit inspired and started drawing up the different ways that soar and the environment could asynchronously run alongside eachother, apologies for the digression. (Click to open)
I thought about how soar-sml would do asynchronous IO for a second, and I figured there'd at least be three configurations possible;

Here's a simple loop; Soar (grey) runs one decision cycle, and then waits before the output (red) has been completed on the connector, then waits before the input (blue) is completed on the connector, and then runs another cycle.

However, due to asyncio, it is possible to say "Hey, run the output code together with the input code, and only wait for the input to complete";

This would have the input and output code fight for the state of the world, but this may be possible in some scenarios (or some sub-connectors), and a programmer might want to enable this for optimisation purposes.

Then there's a final method, which runs the input collector every time, and then feeds soar the last result (encircled), while also asynchronously running the output;

Again, this would have implications, but could be something the programmer wants to weigh off.

All of these have tradeoffs, but supporting all of them is possible, and its good to provide options.

O

A little bit off topic: I am currently working on a Soar ROS2 integration and I implemented something similar to your first idea but I instantly push the elements to a consumer thread via queues so Soar remains responsive for output link wmes. For the input I read all queues and attach the data to the input link.

I did not publish the code (cop) yet.

So far this approach works great and I did not experience any weird behaviour. I hope this helps!

[ShadowJonathan]

CI builds properly now, so there are a few outstanding tasks, and some notes:

• authors, keywords, classifiers, urls, and probably another few fields in the pyproject.toml file needs to be updated/added with the right data.
• as I said before, we should maybe also upload a soar-compat project that exposes a Python_sml_ClientInterface namespace, which just re-exports all the contents of soar_sml (a simple one-liner file from soar_sml import *), able to be easily installed with pip install soar-sml[compat]
  • Installing this would instantly recover all compatibility with all the existing projects, and make them properly portable as a result :)
  • The [compat] bit is an "extra", which'd simply add a dependency to soar-compat, installing that "package" as well.
  • I can very easily make this (its just a subdirectory with a pyproject.toml and a single python file that can then be "compiled" with cibuildwheel and uploaded, and never needs to be updated), so just say 'alright' to that if I should do so in this PR :)
• We should maybe also decide on how many past versions we want to upload to pypi, or if we just start with 9.6 and go from there.

Notes:

• I've uploaded the current build artifacts to pypi.org/project/soar-sml, this is to prevent someone else nicking and squatting it (I've had that happen a few times before), and for the build to be installable by anyone (try it! pip install soar-sml, then import soar_sml from anywhere)
  • Just tag me with a organisation or maintainer username on pypi that I can transfer ownership of the pypi project to, this is just a safety measure.
  • The current version on pypi is an "alpha" version, when a final (non-alpha) version gets uploaded, pip will default to installing that one over the alpha version.
  • Furthermore, releases can be deleted, so this stub release can be safely deleted after real releases are uploaded.
• PyPy does "compile", but it doesn't seem to do it correctly, as running the test command (a simple import and echo of hello world) fails due to import complications. I've disabled pypy as a build target until this is resolved.
  • PyPy might be a desirable target to be able to be installed to, as it does JIT compilation of its running python code, which would have a ton of optimisation.
• Building for 32-bit windows has also been disabled, as I couldn't get Scons to build the 32-bit variant of the project properly, and the build failed due to linking errors (of 32-bit libraries against 64-bit binaries)
• Windows wheels also currently have a quirk where they contain both Soar.lib and Soar.dll, while I'm fairly sure it only requires the latter one. This is due to how scons handles aliases and shared library files internally.
• Python 3.12 isn't currently being built for. I ran into an issue where enscons tried to import imp on 3.12 (which has been deprecated since 3.4, and finally removed in 3.12), which then failed and crashed the build script. I don't know how I can instruct cibuildwheel to run a specific version of python for the build system, but seeing as how the SConstruct/SConscript scripts pull out details from the currently-running interpreter for the build environment, I don't think that'll help.
  • Filed an issue on the original repo, and also on a fork, in the hopes that it'll get resolved.
• The current pypi upload and setup does NOT package source installs. This means that pip install soar-sml needs to happen on a supported platform, or else it will refuse to install.
  • This can be fixed by either;
    • Adding more wheels for more platform variants.
    • Adding a source install, although, this effectively needs to replicate the build system, and there'll be no guarantees that it will build on a system.
  • IMO: Keep it this way, building from source on pip install is a source of headaches, as target systems are often entirely unequipped for building sources, and Soar's *cough* particular build system will likely break in all sorts of ways on an unprepared system. (This is why building in CI is a preferred way of doing it: it keeps hell contained)

[ShadowJonathan]

Added a few notes. These can all be resolved, but just a few thoughts wrt points in the changes that I think should be noted.

[ShadowJonathan]

This makes sure that even if someone is building it without cibuildwheel+enscons, this script can still run normally.

[scijones]

You've given us a lot here. We need to review a few of these things (I haven't caught up on all the stuff that's happened here), but I like this so far. Thanks!

You say "Installing this would instantly recover all compatibility with all the existing projects, and make them properly portable as a result :)" "so just say 'alright' to that if I should do so in this PR"

Alright.

"I've uploaded the current build artifacts to pypi.org/project/soar-sml, this is to prevent someone else nicking and squatting it (I've had that happen a few times before)" --Thanks for that, too!

[ShadowJonathan]

Alright, I added soar-compat and uploaded this small library to pypi as well, together with a new version of build artifacts on an alpha version.

It is now possible to do pip install "soar-sml[compat]", and receive full and simple backwards compatibility with all existing scripts for soar :)

[garfieldnate]

Just tag me with a organisation or maintainer username on pypi that I can transfer ownership of the pypi project to, this is just a safety measure.

I created a garfieldnate account and a SoarGroup organization on pypi today; the organization has to be approved by someone before it exists, so I'll let you know when that comes through.

PyPy does "compile", but it doesn't seem to do it correctly...

Most of the actual running code is going to be compiled C++, and I'm not worried about the speed of the wrapper code. Do you think I should be? Soar does a lot of work, and I would hope that by comparison a simple method call in Python would be no big deal.

We should maybe also decide on how many past versions we want to upload to pypi

I think starting with 9.6.2 is fine. Going back and building earlier versions of Soar is a big pain.

Building for 32-bit windows has also been disabled

That's fine, we don't support it anymore.

Windows wheels also currently have a quirk where they contain both Soar.lib and Soar.dll

Definitely only need the latter, but I wouldn't say it's a show-stopper for now.

Python 3.12 isn't currently being built for.

How does this work? The shard library that Python loads only works with 3.12, which is the version of Python that we build with. This would mean that the wheel wouldn't work at all, since it can only be used with versions of Python that the lib won't work with, but you said it's working fine, and I don't understand how.

The current pypi upload and setup does NOT package source installs

I would have thought it were complete magic if this worked for everyone :D It takes a bit of work to set up the environment to build Soar. I think it's fine to go without this.

[garfieldnate]

This is looking great! I'm really excited. Most requested changes are just documentation/comments.

[garfieldnate]

Beautiful! Backwards-compatibility. Thank you!

[garfieldnate]

I'm really happy about how thorough you're being!

[ShadowJonathan]

...right, I see what's wrong, you're on MacOS 13, while the builder aims for MacOS 14, and so there are no wheels it can download.

This should be configurable, and I'll attempt to get it down to MacOS 12 or lower, to maximise compatibility with old MacOS versions.

• This is something I want to attempt to improve before merging this PR, I'll poke back if I succeed or fail in this, thanks for raising the issue! :)

[ShadowJonathan]

@garfieldnate I've fixed the problem with the latest commit, this should now properly "downversion" the built-for mac wheels, so that it tags itself by the minimum mac version it can get downloaded by, and as a result we got Intel Mac compatibility for Mavericks (10.9!), and all Apple Silicon Macs :)

I've uploaded the new build artifacts that tag these correctly under .dev2, I suspect that the next wheels I'll upload will have that dynamic versioning mechanism I talked about, but for now, soar-sml should be installable on (realistically) all Macs :)

(Excluding python 3.8 on Apple Silicon (ARM64) macs, for reasons I poked earlier, and in the file comments)

[garfieldnate]

I tried it just now and it just worked! Tested on my ARM Mac, then x86_64 Ubuntu and Windows 11.

[ShadowJonathan]

• I currently want to still add the automatic releasing to the build.yml, as I've already began working on it, and it seems to be easy enough.

The way I'll be doing it (by splitting the jobs) will make CI faster, too.

I'll add a CI job that'll publish the build versions to test.pypi.org, and I'll add a CI job that'll publish them to regular pypi if the CI run gets triggered by a tag push / release.

[ShadowJonathan]

• Added a dynamic versioning mechanism, which will:
  • For development and local versions, version according to the offset of the latest release tag, and add extra metadata.
  • For release versions, version according to the tag of the current commit.
• Seperated out the wheel building into seperate jobs, which concurrently run faster.
• Moved documentation for developing and packaging to its separate file.
• Added an uploading step:
  • Dev builds on a schedule, every week, uploaded to https://test.pypi.org/p/soar-sml
  • Final versions on GitHub Releases, uploaded to https://pypi.org/p/soar-sml

---

A development version can be installed via this command now:

pip install --pre -i https://test.pypi.org/simple/ soar-sml

[garfieldnate]

I'm having some trouble understanding the GitHub UI, but as far as I can tell, the one question I just posted is the last unresolved conversation, correct? And then it's ready to merge, right?

[garfieldnate]

One other question 😅 Do we need to upload soar-compat to test.pypi.org?

[ShadowJonathan]

I'm having some trouble understanding the GitHub UI, but as far as I can tell, the one question I just posted is the last unresolved conversation, correct?

Yes

And then it's ready to merge, right?

If you feel like it is, yes :)

Do we need to upload soar-compat to test.pypi.org?

...Yes, actually, I just tested it, and it can't find soar-compat, so I'll upload it right now ^^;

Edit: done :)

[garfieldnate]

Thanks for all the hard work! I look forward to you sharing this at the workshop! 🎉

[ShadowJonathan]

Thank you a lot! 💚