feat: improve marshaller performance

[bdraco]

This is roughly a 10% speed up marshalling vs the benchmark

adds some more missing types

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e634fc2) 83.37% compared to head (288b6d6) 83.37%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #264   +/-   ##
=======================================
  Coverage   83.37%   83.37%           
=======================================
  Files          28       28           
  Lines        3290     3290           
  Branches      682      682           
=======================================
  Hits         2743     2743           
  Misses        331      331           
  Partials      216      216           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jameshilliard]

This appears to have broken the ability to build dbus-fast on buildroot.

>>> python-dbus-fast 2.13.0 Building
(cd /home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0//; _PYTHON_HOST_PLATFORM="linux-x86_64" _PYTHON_PROJECT_BASE="/home/buildroot/buildroot/output/build/python3-3.12.5" _PYTHON_SYSCONFIGDATA_NAME="`{ [ -e /home/buildroot/buildroot/output/per-package/python-dbus-fast/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/python3.12//_sysconfigdata__linux_*.py ] && basename /home/buildroot/buildroot/output/per-package/python-dbus-fast/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/python3.12//_sysconfigdata__linux_*.py .py; } || true`" PATH="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin:/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/sbin:/home/buildroot/.cargo/bin:/home/buildroot/bin:/home/buildroot/.local/bin:/home/buildroot/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" GIT_DIR=. PATH="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin:/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/sbin:/home/buildroot/.cargo/bin:/home/buildroot/bin:/home/buildroot/.local/bin:/home/buildroot/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin" AR="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gcc-ar" AS="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-as" LD="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-ld" NM="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gcc-nm" CC="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gcc" GCC="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gcc" CPP="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-cpp" CXX="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-g++" FC="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gfortran" F77="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gfortran" RANLIB="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-gcc-ranlib" READELF="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-readelf" STRIP="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-strip" OBJCOPY="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-objcopy" OBJDUMP="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-objdump" AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as" CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc" CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld" CPPFLAGS_FOR_BUILD="-I/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/include" CFLAGS_FOR_BUILD="-O2 -I/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/include" CXXFLAGS_FOR_BUILD="-O2 -I/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/include" LDFLAGS_FOR_BUILD="-L/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib -Wl,-rpath,/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib" FCFLAGS_FOR_BUILD="" DEFAULT_ASSEMBLER="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-as" DEFAULT_LINKER="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/x86_64-linux-ld" CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1" CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64  -O2 -g0 -D_FORTIFY_SOURCE=1" LDFLAGS="" FCFLAGS=" -O2 -g0" FFLAGS=" -O2 -g0" PKG_CONFIG="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/pkg-config" STAGING_DIR="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/x86_64-buildroot-linux-gnu/sysroot" INTLTOOL_PERL=/usr/bin/perl PYTHONPATH="/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/python3.12/" PYTHONNOUSERSITE=1 REQUIRE_CYTHON=1 /home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/python3 -m build -n -w )
* Getting build dependencies for wheel...
* Building wheel...
A setup.py file already exists. Using it.
Compiling src/dbus_fast/aio/message_reader.py because it changed.
Compiling src/dbus_fast/message.py because it changed.
Compiling src/dbus_fast/message_bus.py because it changed.
Compiling src/dbus_fast/service.py because it changed.
Compiling src/dbus_fast/signature.py because it changed.
Compiling src/dbus_fast/unpack.py because it changed.
Compiling src/dbus_fast/_private/address.py because it changed.
Compiling src/dbus_fast/_private/marshaller.py because it changed.
Compiling src/dbus_fast/_private/unmarshaller.py because it changed.
[1/9] Cythonizing src/dbus_fast/_private/address.py
[2/9] Cythonizing src/dbus_fast/_private/marshaller.py

Error compiling Cython file:
------------------------------------------------------------
...
            return 0
        for _ in range(offset):
            self._buf.append(0)
        return offset

    def write_boolean(self, boolean: bool, type_: SignatureType) -> int:
                                          ^
------------------------------------------------------------

src/dbus_fast/_private/marshaller.py:47:43: Compiler crash in AnalyseDeclarationsTransform

File 'ModuleNode.py', line 124, in analyse_declarations: ModuleNode(marshaller.py:1:0,
    full_module_name = 'dbus_fast._private.marshaller')
File 'Nodes.py', line 431, in analyse_declarations: StatListNode(marshaller.py:1:0)
File 'Nodes.py', line 4817, in analyse_declarations: CClassDefNode(marshaller.py:18:0,
    class_name = 'Marshaller',
    doc = 'Marshall data for Dbus.',
    visibility = 'private')
File 'Nodes.py', line 431, in analyse_declarations: StatListNode(marshaller.py:19:4)
File 'Nodes.py', line 2359, in analyse_declarations: CFuncDefNode(marshaller.py:47:4,
    is_c_class_method = 1,
    modifiers = [...]/0,
    overridable = 1,
    visibility = 'private')
File 'Nodes.py', line 681, in analyse: CFuncDeclaratorNode(marshaller.py:47:4,
    calling_convention = '')
File 'Nodes.py', line 909, in analyse: CArgDeclNode(marshaller.py:47:43,
    is_generic = 1,
    outer_attrs = [...]/2)

Compiler crash traceback from this point on:
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/Cython/Compiler/Nodes.py", line 909, in analyse
    if self.annotation and env and env.directives['annotation_typing'] and self.base_type.name is None:
                                                                           ^^^^^^^^^^^^^^^^^^^
AttributeError: 'CAnalysedBaseTypeNode' object has no attribute 'name'
Traceback (most recent call last):
  File "/home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0/setup.py", line 29, in <module>
    build(setup_kwargs)
  File "/home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0/build_ext.py", line 23, in build
    ext_modules=cythonize(
                ^^^^^^^^^^
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/Cython/Build/Dependencies.py", line 1115, in cythonize
    cythonize_one(*args)
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/Cython/Build/Dependencies.py", line 1238, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: src/dbus_fast/_private/marshaller.py
Traceback (most recent call last):
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/poetry/core/masonry/api.py", line 58, in build_wheel
    return WheelBuilder.make_in(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
    wb.build(target_dir=directory)
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 121, in build
    self._build(zip_file)
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 182, in _build
    self._run_build_command(setup)
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 248, in _run_build_command
    subprocess.check_call([
  File "/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/x86_64-buildroot-linux-gnu/sysroot/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/buildroot/buildroot/output/per-package/python-dbus-fast/host/bin/python3', '/home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0/setup.py', 'build', '-b', '/home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0/build', '--build-purelib', '/home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0/build/lib', '--build-platlib', '/home/buildroot/buildroot/output/build/python-dbus-fast-2.13.0/build/lib.linux-x86_64-cpython-312']' returned non-zero exit status 1.

ERROR Backend subprocess exited when trying to invoke build_wheel

[bdraco]

It looks like we don't have a pin for >3 for cython in the build system, only the group above

dbus-fast/pyproject.toml

Line 104 in e393611

requires = ['setuptools>=65.4.1', 'wheel', 'Cython', "poetry-core>=1.0.0"]

[bdraco]

v2.24.1 is building now. It will be available in a few hours