Using nanobind and PyObject_CallObject

[baverbud]

I have a module defined with nanobind:

struct TestStruct
{
   int a;
};

NB_MODULE(my_ext, m) {
    nanobind::class_<TestStruct >(m, "TestStruct")
        .def(nanobind::init<>())
        .def_rw("a", &TestStruct::a);
}

I'm trying to wrap an existing TestStruct with nanobind and use it with PyObject_CallObject:

    pArgs = PyTuple_New(1);
    TestStruct * ts = new TestStruct ();
    ts->a = 5;
    nb::handle py_type = nb::type<TestStruct >();
    if (!(nb::type_check(py_type ) &&
            nb::type_size(py_type ) == sizeof(TestStruct ) &&
            nb::type_align(py_type ) == alignof(TestStruct ) &&
            nb::type_info(py_type ) == typeid(TestStruct )))
        throw std::runtime_error("Internal error!");
    nb::object py_object = nb::inst_wrap(py_type , ts);
    nb::inst_set_state(py_object , /* ready */ true, /* destruct */ false);
    PyTuple_SetItem(pArgs, 0, py_object .ptr());

    //Call my function, passing it the number four
    pValue = PyObject_CallObject(pFunc, pArgs);

CMake and nanobind are building the NB_MODULE and I'm linking the PyObject_CallObject code against the built shared object. The code trying to run PyObject_CallObject is not being built using the nanobind_add_module (it's part of a bigger application) but it includes the TestStruct definition and the nanobind headers.

nb::type consistently returns a null pointer in the handle, which seems to indicate it couldn't find the type. I've included the NB_MODULE definition in the same file as the PyObject_CallObject code, and there is no change in the return for nb::type.

Should I be able to look up the python type this way? Do I need to populate the list of python types somehow?

[wjakob]

Careful with reference counting. PyTuple_SetItem steals the reference away from nb::object, and you will get undefined behavior.

Use either PyTuple_SetItem(pArgs, 0, py_type.release().ptr()); or (better) nb::make_tuple

[baverbud]

Calling PyInit_my_ext looks like it registers the type properly so that nb::type can find it.

[baverbud]

This looks to be a bit more complicated now that I have the python script importing my_ext.

Once I import my_ext in the script, I can no longer call PyInit_my_ext in the C code since it is already called by the script (or vice versa - order of the calling doesn't seem to matter):

Critical nanobind error: nanobind::detail::nb_type_new("TestStruct"): type was already registered!

However nb::type is back to having a null ptr() when I try to get a handle for TestStruct.

The way I'm trying to run the script is maybe a bit convoluted and causing an issue. I want to run a function from my script (passing in an object from C++), and I have it in the form of a string, not a file. It's based on https://stackoverflow.com/questions/3789881/create-and-call-python-function-from-string-via-c-api .

    //Create a new module object
    PyObject *pNewMod = PyModule_New("TestModule");

    PyModule_AddStringConstant(pNewMod, "__file__", "");

    //Get the dictionary object from my module so I can pass this to PyRun_String
    pLocal = PyModule_GetDict(pNewMod);

    //Define my function in the newly created module
    pValue = PyRun_String(myScriptString), Py_file_input, pGlobal, pLocal);

   <snip error handling>

    //Get a pointer to the function I just defined
    pFunc = PyObject_GetAttrString(pNewMod, "testFunction");

    // 3 scenarios I've tried:
    // 1. If the python script doesn't import my_ext but I run PyInit_my_ext() before the below line, py_type.ptr() is not null
    // 2. If the python script imports my_ext and I do not run PyInit_my_ext() before the below line, py_type.ptr() is null
    // 3. If the python script improts my_ext and I run PyInit_my_ext(), on the PyInit_my_ext() fails because TestStruct already exists.
    nb::handle py_type = nb::type<TestStruct>();
    nb::object py_object = nb::inst_wrap(py_type, &testObject);
    nb::inst_set_state(py_object , /* ready */ true, /* destruct */ false);
    nb::object args = nb::make_tuple(py_object );

    //Call my function, passing it the number four
    pValue = PyObject_CallObject(pFunc, args.ptr());

This is the "myScriptString" that is loaded as the script:

import sys
sys.path.append("path/to/the/so")
import my_ext

def testFunction(obj):
    print('Hello world!')

[baverbud]

@wjakob the hash_code on the type_info for TestStruct is different during PyInit_my_ext and when I call nb:::type.

This looks exactly like the issue you reported for pybind11 about 6 years ago here: pybind/pybind11#912. Setting the dlopen flags in the python script before importing my nanobind module does not seem to help though.

Thoughts?

[baverbud]

Ok, this seems resolved by adding

__attribute__ ((visibility("default")))

to the struct definition. Besides hash_code's matching and type_index working to find the type in the internals.type_c2p map, I can now see the type in the symbols of the so produced by the nanobind cmake routines.

[wjakob]

For posterity, here is a FAQ entry about this issue: 2cb5338