[QUESTION] Saving polymorphic types (that were overridden in Python) in a C++ struct ─ question about py::cast

[tttapa]

_{(Main question is in bold at the bottom)}

I'm trying to get a struct to store (a pointer to) an instance of a polymorphic type as a member. Basically:

struct Polymorphic { ... };

struct Wrapper {
  std::shared_ptr<Polymorphic> ptr;
};

Doing this in C++ is easy enough, but now I would like Python code to be able to inherit from Polymorphic as well. Inheritance using “trampolines” as explained in the docs works well, but I get into trouble when adding these objects to the Wrapper, because the Python state is lost in the process.

My first unsuccessful attempt was as follows:

#include <cxxabi.h>
#include <iostream>
#include <memory>
#include <string>
#include <typeinfo>

#include <pybind11/pybind11.h>
namespace py = pybind11;

/// Just a class with a virtual method and virtual destructor
struct PolymorphicBase {
    virtual ~PolymorphicBase()            = default;
    virtual std::string to_string() const = 0;
};

// Trampoline for inheriting/overriding in Python
struct PolymorphicTrampoline : PolymorphicBase {
    std::string to_string() const override { PYBIND11_OVERRIDE_PURE(std::string, PolymorphicBase, to_string, ); }
};

// Derived class in C++
struct PolymorphicDerived : PolymorphicBase {
    std::string to_string() const override { return "PolymorphicDerived"; }
};

// Some other class that stores a shared_ptr to PolymorphicBase
struct Wrapper {
    std::shared_ptr<PolymorphicBase> ptr;
    std::string to_string() const { return "Wrapper: " + ptr->to_string(); }
};

static inline std::string demangled_type_name(const std::type_info &ti) {
    int status = 0;
    std::unique_ptr<char, decltype(&std::free)> cstr(abi::__cxa_demangle(ti.name(), nullptr, nullptr, &status), std::free);
    return cstr.get();
}

// Returns a function that makes a Wrapper from the (polymorphic) argument
template <class T>
auto WrapperConstructor() {
    return [](std::shared_ptr<T> t) -> Wrapper {
        std::cout << __PRETTY_FUNCTION__ << "\r\n"
                  << " --> to_string: " << t->to_string() << "\r\n"
                  << " --> C++ type:  " << demangled_type_name(typeid(*t))
                  << std::endl;
        return {std::dynamic_pointer_cast<PolymorphicBase>(t)};
    };
}

PYBIND11_MODULE(_core, m) {
    py::class_<PolymorphicBase, std::shared_ptr<PolymorphicBase>,
               PolymorphicTrampoline>(m, "PolymorphicBase")
        .def(py::init())
        .def("__str__", &PolymorphicBase::to_string);

    py::class_<PolymorphicDerived, std::shared_ptr<PolymorphicDerived>,
               PolymorphicBase>(m, "PolymorphicDerived")
        .def(py::init())
        .def("__str__", &PolymorphicDerived::to_string);

    py::class_<Wrapper>(m, "Wrapper")
        .def(py::init(WrapperConstructor<PolymorphicDerived>()))
        .def(py::init(WrapperConstructor<PolymorphicTrampoline>()))
        .def("__str__", &Wrapper::to_string);
}

Testing this module using the following Python script:

import testmodule as m

class PolymorphicPython(m.PolymorphicBase):
    def __init__(self, name) -> None:
        super().__init__()
        self.name = name
    def to_string(self) -> str:
        return self.name

a = m.Wrapper(m.PolymorphicDerived())
b = m.Wrapper(PolymorphicPython("PolymorphicPython 1"))
c = m.Wrapper(PolymorphicPython("PolymorphicPython 2"))
print(a)
print(b)
print(c)

Results in the following error:

WrapperConstructor<PolymorphicDerived>::<lambda(std::shared_ptr<PolymorphicDerived>)>
 --> to_string: PolymorphicDerived
 --> C++ type:  PolymorphicDerived
WrapperConstructor<PolymorphicTrampoline>::<lambda(std::shared_ptr<PolymorphicTrampoline>)>
 --> to_string: PolymorphicPython 1
 --> C++ type:  PolymorphicTrampoline
WrapperConstructor<PolymorphicTrampoline>::<lambda(std::shared_ptr<PolymorphicTrampoline>)>
 --> to_string: PolymorphicPython 2
 --> C++ type:  PolymorphicTrampoline
Wrapper: PolymorphicDerived
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    print(b)
RuntimeError: Tried to call pure virtual function "PolymorphicBase::to_string"

At the moment the Wrapper is constructed, calling the virtual function works (see lines 5 and 8 in the output), but when accessing it through the Wrapper later, the virtual PolymorphicPython::to_string() function is gone.
This somewhat makes sense, since I only keep the PolymorphicTrampoline object alive, not necessarily the Python state associated with it.

Following this issue (#1049 (comment)), I replaced my WrapperConstructor function by the following:

// Returns a function that makes a copy of the (polymorphic) argument and
// returns a Wrapper for it
template <class T>
auto WrapperConstructor() {
    return [](T &t) -> Wrapper {
        std::cout << __PRETTY_FUNCTION__ << "\r\n"
                  << " --> to_string: " << t.to_string() << "\r\n"
                  << " --> C++ type:  " << demangled_type_name(typeid(t))
                  << std::endl;
        auto full_python_copy = std::make_shared<py::object>(py::cast(t));
        auto base_copy = full_python_copy->template cast<PolymorphicBase *>();
        return {std::shared_ptr<PolymorphicBase>(full_python_copy, base_copy)};
    };
}

Now the Python script works correctly:

WrapperConstructor<PolymorphicDerived>::<lambda(PolymorphicDerived&)>
 --> to_string: PolymorphicDerived
 --> C++ type:  PolymorphicDerived
WrapperConstructor<PolymorphicTrampoline>::<lambda(PolymorphicTrampoline&)>
 --> to_string: PolymorphicPython 1
 --> C++ type:  PolymorphicTrampoline
WrapperConstructor<PolymorphicTrampoline>::<lambda(PolymorphicTrampoline&)>
 --> to_string: PolymorphicPython 2
 --> C++ type:  PolymorphicTrampoline
Wrapper: PolymorphicDerived
Wrapper: PolymorphicPython 1
Wrapper: PolymorphicPython 2

Now for my main question: How can py::cast(t) preserve the Python state?
The argument t is simply a reference to PolymorphicTrampoline, and its dynamic type is PolymorphicTrampoline as well, so it doesn't seem like there are any pybind11 subclasses that store any extra state. So how then does py::cast manage to “find” the Python state in order to make a copy of it (or increase the ref count)?

I've tried going through the pybind11::cast documentation and source code, and stepped through it in a debugger, but it's not clear to me what's going on under the hood.

A second question: is the second version of WrapperConstructor the correct approach to do things like this, or are there better ways to save polymorphic Python objects in the wrapper?
Maybe it's possible to pass the actual Python object to WrapperConstructor (not the reference to PolymorphicTrampoline), so the extra py::cast and make_shared<py::object> can be avoided? Can this be done while still having pybind11 match the correct type? I imagine that adding a def(py::init([](py::object o) { ... })) constructor would match any Python object, not just trampolines?

Finally, I also tried compiling and using this test module using the Address Sanitizer, and it reports many memory leaks in pybind11 code (most of them in pybind11::cpp_function::strdup_guard::operator()(char const*), pybind11::cpp_function::initialize_generic(std::unique_ptr<pybind11::detail::function_record, pybind11::cpp_function::InitializingFunctionRecordDeleter>&&, char const*, std::type_info const* const*, unsigned long) and pybind11::cpp_function::make_function_record()). Is this because of an issue with my code? Are they false positives? The complete output is rather long, but I'll be happy attach it if you think this is useful.