Inheritance with non-exposed base class

[YannickJadoul]

I came across this project, yesterday, and I'd like to say, I really like the project, improving on Boost.Python's environment and API quirks, and I'd like to use it in my C++-to-Python wrapper.

But as I was looking, yesterday, I came across a problem. I was trying to expose a leaf class of a large hierarchy, and encountered a problem when calling a method of the base class that wasn't overridden in the exposed class.

Here is the stripped down example of what I was trying to do:

#include <iostream>

class A
{
public:
    A() : x(1) {}
    virtual void f() { std::cout << "A::f" << std::endl; }
    virtual void g() { std::cout << "A::g" << std::endl; }
            void h() { std::cout << "A::h" << std::endl; }
            void i() { std::cout << "A::i" << std::endl; }

    int x;
};

class B : public A
{
public:
    virtual void f() { std::cout << "B::f" << std::endl; }
            void h() { std::cout << "B::h" << std::endl; }
};

#include "pybind11/pybind11.h"

namespace py = pybind11;

PYBIND11_PLUGIN(test)
{
    py::module m("test");

//  py::class_<A> a(m, "A");

    py::class_<B>(m, "B"/*, a*/)
        .def(py::init<>())
        .def("f", &B::f)
        .def("g", &B::g)
        .def("h", &B::h)
        .def("i", &B::i)
        .def_readwrite("x", &B::x)
    ;
}

If I then import the resulting library in Python, and try to use it, I get the following:

>>> import test
>>> b = test.B()
>>> b.i()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Incompatible function arguments. The following argument types are supported:
    1. (A) -> None

The same goes for calling 'g' and accessing 'x'.

I tried the same with Boost.Python, and it does not seem to have this problem. (Which is a pitty, 'cause again, I really like this slim header-only library, instead of Boost ;-) )

EDIT: Oh, yes, almost forgot. If you remove the 2 comments, and expose the A class and the fact that is is B's base class, everything works as it should.

[wjakob]

This is kind of a unusual use case: why expose a child class with functions from parents, without being willing to at least create a very rough map with stubs of the parent hierarchy?

Changing the library to address this will require changes that significantly complicate the interface. IMHO, the failure of pybind11 to accept this kind of thing is preferable to a more complex implementation.

[wjakob]

also, by the way: if you absolutely need to do this, you can always always do a cast yourself using a trivial lambda function:

        .def("i", [](B &b) { b.i(); })

[YannickJadoul]

Well, I don't want to expose the whole hierarchy as API for the Python library (I'm trying to wrap code that isn't mine). The Python user should not really know about all the underlying classes (i.e. A), as far as I'm concerned, but just use the leaf class (B). And it just struck me that this construction did not work, while the extension module nicely compiles.

But I do not have a view on the internals, and must admit/understand that it is not worth the complication of the pybind11 interface. But do you prefer the way it is now, or would you change it if there was a (reasonably) simple solution?

[wjakob]

If there was a simple and clean solution, I would accept it. I don't see a simple way to do it though.

[mlund]

I have a related issue for a class inheriting from Eigen:

struct PointBase : public Eigen::Vector3d {...};

where in c++ I use baseclass functions for read/write access:

PointBase p;
p.x()=1.0
cout << p.x()

Any suggestions how to expose this? Directly to numpy array?

[wjakob]

You could directly return a numpy array or wrap the eigen types + extra hierarchy. See my layerlab project on github for an example of the latter.

[YannickJadoul]

I have just found out that Boost.Python doesn't support assigning a Subclass* to Superclass* either, actually.

void a_by_val(A  a) { std::cout << "Called a_by_val" << std::endl; }
void b_by_val(B  b) { std::cout << "Called b_by_val" << std::endl; }
void a_by_ref(A &a) { std::cout << "Called a_by_ref" << std::endl; }
void b_by_ref(B &b) { std::cout << "Called b_by_ref" << std::endl; }
void a_by_ptr(A *a) { std::cout << "Called a_by_ptr" << std::endl; }
void b_by_ptr(B *b) { std::cout << "Called b_by_ptr" << std::endl; }

B *create_b() { return new B(); }

...
    def("a_by_val", &a_by_val);
    def("b_by_val", &b_by_val);
    def("a_by_ref", &a_by_ref);
    def("b_by_ref", &b_by_ref);
    def("a_by_ptr", &a_by_ptr);
    def("b_by_ptr", &b_by_ptr);
    def("create_b", &create_b, return_value_policy<manage_new_object>());
...

>>> b_ptr = create_b()
>>> a_by_ptr(b_ptr)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    test_boost.a_by_ptr(A)
did not match C++ signature:
    a_by_ptr(A*)
>>> a_by_ref(b_ptr)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    test_boost.a_by_ref(A)
did not match C++ signature:
    a_by_ref(A {lvalue})
>>> a_by_val(b_ptr)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    test_boost.a_by_val(A)
did not match C++ signature:
    a_by_val(A)

So, maybe the question is whether you'd want this behaviour in general, or not. If you do not want this in general, maybe there is an easy way of just forcing a member function to take an argument of the wrapped class?

[wjakob]

ok, thanks for clarifying. I was myself wondering how boost.python could figure this out at runtime if one of the types is never explicitly referenced.

Generally you'll have to write a small wrapper function which replaces the type in question, as in

 .def("i", [](B &b) { b.i(); })

This function could be written manually in each case, or automatically using a template-based approach.

However, it is really beyond the scope of this library, so I'm closing the ticket.

[YannickJadoul]

Ok, fair enough. There is a simple workaround, anyway. Thanks.

I did notice another minor thing, while looking at the code, though, in the generated signature doc-string:

    m.def("before", &f);
    py::class_<X>(m, "Y")
        .def(py::init<>())
    ;
    m.def("after", &f);

results in

    after(...)
        Signature : (Y) -> None

    before(...)
        Signature : (X) -> None

as the detail::descr d = cast_in::name(kw.data(), def.data()); in cpp_function forces the nested descr instances of the tuple to be evaluated to strings.

I suppose it would be very uncommon to change the name of a class ánd have a circular dependency between classes (or some other reason why you cannot move 'before' after the class). And even then, it's only the documentation.
But just so you know, then ...

[wjakob]

This behavior is intentional. Often the type may be some C++ template (e.g. Eigen::Matrix<Float, ...>) that looks very alien in a docstring. The idea is that you bind the class at the beginning (before instantiating functions that use it) and give it a more sane name that will be used in the Python API (like MatrixXf)

[YannickJadoul]

Ok, yes, so even if you have a circular dependency between 2 classes and their methods, you could first bind the two classes, by calling defonly later?

class B;
class A { public: void do(B &b); }
class B { public: void do(A &b); }

py::class_<A> cX(m, "X");
py::class_<B> cY(m, "Y");

cX.def("do", &A::do);
cY.def("do", &B::do);

Right. Not completely dummy proof (apparently), but it is indeed a solution for the rare case you have something like this.

[wjakob]

(yep, that is what I meant)