Check if pybind11 can be used as an intermediate binding between MATLAB and Drake C++

[EricCousineau-TRI]

UPDATE: The scope of this has been expanded to check if we can MATLAB bindings "for free" using pybind11 and MATLAB's Python support, with some extra shims on top.

ORIGINAL: This is issue is to figure out if there is a way to relax pybind11s constraints on passed types, such as a custom marshaling mechanism on integral types, to accept floating-point values when bound to C-type integral types. If this is possible, then we can check to see if we can do a small run-time check to round it to the correct value if within an acceptable eps.

This would permit using the bindings in a more natural fashion within MATLAB, if the pybind11 bindings overall play well with MATLAB and could be used either directly, or within a wrapper.
If it could be used directly, then we could do something like from Russ's PR test program:

MathematicalProgram = py.pydrake.solvers.mathematicalprogram.MathematicalProgram;
prog = MathematicalProgram();
x = prog.NewContinuousVariables(2,'x');   % Note that int32(2) is not necessary

(could figure out if there is a way to fudge this with import statements, or use evalin('caller', ...) for consistent usage in MATLAB code.)

This would permit more natural syntax, where in general MATLAB permits floating-point values as size arguments (such as zeros(n), etc.).

The alternative is to write wrappers, which is certainly doable, but may be burdensome if we could end up just writing aliases.

\cc @RussTedrake @rdeits

[rdeits]

It might be possible to do this with the custom type conversions in pybind11: http://pybind11.readthedocs.io/en/master/advanced/cast/custom.html although I haven't used that mechanism myself.

[RussTedrake]

i've merged my example:
https://github.com/RobotLocomotion/drake/blob/master/drake/matlab/solvers/test/testMathematicalProgram.m

I view the mismatch between numpy arrays and matlab matrices to be much more of a burden than the occasional int32 cast. The most relevant discussion I found about it on a quick search yesterday is here:
https://www.mathworks.com/matlabcentral/answers/157347-convert-python-numpy-array-to-double

[EricCousineau-TRI]

@rdeits Thanks! Will check into that.

@RussTedrake Thanks for the link! Given that issue, I've set the priority to medium (had intended low originally). Please let me know if you'd like me to up the priority.

In terms of timeline, do you have an idea of when would decide on how we should implement our MATLAB bindings?

[RussTedrake]

Based on today's slack poll, it looks like making the systems classes available in Python should be the first priority. If we can make those same bindings provide the solution for matlab, that would be ideal; otherwise we need to at least lay out an alternative plan for our matlab bindings.

So -- I would say that spike testing syntactic solutions to support the numpy array <--> matlab array conversions would be medium priority simply to provide information for that road-mapping. Thanks!

[EricCousineau-TRI]

Simple Problem (accept py.float in lieu of int)

I've added a hack-and-slash proof-of-concept of replacing things like int with a "relaxed" type that will permit floating point values:

• example source - This leverages the type_caster<> approach. pybind11 has an caster for std::is_arithmetic<> types, but it explicitly prevents casting from floating-point to integral (only permits integral -> floating-point).
  • example python test

The motivation here was to see how easy it may be to shim input types, and possibly output types, for the useful problem.

Useful Problem and Potential Caveats (array conversions)

I will dig a little deeper on the numpy array <--> matlab array conversions, to see if we can automate the conversion via something like Christoph Wiedemanns's conversion utilties.

I'd like to avoid copying and pasting and repeating the same interface structure if we can use / automate marshaling for an already existing set of bindings,

Some potential issues that may arise:

• Writing a proxy class in Python may be easiest, given Python's reflection foo abilities. However:
  • As of now, I don't see anything in MATLAB may not permit Python to return proper MATLAB types.
    • The MATLAB Engine API for Python does allow this, but my guess is that this is a separate layer that won't work in MATLAB proper.
  • Special care will be needed to ensure that virtual methods are not covered up by marshaling.
• Alternative: Use reflection in Python or what not, and generate the appropriate MATLAB proxy classes.

NOTE: These are only brief thoughts. I have actually not yet tried this out in MATLAB.

[EricCousineau-TRI]

As a quick test, I fleshed out a variant of the options listed above, proxying Python objects via MATLAB dynamicprops.

Since MATLAB R2016b does not support converting 2D arrays to Python arrays, I've added in the mat2py conversion routines, and made a quick PyProxy class.
Additionally, I've added a pybind11 relaxation for MatrixXd (which can be extended to more general types, if need be), that permits scalars to be used to initialize the matrix. This is useful in MATLAB land, where scalars are 1x1 matrices always.

I have added an example MATLAB script that takes in / modifies a NumPy array, which looks natural in MATLAB and in Python:

• C++ pybind11 stuff with relaxed int / MatrixXd type
• Example MATLAB script with PyProxy and the output
  • This does not interface with pybind11 code, but just shows the proxy working.
• MATLAB Unittest and Python Unittest showing similar behavior with relaxed types / proxying - both pass.
  • These interface with pybind11 code.

Code snippet:

% scratch.py is a simple Python module
pyscratch = pyimport('scratch');
mscratch = PyProxy(pyscratch);
% Construct object
mlo = mscratch.Test('hello');
% Tinker with arrays
mlo.nparray = [5, 10, 15];
mlo.nparray(3) = 20; % Permitted - getter/setter combo in MATLAB permits natural indexing (but slow due to copies)

Caveats:

• Slow. Each Python instance is proxy-fied. This could be semi-optimized by binding the proxy to the class, but that still is a band-aid.
• Complicated and wrapped. Things in PyProxy land should stay in PyProxy land.
• This does not support operator overloads (e.g. for symbolics).
  • I can add those in if it starts to look like this is the easiest path forward.
• All matrices get converted to numpy arrays. May not play nicely with other libraries..
  • (We may want to use numpy.matrix as well, to keep linear algebra consistent.)
• Python-wrapped MATLAB functions are handles, and must be called with parenthesis (my_func()), versus natural MATLAB-style (my_func).
• Does not exactly lend itself to easy inheritance...
  • Generic PyProxy class could be extended, and then additional checks to make sure the basic class is as it should be.
  • May still not handle virtual inheritance like pybind11 does.
    • We could solve this if we can pass a MATLAB function handle to Python. (Haven't seen this function, though...)

Per a discussion with @avalenzu, I'll check to see if there are things in MATLAB that offer any better, more concrete mechanism to do this type-simplifying job.

[rdeits]

Just as a note: Scipy specifically recommends not using numpy.matrix, and I agree with them: https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html#array-or-matrix-which-should-i-use

Since matrices mostly behave like numpy.ndarray, it's easy to accidentally use one when you mean the other, which causes all kinds of chaos. Better to just use ndarray everywhere and use .dot() when you want matrix multiplication. And if .dot() is too verbose, then you can use the brand-new @ infix operator on modern versions of Python.

[EricCousineau-TRI]

Awesome, thanks for sharing! I've updated the above post.

[RussTedrake]

@rdeits -- to be clear, do you think we should bind the existing C++ interfaces taking Eigen matrices twice, once with numpy arrays and once with C++ double arrays. Is that something that we could do relatively cleanly with pybind11, and without cluttering the python drake namespace?

[rdeits]

Er, no. We should just bind them to take and return numpy arrays. I don't think there's any reason to have C++ double arrays around on the Python side.

[RussTedrake]

I'm sorry... I didn't read carefully enough. What I meant is that we bind both numpy.matrix and numpy.ndarray? Or is ndarray an acceptable replacement for a python-only workflow?

[rdeits]

No, the scipy docs suggest that we don't use numpy.matrix at all. numpy.ndarray is already the default everywhere else and it's what new users learn first (for example, numpy.ones() returns an ndarray).

[EricCousineau-TRI]

I've taken another crack at updating some of the MATLAB - Python stuff. Didn't know how comprehensive subsref / subsasgn was, and was able to simplify the proxy role by overloading those, and make NumPyProxy that permits mapping between MATLAB indexing and Python slicing (hopefully with minimal overhead).

I've updated the readme: repro/bindings/README.md

• This has instructions on previewing the proxies (without py_relax_* in C++) using drake in Bazel, with a simple shell script.
  • Redux of testMathematicalProgram.m - this is a little easier to digest, and permits slicing into the VectorXDecisionVariable stuff.

[RussTedrake]

The redux looks very promising!

[EricCousineau-TRI]

Checked on the following items with MathWorks regarding MATLAB (R2017a as of the time of this writing):

1. Better way to pass n-D matrices from MATLAB to Python:
   • Outcome: Unfortunately, this is not supported.
2. Ability to call MATLAB functions from Python from MATLAB (to support virtual inheritance)
   • Outcome: Unfortunately, this is also not supported.

We have an easy workaround to 1, though we should stress test it.

Item 2 may be a bit of a blocker, as that would prevent meaningful System authoring from MATLAB.

Possible workarounds:

• For any interface that has virtual inheritance, somehow implement the call dispatch interfaces in MATLAB - this may quickly become unmaintainable.
• Check if we can use MEX hackery to wrap MATLAB function handles in such a way that pybind11 can access them. (NOTE: This may involve globals and additional type marshaling :( )

[RussTedrake]

For (2), i think that we can write a matlab-specific leafsystem class in c++ that takes only the string of the matlab classfile name in the constructor, then calls mexCallMATLAB to call into matlab directly (obviously this c++ code will need to link against the matlab libraries). That is essentially how the "matlab-only" workflow (which actually ran through a c++ s-function) worked before:
https://github.com/RobotLocomotion/drake/blob/last_sha_with_original_matlab/drake/matlab/systems/DCSFunction.cpp

[EricCousineau-TRI]

Looks awesome!

I may have a simple setup, which builds on top of how DCSFunction works, but would rely more on Python / pybind11 for marshaling that would enable (2) in a generic fashion.
Rather than doing any site-specific MATLAB <-> MEX <-> C++ marshaling ourselves, we can rely on MATLAB <-> Python <-> pybind11 <-> C++, with an option to do MEX <-> C++ (or even supply MEX <-> Python to tightly integrate MATLAB + NumPy matrices) if performance is ever a problem.

This means that we could call any MATLAB method from Python from C++ with only about ~100 lines extra. Then the virtual inheritance could come from generating a MATLAB base class on the fly using Python's reflection (my guess is ~200 lines) to handle which methods are actually virtual, in a generic fashion, such that a user could then use the intended base class with minimal API duplication on our part (just relying on the pybind11 bindings to guide everything).

I am aiming to have a simple prototype of calling a MATLAB method from Python in a few hours, and will post back if/when I get it working.

[EricCousineau-TRI]

Here's a simple prototype of the above setup working, albeit still dirty:

• py_mex
  • matlab_examples.sh - Environment setup to run MATLAB with access to Bazel targets (on actual filesystem, not in a sandbox)
  • callback_example.m - shows a MATLAB callback being passed to Python
  • inheritance_example.m - shows a MATLAB class derived from a Python class, which is derived from a C++ class.
    • C++ class - pure is not actually pure, but a minor detail
    • Python class
      * MATLAB class

NOTE: Only tested on my system (Ubuntu 16.04, MATLAB R2016b)

Cons:

• Works via a hack, passing C function pointers from MATLAB to Python via a MEX function
• Since Python cannot do native reference counting on MATLAB objects (but MATLAB can reference count Python objects), I had to implement a simple reference counter, which leads to simple but nasty reference cycles that require a pseudo-GC step.
  • NOTE: It'd be nice to have weak references, but I have not tested out mexMakeArrayPersistent
• Does not strictly stick to MATLAB's inheritance rules (as they are much more strict that Python's)
• Subject to dependency hell (but similar dependency hell as what we'd deal with in MATLAB + C++).

Pros:

• Generic, not pybind11 specific, nor drake specific.
• MATLAB code can build off of Python code, and vice versa (if Python is run from within MATLAB)

Given that this is a concrete but hacky solution, I will use this to ask for better ways to do this.

[EricCousineau-TRI]

Given that there are prototype solutions of MATLAB-ifying Python w.r.t. pybind (scalar and int/double overloads), and given this is a spike test, I will say this is complete.

The conclusion is yes, C++ -> pybind -> Python -> custom PyProxy -> MATLAB should be sufficient for basic offerings.