move to sdl for dub / REPL mode

[Laeeth]

Point 3.1 below is the quick win. 1 and 2 are easy enough. 3.2 needs thought

0. way to use pdef to wrap entire struct recursively with some UDA to disable members you don't want converted.

1. should allow use of sdl type package format as well as json as it becomes a pain for maintenance having some stuff in sdl and some in json. a hassle, but not hard.

2. if you wrap everything with pdef it won't compile if you are compiling code without pydmagic. it's tiresome to have to put a version(pydmagic) around every pdef. could we have that as a build option. in other words when you build your code from the command line with dub without pydmagic then you should have an option to use the pdefs as sugar way to access pyd. but you should also have a build option to ignore them so you can use the same code as pure D with no pyd/pydmagic stuff going on. when pydmagic compiles it from the notebook, it should automatically enable the appropriate build option. so it's transparent.

1. not sure if its feasible but could we think about some kind of eval or repl mode? this is useful in two ways: 3.1) for quick things, and 3.2) for extended exploration of something.

so 3.1) would avoid having to type in main() and import headers. maybe you could use rdmd for this (it has eval and loop arguments). it's nice if you want to check syntax of something, or see what you actually get from a library function when you call it with some argument. so you can do this:

  rdmd --eval='writefln("%s",sqrt(2.0))'

but not this (it doesn't throw, just doesn't print anything):

  rdmd --eval='sin(100.0)'

you can also do this fwiw (I didn\t know till Andrei pointed it out). may not be helpful for pydmagic though:

  rdmd --loop='writefln("%s",sqrt(line.to!double))'

[user enters] 3.0
[D prints] 1.73205

you could have %pyd_eval and %pyd_loop I guess [if %pyd doesn't take arguments - I don't know jupyter well].

as an aside would be really nice if rdmd worked with dub/code.dlang.org - so you could specify your library in the dub.sdl/dub.json and call it quickly from rdmd.

3.2) at the moment one can maintain state via storing D in python. that's a bit slow which mostly doesn't matter but sometimes might. it also depends on having conversions working and set up for your particular data. and I think if you wrap everything with pdef it won't compile if you are compiling code without pyd. so for example suppose I have:

@pdef auto priceBars(string ticker)
{
  //  go and get the data
  return bars;
}

The line below works in the notebook and will convert bars to python object.
bars=priceBars("NASDAQ/AAPL")

But some D structures may be a pain to write conversions for, and it would be nice to be able to use Jupyter as a pure D repl.

Suppose you wrap D objects as a typed blob. In other words within a %%pyd section at the moment immediate commands are illegal eg:

import std.math;
auto a=sqrt(2.0);
writefln("%s",a);

But within pydmagic you could:

• a wrap the immediate code in a function and call it
• recognize the assignment to a global variable, serialize it (msgpack or something else) and store it as a blob in python, along with its name and type information.

Going back to more realistic example:
auto bars=priceBars("NASDAQ/AAPL");

So bars are serialized to msgpack with name and type. That code might turn in to:

@pdef struct PydImmediateVariable
{
  @pdef string name;
  @pdef string typeinfo;
  @pdef ubyte[] data;
}
@pdef PydImmediateVariable pydMagicImmediateBars000001()
{
  //imports
  auto bars=priceBars("NASDAQ/AAPL");
  return bars.toPydImmediateVariable!"bars"; // I guess alias would keep name
}
PydImmediateVariable(string name,T)(T arg)
{
  import msgpack;
  PydImmediateVariable ret;
  ret.data=pack!T(arg)
 ret.name=name;
  ret.typeinfo=T.stringof; // or something
  return ret;
}

then you have python code that calls pydMagicImmediateBars00001() and sticks return value in a blob hashmap indexed by variable name

Then when someone enters immediate code in the next cell:
prettyPrint(bars);

and prettyPrint is defined as:

  void prettyPrint(PriceBar[] bars)
  {}

you can lookup bars, get the type, deserialize it to the write type and call prettyPrint.

still some overhead, but this way you can write pure D in immediate mode.

thoughts? I think it sounds more complicated than it is to actually do. the bits that would take me some time to figure out would just be getting typetuple of global variables. am I missing a barrier that would prevent this working?

I know it sounds fiddly, but interactivity can be super useful. also for new users.

1. Compiler Error messages. You get a line number, but you can't see line numbers in the ipython notebook for your D code. It shouldn't be hard to find a solution, but I can't think of one immediately.
2. D returning PNG (sounds, videos etc)
   Python code like matlab can display an image in the notebook. Nice for plotting data etc. I haven't yet tried, but I guess the D code could write to a file and then python code could do:
   from IPython.display import Image
   Image(filename='test.png')

That might be cool as part of a demo anyway, but is there a way to do this directly as a blob without going to disk ?