Hyphen allows one to access Haskell modules from Python (3 or better). (More precisely, it allows one to access Haskell modules compiled with GHC from CPython.) It is in some sense the dual of the cpython package on Hackage, which allows Haskell code to access Python modules.
For instance:
>>> import hyphen, hs.Prelude
>>> hs.Prelude.drop(1, [1,2,3])
<hs.GHC.Types.[] object of Haskell type [GHC.Integer.Integer], containing '[2,3]'>
>>> list(hs.Prelude.drop(1, [1,2,3])) # Convert back to Python list
[2, 3]
>>> hs.Prelude.id(3)
3
The other obvious portmanteau is 'Pascal', which is taken.
For a guide to building Hyphen, see BUILDING.md. Hyphen has been successfully built and used on Mac OS X, Ubuntu, Windows (32 bit and 64 bit) and Windows+Cygwin (32 bit and 64 bit).
Once you have imported hyphen, you can import Haskell modules as though they were Python modules with an hs prefix, so for example
>>> import hyphen
>>> import hs.Prelude
>>> import hs.Data.Text
The functions defined in these modules can now be called from
Python. For instance, the Haskell function Prelude.drop can be
referred to from python as hs.Prelude.drop. The only slightly fiddly
thing is that some Haskell functions can have names that Python won't
accept. (So for instance there's a function Prelude.(+), but Python
won't let you refer to hs.Prelude.(+).) In such cases, you can find
these symbols as, for example, hs.Prelude._["+"]. (Note that
hs.Prelude._["drop"] works too.)
(Quick note: so far we're just talking about how to access Haskell
code from installed modules. In this usage pattern, the Haskell code
that you want to access from Python should have Cabal install it to
the (system or user) library, and you'll be able to import it from
there. Hyphen also supports just having a file Whatever.hs in the
same directory as your python script and importing functions from
there into python: see 'non-library modules' below for more.)
When you call a Haskell function, you can call it with arguments that are themselves Haskell objects, but you can also call it with python objects as arguments, in which case we will try to marshall those arguments into Haskell objects before calling the Haskell function. For instance
>>> hs.Prelude._["+"] (1, 2)
3
>>> hs.Prelude.sum([1,2,3]) # list converted to Haskell list
6
>>> hs.Prelude.drop(5, "Hello, world")
", world"
In these three cases, the return value has been marshalled back from Haskell to Python. Our philosophy is to aggressively try to marshall parameters to Haskell types, but only to convert the very simplest types (String, Text, Int, Integer, Float, Complex) back to Python types on return. More complicated objects stay as Haskell objects. For instance:
>>> hs.Prelude.drop(1, [1,2,3])
<hs.GHC.Types.[] object of Haskell type [GHC.Integer.Type.Integer], containing '[2,3]'>
These Haskell objects can be used from python in many ways. The basic way of using them is to pass them to further Haskell functions; and this certainly works, for example:
>>> my_list = hs.Prelude.drop(1, [1,2,3])
>>> hs.Prelude.sum(my_list)
5
But we can also do other things. For instance, if the return value is a Haskell list, its python representation will be a python iterable:
>>> for x in my_list:
... print(x)
...
2
3
...and if the return value is a Haskell Map or HashMap, the resulting Haskell object will behave like a python dict...
>>> import hs.Data.Map
>>> my_map = hs.Data.Map.fromList([(1, 'Hello'), (2, 'World')])
>>> my_map[1]
'Hello'
>>> print(sorted([key for key in my_map]))
[1, 2]
...and if the return value is a function object, we can call it. (Indeed, the Haskell function objects we get returned by calling Haskell-functions-that-return-functions are the exact same kind of thing as the Haskell function objects that get imported into modules when we import hs.something; in other words, they're the same as the objects we've been calling so far.)
>>> my_func = hs.Prelude.const(4)
>>> my_func
<hyphen.HsFunObj object of Haskell type b_0 -> GHC.Integer.Type.Integer>
>>> my_func('Hello')
4
In a similar vein, if the Haskell object is in Cmp or Eq then the
corresponding python object will support comparisons or equality
tests, and if the Haskell object is hashable, the python object will
be too.
As you'd expect, you can partially apply any function just by calling it with fewer than the full number of arugments, and you'll get a function which accepts the remaining arguments later.
In addition, Haskell objects that represent IO actions can be induced to actually perform the action by calling .act on them. This returns whatever the return type of the action might be.
>>> hs.Prelude.putStrLn("Test") # Construct IO action, but don't perform it
<hs.GHC.Types.IO object of Haskell type GHC.Types.IO ()>
>>> hs.Prelude.putStrLn("Test").act()
Test
<hs.GHC.Tuple.() object of Haskell type (), containing '()'>
It goes without saying that it is important to remember to call .act: if your code doesn't seem to be doing what it's meant to be doing, it's possible that you're constructing an IO action but then discarding it without performing it!
Finally, if we find a a Haskell type T and a function f :: T -> <something> which is defined in the same module, then this gives
rise to a member function on the Python representation of objects of
type T, such that foo.f(*args) means the same as f(foo, *args). For example, if we have Haskell code in the module Test as
follows:
data Test = Test Integer deriving (Typeable, Show)
extract_number :: Test -> Integer
extract_number (Test i) = i
make_sum :: Test -> Integer -> Integer
make_sum (Test i) j = i + j
Then from python we can do:
>>> import hyphen
>>> hyphen.find_and_load_haskell_source()
>>> from hs.Test import Test
>>> my_test_obj = Test(3)
>>> my_test_obj
<hs.Test.Test object of Haskell type Test.Test, containing 'Test 3'>
>>> my_test_obj.extract_number
<bound method Test.extract_number of <hs.Test.Test object of Haskell type Test.Test, containing 'Test 3'>>
>>> my_test_obj.extract_number()
3
>>> my_test_obj.make_sum(4)
7
Names of Haskell functions that begin with an underscore are exempt
from this rule because we don't want users to accidentally end up
defining python members like __getitem which can change the behavior
of an object quite dramatically. If you want to create such members
deliberately (which you might be: perhaps you're trying to use Haskell
to build a python type that has interesting non-standard behaviors),
then you can escape the exemption as follows: if you define a member
is Haskell with a name like hy__<type-name>__<something>__, then
this will be used to create a python member called __<something>__.
For instance, continuing the example above, if the Haskell code continues
hy__Test__getitem__ :: Test -> Integer -> Integer
hy__Test__getitem__ (Test i) j = i + j
Then the python example could have continued
>>> my_test_obj[5]
8
We have already seen some examples of how objects are marshalled from Haskell to Python. A key point about this process is that a single Python object could be used to construct Haskell objects of various different types, depending on type expected by the Haskell function which we're applying. For instance:
>>> import hyphen, hs.Prelude, hs.Data.Text
>>> hs.Prelude.drop(6, "Hello world") # Python string -> Haskell String
'world'
>>> hs.Data.Text.drop(6, "Hello world") # Python string -> Haskell Text
'world'
>>> hs.Prelude.drop(1, (1, 2)) # Python tuple -> Haskell list
<hs.GHC.Types.[] object of Haskell type [GHC.Integer.Type.Integer], containing '[2]'>
>>> hs.Prelude.snd((1, 2)) # Python tuple -> Haskell tuple
2
On the other hand, you can apply polymorphic Haskell functions to Python objects, and the type of the Python object to which we apply the function will be used to determine what Haskell type should be used for the polymorphic arguments; for instance:
>>> hs.Prelude._['+'] (1, 2) # Select Integer version
3
>>> hs.Prelude._['+'] (1+0j, 2+3j) # Select Complex Float version
(3+3j)
>>> hs.Prelude.id([1, 2, 3]) # Invoke version of id for lists of integers
<hs.GHC.Types.[] object of Haskell type [GHC.Integer.Type.Integer], containing '[1,2,3]'>
When a Python object could have been conerted into multiple Haskell types, we will 'break the tie' and convert it to some preferred type:
>>> hs.Prelude.id((1, 2)) # Prefer to convert Python tuples to Haskell tuples, not lists
<hs.GHC.Tuple.(,) object of Haskell type (GHC.Integer.Type.Integer, GHC.Integer.Type.Integer), containing '(1,2)'>
>>> hs.Prelude.id((1, "Test")) # Prefer to convert Python strings to Haskell Text
<hs.GHC.Tuple.(,) object of Haskell type (GHC.Integer.Type.Integer, Data.Text.Internal.Text), containing '(1,"Test")'>
This is not foolproof however; for instance, we get an error in the following case:
>>> hs.Prelude._['+'] (1, 2+3j)
Traceback (most recent call last):
...
TypeError: Incompatible types: cannot resolve object of type
a -> a -> a
to type
GHC.Integer.Type.Integer -> Data.Complex.Complex GHC.Types.Float -> a
Before we close this section, we'll cover two other behaviors of the marshalling code that are important or useful.
One key point is that Python functions can be marshalled into Haskell functions. For example:
>>> hs.Prelude.foldr((lambda x, y: x + y), 0, [1, 2, 3])
6
Although you should be careful when doing this; Haskell idioms make full use of Haskell's ability to have infinite stack depth, whereas Python has a finite stack depth. This can cause problems in examples like:
>>> hs.Prelude.foldr((lambda x, y: x + y), 0, range(10000))
...
RuntimeError: maximum recursion depth exceeded while calling a Python object
Similarly, Python functions can be marshalled into IO actions. In this
case the Python function will be called with no arguments. For
example, we can take the usual 'replicate' function in the Prelude,
force its type to be Int -> IO Text -> IO [Text] (see the section on
types below for more on this), then play with it as follows:
>>> hs.Prelude.replicate
<hyphen.HsFunObj object of Haskell type GHC.Types.Int -> a -> [a]>
>>> specialized_repl = hs.Prelude.replicate.subst(a=hs.Prelude.IO(hs.Data.Text.Text()))
>>> specialized_repl
<hyphen.HsFunObj object of Haskell type GHC.Types.Int -> GHC.Types.IO Data.Text.Internal.Text -> [GHC.Types.IO Data.Text.Internal.Text]>
>>> specialized_repl(4, (lambda : input()))
<hs.GHC.Types.[] object of Haskell type [GHC.Types.IO Data.Text.Internal.Text]>
>>> hs.Prelude.sequence(specialized_repl(4, (lambda : input())))
<hs.GHC.Types.IO object of Haskell type GHC.Types.IO [Data.Text.Internal.Text]>
>>> hs.Prelude.sequence(specialized_repl(4, (lambda : input()))).act()
Fee
Fi
Fo
Fum
<hs.GHC.Types.[] object of Haskell type [Data.Text.Internal.Text], containing '["Fee","Fi","Fo","Fum"]'>
(The reason we have to force the type is that otherwise we have no way of knowing that we should marshall the Python function into an IO action, nor what that IO action should return.)
The other useful behavior is that if we can marshall a python object
to a Haskell object of type T, then we can generally also marshall
it to Haskell type Maybe T (in which case we insert an implicit
Just); we can also marshall python None to any Haskell type Maybe x, in which case we render it as Nothing.
For example:
>>> import hyphen, hs.Prelude, hs.Data.Maybe, hs.Data.Text
>>> identity_on_Maybe_Text = hs.Prelude.id.subst(a=hs.Data.Maybe.Maybe(hs.Data.Text.Text()))
>>> identity_on_Maybe_Text
<hyphen.HsFunObj object of Haskell type GHC.Base.Maybe Data.Text.Internal.Text -> GHC.Base.Maybe Data.Text.Internal.Text>
>>> identity_on_Maybe_Text("Hello")
<hs.GHC.Base.Just object of Haskell type GHC.Base.Maybe Data.Text.Internal.Text, containing 'Just "Hello"'>
>>> identity_on_Maybe_Text(None)
<hs.GHC.Base.Nothing object of Haskell type GHC.Base.Maybe Data.Text.Internal.Text, containing 'Nothing'>
In general, the goal with hyphen is that things Just Work (TM). I'll leave it to the reader to decide whether that goal is met in general, but there's one really nasty case where it seems pretty impossible to make things Just Work in the best possible way. So now I'll give the single most important warning about using Hyphen: you must not construct reference loops that consist of Haskell and python objects. If you do this, then neither the Haskell garbage collection nor the Python garbage collection will be able to collect the objects in the loop, because neither will be able to see the 'whole loop' and recognize that it can be collected.
To be clear, it's perfectly fine to have Python objects refer to Haskell objects (whenever you get a return value from a Haskell function which is not marshalled back to Python, and you store that result somewhere in a Python object, you're building Python objects which refer to Haskell objects. It's perfectly fine to have Haskell objects refer to Python objects. (When you marshall a python closure over to Haskell for use as a Haskell function), then that closure (and hence, indirectly, any python objects which that closure refers to) will now be referred to from Haskell. It's perfectly fine, even, to have Haskell objects refer to Python objects which themselves refer to Haskell objects which refer to Python objects which refer to Haskell objects and so on. What you can't do is have (say) a Haskell object refer to a Python object which refers to the original Haskell object.
In general, the preferred way of accessing Haskell code from Python via hyphen is to install the Haskell code as a ghc-pkg visible library; such libraries can always be directly imported into Python with hyphen (as we have been doing so far).
But you might want to have some Haskell source code in the same directory as your python source, and then magically import Haskell functions from that source. This is possible, but there are some limitations. The basic issue is that to ensure type-safety, the hyphen system must enforce that we only ever compile and import from source once per program run. (Reason: Haskell will recompile the source if it has changed, which might lead us to have binary-incompatible objects floating around which the type system thinks can be freely inter-substituted, leading to Problems.) So we have to have a system for deciding which Haskell modules we want compiled as part of this 'one shot' compilation.
We provide two options for this.
Option one is enabled by calling
hyphen.find_and_load_haskell_source(). This looks in the directory of
the running script for Haskell source, and (if any is found) we will
also check subdirectories for more source, and (if any is found in a
subdirectory) we check sub-sub directories recursively. All the source
we find, we try to compile; if there are .hs files lying around which
are not valid source, we will get errors. Once compiled, we can import
the contents of these files from the hs.* namespace as usual.
This is meant to cover the case when you're writing a little script and you want to import a little bespoke Haskell routine. It assumes that you can control the contents of the directory where your script lives; not an unreasonable assumption.
Option two is not really recommended. It covers the case where you're
writing a python library which is being imported from somewhere on the
python path and which (in turn) wants to import a little Haskell piece
of code that lives in the same directory. (As we've said, the
recommended way of handling this is to install the Haskell code as a
Haskell library visible form ghc-pkg, then to import it from
python... but we'll assume this isn't possible for you.) To enable
option two, call find_and_load_haskell_source(check_full_path=True)
as your python library module is imported. This will check the entire
python path for Haskell files (using the same rules as were used to
check the script directory above, including recursively reading
subdirectories). We then compile them and again they may be
imported. This is (a) somewhat slow, and (b) runs the risk that we'll
come across an .hs file somewhere in the path which isn't valid
Haskell and die.
So far, we've talked a lot about what happens to objects imported from
Haskell modules. But Haskell modules can also contain type
constructors and data constructors. What happens to them? The answer
is that they are both transformed into python type objects. So if we
have a Haskell type constructor with a couple of data constructors, we
will end up with a little class hierarchy on the Python side. For
example, if the Haskell module Test has:
data Example = ExampleWithInt Int
| ExampleWithString String deriving (Typeable, Show)
we'll end up with python classes arranged as follows
hyphen.HsObj (base class of all Haskell objects viewed from python via hyphen)
|
\--- hs.Test.Example
|
\---- hs.Test.ExampleWithInt
|
\---- hs.Test.ExampleWithString
As we can see from Python:
>>> hs.Test.Example
<class 'hs.Test.Example'>
>>> type(hs.Test.Example)
<class 'type'>
>>> hs.Test.ExampleWithInt
<class 'hs.Test.ExampleWithInt'>
>>> hs.Test.ExampleWithString
<class 'hs.Test.ExampleWithString'>
>>> hs.Test.ExampleWithString.__bases__
(<class 'hs.Test.Example'>,)
>>> hs.Test.ExampleWithInt.__bases__
(<class 'hs.Test.Example'>,)
We can call the data constructors to make Haskell objects of the relevant types:
>>> hs.Test.ExampleWithInt(1)
<hs.Test.ExampleWithInt object of Haskell type Test.Example, containing 'ExampleWithInt 1'>
>>> hs.Test.ExampleWithString("hello")
<hs.Test.ExampleWithString object of Haskell type Test.Example, containing 'ExampleWithString "hello"'>
Things behave as you'd expect
>>> isinstance(hs.Test.ExampleWithInt(1), hs.Test.ExampleWithInt)
True
>>> isinstance(hs.Test.ExampleWithInt(1), hs.Test.ExampleWithString)
False
>>> isinstance(hs.Test.ExampleWithInt(1), hs.Test.Example)
True
>>> type(hs.Test.ExampleWithInt(1))
<class 'hs.Test.ExampleWithInt'>
It's also worth remarking that you can take apart a Haskell object to see what arguments were applied to its constructor as follows:
>>> hs.Test.ExampleWithInt(1)._components
(1,)
One potentially confusing thing: you need to realize that, for data constructors which take no parameters, there's a big difference between the python representation of the data constructor itself and the value you get when you call it. (Whereas in Haskell they're basically the same thing.) For instance:
>>> hs.Prelude.LT
<class 'hs.GHC.Types.LT'>
>>> hs.Prelude.LT()
<hs.GHC.Types.LT object of Haskell type GHC.Types.Ordering, containing 'LT'>
>>> type(hs.Prelude.LT)
<class 'type'>
>>> type(hs.Prelude.LT())
<class 'hs.GHC.Types.LT'>
Another thing to bear in mind is that since a Haskell module's
type/type constructor namespace and object namespace are completely
separate things, it is possible to have the same name defined in both
namespaces (in such cases, because of capitalization rules, the object
in this case will necessarily be a Data constructor). Python has onyl
a single namespace, so we must somehow resolve this contention. We
generally do this by creating a lightweight 'doublet' object; then you
can do hs.Foo.Bar.MyName.the_tycon or hs.Foo.Bar.MyName.the_dacon
to specify exactly what you mean. There is one exception to this,
which is when you have a type constructor with only one data
constructor, and which has the same name as the type constructor. This
is a common pattern in Haskell and it's annoying to have to do the
.the_dacon stuff in this case. So in this case alone we have
MyName refer to the data constructor, which is probably what you
mean, and allow you to write MyName.the_type to get the type
constructor.)
We've just described how hyphen creates python types to represent
Haskell type constructors and data constructors. Every Haskell
object viewed from python via hyphen will be given a python type using
these types. (Exception: if the type constructor is invisible because
it's not exported from anywhere, then the type will just be HsObj.)
If follows from all this that the Python type of a Haskell object
viewed through hyphen can be rather different to its Haskell type. On
the one hand, the Python type will depend on exactly which data
constructor was used. On the other hand, if the Haskell type is
something like Map Int Int, the python type will not pick up the
arguments (Int and Int) provided to the type constructor; the
python type will just be hs.Data.Map.Map, and so:
>>> map1 = hs.Prelude.id({1 :1, 2:2})
>>> map1
<hs.Data.Map.Base.Map object of Haskell type Data.Map.Base.Map GHC.Integer.Type.Integer GHC.Integer.Type.Integer, containing 'fromList [(1,1),(2,2)]'>
>>> map2 = hs.Prelude.id({1:'one', 2:'two'})
>>> map2
<hs.Data.Map.Base.Map object of Haskell type Data.Map.Base.Map GHC.Integer.Type.Integer Data.Text.Internal.Text, containing 'fromList [(1,"one"),(2,"two")]'>
>>> type(map1) == type(map2)
True
If you care to know the full Haskell type of a Haskell object viewed
from hyphen, we can certainly do that; it's visible as the .hstype
member. This will be a python object of python type hyphen.HsType:
>>> map1.hstype
hs.Data.Map.Map(hs.GHC.Integer.Integer(), hs.GHC.Integer.Integer())
>>> str(map1.hstype) # NB: str(...) representation easier to read than repr, more closely matches Haskell notation
'<hyphen.HsType object representing Data.Map.Base.Map GHC.Integer.Type.Integer GHC.Integer.Type.Integer>'
>>> str(map2.hstype)
'<hyphen.HsType object representing Data.Map.Base.Map GHC.Integer.Type.Integer Data.Text.Internal.Text>'
>>> map1.hstype == map2.hstype
False
HsType objects can be manipulated in a few ways. You can break them
up by calling the head and tail members. (head will return a
triple giving the name/module/package key of a type constructor, or
just a string in the case that the head of the type is a type
variable). You can build HsTypes by invoking type constructors (pass
other HsTypes as the parameters, and strings for type variables). To
build HsTypes representng type variables do HsType('foo') (or
HsType('foo', kind='* -> *') etc. for variables of more complicated
kinds), and if you want to create an odd HsType with a type variable
as its head and a nontrivial tail (legal Haskell, but very odd), you
can do something like HsType('a', hs.Prelude.Integer(), kind="*->*")).
>>> map1.hstype.head
('Map', 'Data.Map.Base', 'conta_LKCPrTJwOTOLk4OU37YmeN')
>>> map1.hstype.tail
(hs.GHC.Integer.Integer(), hs.GHC.Integer.Integer())
>>> import hs.Data.Map
>>> hs.Data.Map.Map('a', 'b')
hs.Data.Map.Map(hyphen.HsType("a"), hyphen.HsType("b"))
>>> hyphen.HsType("a")
hyphen.HsType("a")
>>> hyphen.HsType("a", kind="* -> *")
hyphen.HsType("a", kind="* -> *")
>>> hyphen.HsType("a").head
'a'
>>> hyphen.HsType('a', hs.Prelude.Integer(), kind="*->*")
hyphen.HsType("a", hs.Prelude.Integer(), kind="* -> *")
In this connexion, it's important when there are nullary type constructors in play to be mindful of the difference between the python object we create to represent the type constructor and the HsType you get by invoking it:
>>> hs.Prelude.Integer
<class 'hs.GHC.Integer.Type.Integer'>
>>> type(hs.Prelude.Integer)
<class 'type'>
>>> hs.Prelude.Integer()
hs.Prelude.Integer()
>>> str(hs.Prelude.Integer())
'<hyphen.HsType object representing GHC.Integer.Type.Integer>'
>>> type(hs.Prelude.Integer())
<class 'hyphen.HsType'>
There are a few other interesting things to mention. One is that you
can call subst on HsTypes to provide substitutions for type
variables therein. You can call kind to get the kind of the
type. And you can call fvs to get the free variables (as a
dictionary mapping variables to strings representing their kinds).
>>> my_hstype = hs.Prelude.id.hstype
>>> my_hstype
hs.GHC.Prim._['(->)'](hyphen.HsType("a"), hyphen.HsType("a"))
>>> str(my_hstype)
'<hyphen.HsType object representing a -> a>'
>>> my_hstype.fvs
{'a': '*'}
>>> my_hstype.kind
'*'
>>> my_hstype2 = my_hstype.subst(a=map1.hstype)
>>> str(my_hstype2)
'<hyphen.HsType object representing Data.Map.Base.Map GHC.Integer.Type.Integer GHC.Integer.Type.Integer -> Data.Map.Base.Map GHC.Integer.Type.Integer GHC.Integer.Type.Integer>'
Once you know how to construct and play with HsType objects, this
also allows you to do some new things with HsObj. You can also call
subst on HsObjs, in order to substitute for the type variables that
occur in their types, and you can call narrow_type to narrow an
object to a specific type.
>>> int_identity = hs.Prelude.id.subst(a=hs.Prelude.Int())
>>> int_identity
<hyphen.HsFunObj object of Haskell type GHC.Types.Int -> GHC.Types.Int>
>>> int_identity('Foo')
Traceback (most recent call last):
...
TypeError: an integer is required (got type str)
>>> int_identity(1)
1
>>> hs.Prelude.id.narrow_type(int_identity.hstype)
<hyphen.HsFunObj object of Haskell type GHC.Types.Int -> GHC.Types.Int>
We try to be sensible about converting Haskell exceptions to Python exceptions whenever they escape from Haskell code. For instance:
>>> hs.Prelude.quot(1, 0)
Traceback (most recent call last):
...
ZeroDivisionError: divide by zero
(here the Haskell DivideByZero was converted to a python
ZeroDivisionError). Python exceptions raised by python functions
that were marshalled into Haskell functions will propagate through
Haskell code and finally come out to the python code on the other
side.
A brief point about the implementation of hyphen. Basically, hyphen consists of two rather separate componnents. The first component is a fairly low-level bridge between Haskell and Python. This low-level bridge is written mostly in Haskell with a little bit of C. The second component is built on top of this and gives the high-level bridge we've just described. This second part is in fact implemented in Python. There are two nice results of this. First, if something goes wrong in the high-level layer, you'll get a nice traceback (at least until the problem hits the low-level layer), which may help you to diagnose what's up and what can be done to fix it. Second, the high level layer is full of hooks that can be manipulated in Python: so if you, the Python user, want to add functionality (say) to marshall python objects to some particular Haskell type in some particular way, that's totally something you can do, from pure python, by just customizing the high-level layer via its hooks.
If this is something that interests you, the best way to start is by
reading the sources of the high level layer, especially
marshall_ctor.py, marshall_obj_to_hs.py and
marshall_obj_to_py,py. There are comments in each of those files
which provide a guide for customizing behavor.
For more documentation on the low-level layer, see LOWLEVEL.md.
The previous section describes the implementation of hyphen in terms of a low-level layer sitting on a high level layer. This allows us to segue into the question of efficiency. Obviously, if you call a Haskell function from Python via hyphen then once we actually get through to running Haskell code, things will be just as efficient as running any other Haskell code anywhere else. So as long as you don't have context switches between Haskell and python inside your inner loop (the intended case for using hyphen is that you should not have such things!),then you're probably fine.
If you do want to do lots of context switches in an inner or nearly-inner loop, then you might have a problem. The python high-level layer which handles marshalling of objects between the two langauges is not designed to be super fast, so your code may be slow. If it's not possible to avoid having context switches close to your inner loop, you can try bypassing the high-level layer and using the low-level layer directly. If you do that, then things will be pretty efficient: if you invoke a Haskell function from Python using the low-level layer, then it's basically a type check followed by a direct jump from python to a C function and from that C function to the Haskell machine code. So it should be reasonably fast.
One final point in this connection is that, even going via the low-level layer, resolving polymorphic Haskell objects down to monomorphic ones is reasonably expensive. So you should resolve any polymorphism once and for all, outside your inner loop, then call the monomorphic functions from inside the inner loop.
Python notoriously has a locking structure, called the Global Interpreter Lock or GIL, which must be held whenever the python interpreter is doing anything. Hyphen allows you to optionally release this lock before calling in to Haskell code, and re-acquire it on returning from the Haskell code. This is often a good idea (it allows other python threads to make progress while the Haskell code is doing its work); but it might not be (releasing and re-aquiring the GIL is a little expensive and if all your Haskell calls return quickly and/or your python code isn't multithreaded anyway, then there's no benefit to releasing the GIL).
If you want to have hyphen release the GIL while running Haskell code,
do hyphen.hslowlevel.set_GIL_mode_fancy(). If you want it not to
bother to release the GIL, do hyphen.hslowlevel.set_GIL_mode_lazy.
If you want to know the current state, do
hyphen.hslowlevel.get_GIL_mode() (which will return 'fancy' or
'lazy' as appropriate).
This section only applies on unix-like OSs.
Sometimes, the operating system will send a signal to a running
program, telling it (say) to terminate, or to interrupt what it's
doing. For instance, when you press ctrl+C in the console, the
runnning program recieves a interrupt signal. Python allows you to
install a Python signal handler for any signal, which is basically
some python code which will get executed when the signal is sent by
the operating system. A default handler is installed for the interrupt
signal, which basically raises a KeyboardInterrupt exception which
then propagates outward through your Python code, causing the code to
stop unless the exception is caught.
Basically the same story is true in Haskell: Haskell allows you to
install Haskell signal handler for any signal, which is basically
some Haskell code which will get executed when the signal is sent by
the operating system. Again, a default handler is installed for the
interrupt signal, which basically raises a UserInterrupt exception
which then propagates outward as before.
The wrinkle in all this is the following. It turns out that the way Python implements python signal handlers goes like this. The OS provides a low-level primtive that lets you install some C code to run when a signal is delivered (we'll call that an OS signal handler). Unfortunately, what you're allowed to do inside an OS signal handler is extremely limited. So when you ask python to install a python signal handler, python will (in turn) install an OS signal handler, but all that handler does is to set a flag somewhere. Then python constantly polls that flag as it interprets code, and if it's set, Python suspends whatever execution is currently in progress and rushes to call the (python) signal handler code.
(The same is true for Haskell.)
What does this have to do with Hyphen? Well, when you use hyphen to
jump into Haskell code, then while that Haskell code is running,
Python's OS signal handler is still installed, but no one is checking
the flag. So if you (say) ctrl-C your program, the flag will get set,
but the program will not stop running because no one will check the
flag, and so KeyboardInterrupt will not be raised, so the program
will not stop. (Until, that is, the Haskell routine returns and Python
finally checks the flag.) This is Not Ideal.
There are two ways around this. One option is to have Haskell
periodically check python's 'has a signal been recieved' flags, and
(if a signal has been recieved), go service the python signal
handler. If the signal handler (in turn) raises an exception (like
KeyboardInterrupt), then we propagate that exception to the running
Haskell code, which will pass it up the stack, and the exception will
eventually escape to the python code that called the Haskell code,
which is what we want. This is generally the best option, but it does
require some multithreadying overhead on the Haskell side (we have to
spawn another Haskell thread just to service the Python signal
handlers), and while multithreading in Haskell is cheap compared to
python, it's not free. (Another proviso is that Haskell will only
check the flag every 50ms or so, which is not nearly as often as
Python would check it normally.)
An alternative is the following. While Haskell code is running, we can
install Haskell's OS signal handler together with a Haskell signal
handler that processes ctrl-C. (We then restore python's OS signal
handler at the end of the Haskell code.) Then, if a ctrl-C is received
during the executing of some Haskell code, Haskell's flag will be
set to say that there's a signal waiting, and (since Haskell is
constantly polling that flag while Haskell code is running), Haskell
will notice that ctrl-C was pressed, which will invoke the Hasell
signal handler, which will raise UserInterrupt, which will propagate
up through the Haskell code and finally be translated to
KeyboardInterrupt when it crosses over into python. This again will
have the effect that our Haskell code is properly interrupted, and we
will be polling the flag much more frequently than once every
100ms. The disadvantage is that which signal handlers run depends on
whether the signal arrives while Python or Haskell code is
running. This may be quite confusing! But if all the signal handlers
actually end up doing is raising UserInterrupt or
KeyboardInterrupt, it's probably managable.
(A final alternative is to do nothing, and just live with the fact that signals that are recieved while hyphen is executing Haskell code will not be processed by python until the Haskell code returns. This is obviously the lowest-overhead option.)
If you want to have hyphen service python signal handlers while
Haskell code is running, do
hyphen.hslowlevel.set_signal_mode_python(). If you want to have
hyphen replace the OS signal handler while Haskell code is running so
that Haskell signal handlers will process any signals received while
Haskell code is running, do
hyphen.hslowlevel.set_signal_mode_haskell(). If you want to be lazy
and not check for signals at all until Haskell returns control to
python, do hyphen.hslowlevel.set_signal_mode_lazy(). If you want to
know the current state, do hyphen.hslowlevel.get_signal_mode()
(which will return 'python', 'haskell' or 'lazy' as
appropriate).
