Start using Cython and providing Cython/nogil API

Start transforming Pygolang from being pure Python project into project
that uses both Python and Cython and provides both Python and Cython API
to its users.

For Cython API there is a goal for it to be usable independently of
Python GIL - i.e. independently of Python runtime and limitations
imposed by Python's global interpreter lock. The motivation for this
work is wendelin.core v2, which, due to its design, would deadlock if it
tries to take the GIL in its pinner thread.

This commit brings bare minimum establishing libraries and build system.
It:

- Introduces C/C++ library libgolang. Libgolang will be serving most of
  nogil functionality and Pyx API will be a small wrapper over it. At
  this step Libgolang comes only with C-level panic;

- Introduces Pyx package golang that is aliased to Pyx package
  golang._golang . Users can do `from golang cimport ...` similarly to how
  they do py-level `from golang import ...`. At this step golang.pyx
  only wraps panic and provides way to transform C-level panic into Python
  exception;

- Introduces Py package golang.pyx.build . This package, similarly to
  golang.org/pkg/go/build, serves as the system to build Pyx packages -
  both external packages that use Pygolang in pyx mode, and to build
  Pygolang itself. The build, internally, is served by setuptools_dso
  and cython. An example how to use golang.pyx.build from external
  project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
  tests to verify golang.pyx.build functionality;

- Introduces Pygolang build dependency to cython and setuptools_dso.

.gitignore

@@ -6,6 +6,10 @@ build/
 
 *.o
 *.so
+*.so.*
+*.dylib
+*.dll
+*.pyd
 
 /dist/
 *.egg-info

MANIFEST.in

@@ -1,3 +1,5 @@
-include COPYING README.rst CHANGELOG.rst tox.ini
-recursive-include golang    *.py
+include COPYING README.rst CHANGELOG.rst tox.ini pyproject.toml
+include golang/libgolang.h
+include golang/runtime/libgolang.cpp
+recursive-include golang    *.py *.pxd *.pyx
 recursive-include gpython   *.py

README.rst

@@ -1,6 +1,6 @@
-========================================
- Pygolang - Go-like features for Python
-========================================
+===================================================
+ Pygolang - Go-like features for Python and Cython
+===================================================
 
 Package `golang` provides Go-like features for Python:
 
@@ -11,8 +11,12 @@ Package `golang` provides Go-like features for Python:
 - `defer` allows to schedule a cleanup from the main control flow.
 - `gimport` allows to import python modules by full path in a Go workspace.
 
+Package `golang.pyx` provides__ similar features for Cython/nogil.
+
+__ `Cython/nogil API`_
+
 Additional packages and utilities are also provided__ to close other gaps
-between Python and Go environments.
+between Python/Cython and Go environments.
 
 __ `Additional packages and utilities`_
 
@@ -162,13 +166,54 @@ will import either
 
 located in `src/` under `$GOPATH`.
 
+
+Cython/nogil API
+----------------
+
+Cython package `golang` provides *nogil* API with
+features that mirror corresponding Python package. Cython API is not only
+faster compared to Python version, but also, due to *nogil* property, allows to
+build concurrent systems without limitations imposed by Python's GIL. All that
+while still programming in Python-like language. Brief description of
+Cython/nogil API follows:
+
+`panic` stops normal execution of current goroutine by throwing a C-level
+exception. On Python/C boundaries C-level exceptions have to be converted to
+Python-level exceptions with `topyexc`. For example::
+
+   cdef void _do_something() nogil:
+      ...
+      panic("bug")   # hit a bug
+
+   # do_something is called by Python code - it is thus on Python/C boundary
+   cdef void do_something() nogil except +topyexc:
+      _do_something()
+
+   def pydo_something():
+      with nogil:
+         do_something()
+
+
+See |libgolang.h|_ and |golang.pxd|_ for details of the API.
+See also |testprog/golang_pyx_user/|_ for demo project that uses Pygolang in
+Cython/nogil mode.
+
+.. |libgolang.h| replace:: `libgolang.h`
+.. _libgolang.h: https://lab.nexedi.com/kirr/pygolang/tree/master/golang/libgolang.h
+
+.. |golang.pxd| replace:: `golang.pxd`
+.. _golang.pxd: https://lab.nexedi.com/kirr/pygolang/tree/master/golang/_golang.pxd
+
+.. |testprog/golang_pyx_user/| replace:: `testprog/golang_pyx_user/`
+.. _testprog/golang_pyx_user/: https://lab.nexedi.com/kirr/pygolang/tree/master/golang/pyx/testprog/golang_pyx_user
+
 --------
 
 Additional packages and utilities
 ---------------------------------
 
 The following additional packages and utilities are also provided to close gaps
-between Python and Go environments:
+between Python/Cython and Go environments:
 
 .. contents::
    :local:

golang/.gitignore

@@ -0,0 +1 @@
+/_golang.cpp

golang/__init__.pxd

@@ -0,0 +1,29 @@
+# cython: language_level=2
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <[email protected]>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""Package golang.pyx provides Go-like features for Cython/nogil and runtime for golang.py.
+
+See _golang.pxd for package documentation.
+"""
+
+# redirect cimport: golang -> golang._golang
+#
+# we do this because we cannot put pyx code into __init__.pyx - else Python and
+# other tools (e.g. setuptools) fail to recognize golang/ as Python package.
+from golang._golang cimport *

golang/__init__.py

@@ -26,6 +26,7 @@
 - `gimport` allows to import python modules by full path in a Go workspace.
 
 See README for thorough overview.
+See also package golang.pyx which provides similar functionality for Cython nogil.
 """
 
 from __future__ import print_function, absolute_import
@@ -42,12 +43,9 @@
 from golang._pycompat import im_class
 
 
-# panic stops normal execution of current goroutine.
-def panic(arg):
-    raise _PanicError(arg)
-
-class _PanicError(Exception):
-    pass
+from ._golang import    \
+    _PanicError,            \
+    pypanic     as panic
 
 
 # @func is a necessary decorator for functions for selected golang features to work.

golang/_golang.pxd

@@ -0,0 +1,62 @@
+# cython: language_level=2
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <[email protected]>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""Package golang.pyx provides Go-like features for Cython/nogil and runtime for golang.py.
+
+Cython/nogil API
+----------------
+
+- `panic` stops normal execution of current goroutine by throwing a C-level exception.
+
+Everything in Cython/nogil API do not depend on Python runtime and in
+particular can be used in nogil code.
+
+See README for thorough overview.
+See libgolang.h for API details.
+See also package golang.py which provides similar functionality for Python.
+
+
+Golang.py runtime
+-----------------
+
+In addition to Cython/nogil API, golang.pyx provides runtime for golang.py:
+
+- Python-level panic is represented by pypanic.
+"""
+
+
+# nogil pyx-level golang API.
+#
+# NOTE even though many functions may panic (= throw C++ exception) nothing is
+# annotated with `except +`. Reason: `f() except +` tells Cython to wrap every
+# call to f with try/catch and convert C++ exception into Python one. And once
+# you have a Python-level exception you are in Python world. However we want
+# nogil golang.pyx API to be usable without Python at all.
+#
+# -> golang.pyx users need to add `except +topyexc` to their functions that are
+# on the edge of Python/nogil world.
+cdef extern from "golang/libgolang.h" namespace "golang" nogil:
+    void panic(const char *)
+    const char *recover()
+
+
+# ---- python bits ----
+
+cdef void topyexc() except *
+cpdef pypanic(arg)

golang/_golang.pyx

@@ -0,0 +1,61 @@
+# -*- coding: utf-8 -*-
+# cython: language_level=2
+# distutils: language = c++
+# distutils: depends = libgolang.h
+#
+# Copyright (C) 2018-2019  Nexedi SA and Contributors.
+#                          Kirill Smelkov <[email protected]>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""_golang.pyx provides Python interface to libgolang.{h,cpp}.
+
+See _golang.pxd for package overview.
+"""
+
+from __future__ import print_function, absolute_import
+
+from cpython cimport PY_MAJOR_VERSION
+from cython cimport final
+
+# ---- panic ----
+
+@final
+cdef class _PanicError(Exception):
+    pass
+
+# panic stops normal execution of current goroutine.
+cpdef pypanic(arg):
+    raise _PanicError(arg)
+
+# topyexc converts C-level panic/exc to python panic/exc.
+# (see usage in e.g. *_pyexc functions in "misc")
+cdef void topyexc() except *:
+    # TODO use libunwind/libbacktrace/libstacktrace/... to append C-level traceback
+    #      from where it panicked till topyexc user.
+    # TODO install C-level traceback dump as std::terminate handler.
+    #
+    # recover_ is declared `except +` - if it was another - not panic -
+    # exception, it will be converted to py exc by cython automatically.
+    arg = recover_()
+    if arg != NULL:
+        pyarg = <bytes>arg
+        if PY_MAJOR_VERSION >= 3:
+            pyarg = pyarg.decode("utf-8")
+        pypanic(pyarg)
+
+cdef extern from "golang/libgolang.h" nogil:
+    const char *recover_ "golang::recover" () except +

golang/libgolang.h

@@ -0,0 +1,85 @@
+#ifndef _NXD_LIBGOLANG_H
+#define _NXD_LIBGOLANG_H
+
+// Copyright (C) 2018-2019  Nexedi SA and Contributors.
+//                          Kirill Smelkov <[email protected]>
+//
+// This program is free software: you can Use, Study, Modify and Redistribute
+// it under the terms of the GNU General Public License version 3, or (at your
+// option) any later version, as published by the Free Software Foundation.
+//
+// You can also Link and Combine this program with other software covered by
+// the terms of any of the Free Software licenses or any of the Open Source
+// Initiative approved licenses and Convey the resulting work. Corresponding
+// source of such a combination shall include the source code for all other
+// software used.
+//
+// This program is distributed WITHOUT ANY WARRANTY; without even the implied
+// warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+//
+// See COPYING file for full licensing terms.
+// See https://www.nexedi.com/licensing for rationale and options.
+
+// Library Libgolang provides Go-like features for C and C++.
+//
+// Library Libgolang provides Go-like
+// features. The library consists of high-level type-safe C++ API,
+// and low-level unsafe C API.
+//
+// The primary motivation for Libgolang is to serve as runtime for golang.pyx -
+// - Cython part of Pygolang project. However Libgolang is independent of
+// Python and should be possible to use in standalone C/C++ projects.
+//
+// Brief description of Libgolang API follows:
+//
+// C++-level API
+//
+//  - `panic` throws exception that represent C-level panic.
+//
+// For example:
+//
+//      if (<bug condition>)
+//          panic("bug");
+//
+//
+// C-level API
+
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+// DSO symbols visibility (based on https://gcc.gnu.org/wiki/Visibility)
+#if defined _WIN32 || defined __CYGWIN__
+  #ifdef BUILDING_LIBGOLANG
+    #define LIBGOLANG_API __declspec(dllexport)
+  #else
+    #define LIBGOLANG_API __declspec(dllimport)
+  #endif
+#elif __GNUC__ >= 4
+    #define LIBGOLANG_API __attribute__ ((visibility ("default")))
+#else
+    #define LIBGOLANG_API
+#endif
+
+
+// ---- C-level API that is always available ----
+// (most of the functions are documented in libgolang.cpp)
+
+#ifdef  __cplusplus
+namespace golang {
+extern "C" {
+#endif
+
+#ifdef __cplusplus
+    [[noreturn]]
+#else
+    _Noreturn
+#endif
+LIBGOLANG_API void panic(const char *arg);
+LIBGOLANG_API const char *recover(void);
+
+#ifdef __cplusplus
+}}
+#endif
+
+#endif  // _NXD_LIBGOLANG_H