ctypes stanza

CHANGES.md

@@ -340,6 +340,12 @@ Unreleased
 - Do not pass include directories containing native objects when compiling
   bytecode (#4200, @nojb)
 
+- If an .ml file is not used by an executable, Dune no longer report
+  parsing error in this file (#4330, @jeremiedimino)
+
+- Experimental ctypes support (#3905, fixes #135, @mbacarella)
+
+
 2.8.2 (21/01/2021)
 ------------------
 

Makefile

@@ -15,6 +15,7 @@ cinaps \
 coq-native \
 "coq>=8.12.1" \
 core_bench \
+ctypes \
 "csexp>=1.3.0" \
 js_of_ocaml \
 js_of_ocaml-compiler \
@@ -28,6 +29,7 @@ ocamlformat.0.19.0 \
 ppx_inline_test \
 ppxlib \
 result \
+ctypes \
 "utop>=2.6.0"
 
 # Dependencies recommended for developing dune locally,

doc/dune-files.rst

@@ -607,6 +607,11 @@ to use the :ref:`include_subdirs` stanza.
   is useful whenever a library is shadowed by a local module. The library may
   then still be accessible via this root module
 
+- ``(ctypes <ctypes stanza>)`` instructs dune to use ctypes stubgen to process
+  your type and function descriptions for binding system libraries, vendored
+  libraries, or other foreign code.  See :ref:`ctypes-stubgen` for a full
+  reference. This field is available since the 3.0 version of the dune language.
+
 - ``(empty_module_interface_if_absent)`` causes the generation of empty
   interfaces for every module that does not have an interface file already.
   Useful when modules are used solely for their side-effects. This field is
@@ -811,14 +816,21 @@ files for executables. See `executables_implicit_empty_intf`_.
   flag if some of the libraries listed here are not referenced from any of the
   plugin modules.
 
-Linking Modes
-=============
+- ``(ctypes <ctypes stanza>)`` instructs dune to use ctypes stubgen to process
+  your type and function descriptions for binding system libraries, vendored
+  libraries, or other foreign code.  See :ref:`ctypes-stubgen` for a full
+  reference. This field is available since the 3.0 version of the dune language.
+
+- ``(empty_module_interface_if_absent)`` causes the generation of empty
 
-``(empty_module_interface_if_absent)`` causes the generation of empty
+- ``(empty_module_interface_if_absent)`` causes the generation of empty
   interfaces for every module that does not have an interface file already.
   Useful when modules are used solely for their side-effects. This field is
   available since the 3.0 version of the Dune language.
 
+Linking Modes
+=============
+
 The ``modes`` field allows selecting which linking modes will be used
 to link executables. Each mode is a pair ``(<compilation-mode>
 <binary-kind>)``, where ``<compilation-mode>`` describes whether the
@@ -1916,21 +1928,21 @@ Where ``<optional-fields>`` are:
 - ``(libraries <libraries>)`` are libraries that should be
   statically linked in the MDX test executable.
 
-- ``(enabled_if <blang expression>)``  is the same as the 
-  corresponding field of `library`_.
+- ``(enabled_if <blang expression>)``  is the same as the corresponding field
+  of `library`_.
 
-- ``(package <package>)`` specifies which package to attach
-  this stanza to (similarly to when ``(package)`` is attached to a ``(rule)``
-  stanza). When  ``-p`` is passed, ``(mdx)`` stanzas with another package will
-  be ignored. Note that this feature is completely separate from 
-  ``(packages)``, which specifies some dependencies.
+- ``(package <package>)`` specifies which package to attach this stanza to
+  (similarly to when ``(package)`` is attached to a ``(rule)`` stanza). When
+  ``-p`` is passed, ``(mdx)`` stanzas with another package will be ignored.
+  Note that this feature is completely separate from ``(packages)``, which
+  specifies some dependencies.
 
 Upgrading from Version 0.1
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 - The 0.2 version of the stanza requires at least MDX 1.9.0. If you encounter
-  an error such as, ``ocaml-mdx: unknown command `dune-gen'``, then you
-  should upgrade MDX.  
+  an error such as, ``ocaml-mdx: unknown command `dune-gen'``, then you should
+  upgrade MDX.  
 
 - The field ``(packages <packages>)`` is deprecated in version 0.2. You can
   use package items in the generic ``deps`` field instead:

doc/foreign-code.rst

@@ -64,6 +64,215 @@ without the ``.h`` extension. When a library install header files,
 these are made visible to users of the library via the include search
 path.
 
+.. _ctypes-stubgen:
+
+Stub Generation with Dune Ctypes
+================================
+
+Beginning in dune 3.0, it is possible to use the ctypes_ stanza to generate
+bindings for C libraries without writing any C code.
+
+Note that dune support for this feature is experimental and is not subject
+to backward compatibility guarantees.
+
+To use dune ctypes stub generation, you must provide two OCaml modules: a "type
+description" module for describing the C library types and constants, and a
+"function description" module for describing the C library functions.
+Additionally, you must list any C headers and a method for resolving build and
+link flags.
+
+If you're binding a library distributed by your OS, you can use the
+``pkg-config`` utility to resolve any build and link flags.  Alternatively, if
+you're using a locally installed library or a vendored library, you can provide
+the flags manually.
+
+The "type description" module must define a functor named ``Types`` with
+signature ``Ctypes.TYPE``.  The "function description" module must define a
+functor named ``Functions`` with signature ``Ctypes.FOREIGN``.
+
+A toy example
+-------------
+
+To begin, you must declare the ctypes extension in your ``dune-project`` file:
+
+.. code:: scheme
+
+  (lang dune 3.0)
+  (using ctypes 0.1)
+
+
+Next, here is a ``dune`` file you can use to define an OCaml program that
+binds a C system library called ``libfoo``, which offers ``foo.h`` in a
+standard location.
+
+.. code:: scheme
+
+   (executable
+    (name foo)
+    (libraries core)
+    ; ctypes backward compatibility shims warn sometimes; suppress them
+    (flags (:standard -w -9-27))
+    (ctypes
+     (external_library_name libfoo)
+     (build_flags_resolver pkg_config)
+     (headers (include "foo.h"))
+     (type_description
+      (instance Type)
+      (functor Type_description))
+     (function_description
+      (concurrency unlocked)
+      (instance Function)
+      (functor Function_descriptio))
+     (generated_types Types_generated)
+     (generated_entry_point C)))
+
+This stanza will introduce a module named ``C`` into your project, with the
+sub-modules ``Types`` and ``Functions`` that will have your fully bound C
+types, constants and functions.
+
+Given libfoo with the C header file ``foo.h``:
+
+.. code:: c
+
+  #define FOO_VERSION 1
+
+  int foo_init(void);
+
+  int foo_fnubar(char *);
+
+  void foo_exit(void);
+
+Your example ``type_description.ml`` file is:
+
+.. code:: ocaml
+
+  open Ctypes
+
+  module Types (F : Ctypes.TYPE) = struct
+    open F
+
+    let foo_version = constant "FOO_VERSION" int
+  end
+
+Your example ``function_description.ml`` file is:
+
+.. code:: ocaml
+
+  open Ctypes
+
+  (* This Types_generated module is an instantiation of the Types
+     functor defined in the type_description.ml file. It's generated by
+     a C program that dune creates and runs behind the scenes. *)
+  module Types = Types_generated
+
+  module Functions (F : Ctypes.FOREIGN) = struct
+    open F
+
+    let foo_init = foreign "foo_init" (void @-> returning int)
+
+    let foo_fnubar = foreign "foo_fnubar" (string_opt @-> returning int)
+
+    let foo_exit = foreign "foo_exit" (void @-> returning void)
+  end
+
+Finally, the entry point of your executable named above, ``foo.ml``,
+demonstrates how to access the bound C library functions and values:
+
+.. code:: ocaml
+
+  let () =
+    if (C.Types.foo_version <> 1) then
+      failwith "foo only works with libfoo version 1";
+
+    match C.Functions.foo_init () with
+    | 0 ->
+      C.Functions.foo_fnubar "fnubar!";
+      C.Functions.foo_exit ()
+    | err_code ->
+      Printf.eprintf "foo_init failed: %d" err_code;
+  ;;
+
+From here, one only needs to run ``dune build ./foo.exe`` to generate the
+stubs and build and link the example ``foo.exe`` program.
+
+Complete information about the ctypes combinators used above is available at
+the ctypes_ project.
+
+Ctypes stanza reference
+------------------------
+
+The ``ctypes`` stanza can be used in any ``executable(s)`` or ``library``
+stanza.
+
+.. code:: scheme
+
+  ((executable|library)
+    ...
+    (ctypes
+      (external_library_name <package-name>)
+      (type_description
+        (instance <module-name>)
+        (functor <module-name>))
+      (function_description
+        (instance <module-name>)
+        (functor <module-name>)
+        <optional-function-description-fields>)
+      (generated_entry_point <module-name>)
+      <optional-ctypes-fields>)
+    )
+
+- ``type_description``: the ``functor`` module is a description of the C library
+  types and constants written in the ``ctypes`` domain-specific language you
+  wish to bind.  The ``instance`` module is the name the functor will be
+  instantiated under, inserted into the top-level of the
+  ``generated_entry_point`` module.
+
+- ``function_description``: the ``functor`` module is a description of the C
+  library functions written in the ``ctypes`` domain-specific language you wish
+  to bind.  The ``instance`` module is the name the functor will be
+  instantiated under, inserted into the top-level of the
+  ``generated_entry_point`` module. The ``function_description`` stanza can be
+  repeated. This is useful if you need to specify sets of functions with
+  different concurrency policies (see below).
+
+The instantiated types described above can be accessed from the function
+descriptions by referencing them as the module specified in optional
+``generated_types`` field.
+
+``<optional-ctypes-fields>`` are:
+
+- ``(build_flags_resolver <pkg_config|vendored-stanza>)`` tells dune how to
+  compile and link your foreign library.  Specifying ``pkg_config`` will use
+  the ``pkg-config`` tool to query the compilation and link flags for
+  ``external_library_name``. For vendored libraries, provide the build and link
+  flags using ``vendored`` stanza.  If ``build_flags_resolver`` is not
+  specified, the default of ``pkg_config`` will be used.
+
+- ``(generated_types <module-name>)`` is the name of an intermediate module. By
+  default it is named ``Types_generated``. You can use this module to access
+  the types defined in ``Type_description`` from your ``Function_description``
+  module(s).
+
+- ``(generated_entry_point <module-name>)`` is the name of a generated module
+  that your instantiated ``Types`` and ``Function`` modules will instantiated
+  under. We suggest calling it ``C``.
+
+``<optional-function-description-fields>`` are:
+
+- ``(concurrency <sequential|unlocked|lwt_jobs|lwt_preemptive>)`` tells ctypes
+  stubgen whether to call your C functions with the runtime lock held or
+  released. These correspond to the ``concurrency_policy`` type in the
+  ``ctypes`` library. If ``concurrency`` is not specified, the default of
+  ``sequential`` will be used.
+
+``<vendored-stanza>`` is:
+
+- ``(vendored (c_flags <flags>) (c_library_flags <flags>))`` provide the build
+  and link flags for binding your vendored code. You must also provide
+  instructions in your ``dune`` file on how to build the vendored foreign
+  library; see the :ref:`foreign_library` stanza.
+
+
 .. _foreign-sandboxing:
 
 Foreign build sandboxing
@@ -148,3 +357,5 @@ The `re2 project <https://github.com/janestreet/re2>`_ uses this
 method to build the re2 C library. You can look at the file
 ``re2/src/re2_c/dune`` in this project to see a full working
 example.
+
+.. _ctypes: https://github.com/ocamllabs/ocaml-ctypes

nix/default.nix

@@ -40,6 +40,7 @@ let
     "ppxlib"
     "result"
     "utop"
+    "ctypes"
     "${coq}/coq-core.opam"
   ]);