ctypes stanza

[mbacarella]

as discussed in #135

Rather than developing this privately and pushing a gigantic changeset of completed code, I thought it might be better to iterate on this as publicly as possible so we can all learn together 😇

So far, there's just enough here to register an extension, "ctypes", and require its declaration in dune-project as (using ctypes 0.1). It also places a stub entry in library that doesn't quite do anything.

[mbacarella]

@rgrinberg is it right to have ctypes handled in the Buildable section for library?

[rgrinberg]

is it right to have ctypes handled in the Buildable section for library?

Buildable is shared between libraries and executables. So it's only right if there's some notion of ctypes for executables. In other words, it's probably better to stick to libraries only for now.

[mbacarella]

My naive, first pass conclusion is that the ctypes stanza should work by expanding into two additional library stanzas, two or three executable stanzas, and some rules. And it'll have to generate some ml files too.

So, at first blush, I would try going into Dune_file.parse and for each Library stanza with a defined ctypes field, hand-construct additional Library, Executable and Rule stanzas to tack onto the end.

That sound right?

[rgrinberg]

So, at first blush, I would try going into Dune_file.parse and for each Library stanza with a defined ctypes field, hand-construct additional Library, Executable and Rule stanzas to tack onto the end.

Seems good. To generate executables, you can use exe.ml directly. Instead of using the rule stanza, you can use our rule api - see examples in simple_rules.ml.

[nojb]

@mbacarella If it is not too much work, could you write down a sample "ctypes"-enabled stanza and what it would elaborate to, in terms of stanzas without "ctypes" ? It would help clarify things for those of us which are not very familiar with Ctypes itself. Thanks!

[mbacarella]

mbacarella If it is not too much work, could you write down a sample "ctypes"-enabled stanza and what it would elaborate to, in terms of stanzas without "ctypes" ? It would help clarify things for those of us which are not very familiar with Ctypes itself. Thanks!

@nojb Sure, let's use a real world example.

If you look at https://github.com/mbacarella/mpg123/blob/main/dune right now, it's this:

(library
  (public_name mpg123)
  (libraries ctypes mpg123.c))

It leans pretty heavily on the c/dune file with this somewhat daunting set of boilerplate.

(rule
 (targets c_flags.sexp c_flags.txt c_library_flags.sexp)
 (action  (run ../config/discover.exe)))

(library
  (name mpg123_c_type_descriptions)
  (public_name mpg123.c_type_descriptions)
  (modules Mpg123_c_type_descriptions)
  (libraries ctypes))

(executable
  (name type_gen)
  (modules type_gen)
  (libraries ctypes.stubs ctypes.foreign mpg123_c_type_descriptions))

(rule
  (with-stdout-to c_generated_types.c
    (run ./type_gen.exe)))

(rule
  (targets c_generated_types.exe)
  (deps (:c c_generated_types.c))
  (action (system "%{cc} %{c} \
    -I '%{lib:ctypes:.}' \
    -I %{ocaml_where} \
    %{read:c_flags.txt} \
    -o %{targets}")))

(rule
  (with-stdout-to mpg123_c_generated_types.ml
    (run ./c_generated_types.exe)))

(library
  (name mpg123_c_function_descriptions)
  (public_name mpg123.c_function_descriptions)
  (modules Mpg123_c_generated_types Mpg123_c_function_descriptions
           Mpg123_c_types)
  (wrapped false)
  ; c_generated_types.exe emits code with unused vars and unbound record labels
  ;  so, we supress them here
  (flags (:standard -w -27 -w -9))
  (libraries ctypes mpg123_c_type_descriptions))

(executable
  (name function_gen)
  (modules Function_gen)
  (libraries ctypes.stubs mpg123_c_function_descriptions))

(rule
  (with-stdout-to c_generated_functions.c
    (run ./function_gen.exe c mpg123_stub)))

(rule
  (with-stdout-to mpg123_c_generated_functions.ml
    (run ./function_gen.exe ml mpg123_stub)))

(library
  (name mpg123_c)
  (public_name mpg123.c)
  (libraries ctypes mpg123_c_function_descriptions)
  (modules Mpg123_c Mpg123_c_generated_functions)
  (foreign_stubs
    (language c)
    (names c_generated_functions)
    (flags (:include c_flags.sexp)))
  (c_library_flags (:include c_library_flags.sexp)))

(Not even included here is the discover script in the config sub-directory.)

I would like that all to go away and for the top-level dune file to become:

(library
  (public_name mpg123)
  (ctypes
    (pkg_config "mpg123")
    (c_headers ("mpg123.h"))
    (modules Mpg123_c_type_descriptions Mpg123_c_function_descriptions)
    (generated_modules
      (mpg123_c -> entry_point)
      (mpg123_c_types -> (types Mpg123_c_type_descriptions.Types))
      (mpg123_c_generated_functions -> (functions Mpg123_c_function_descriptions.Functions)))))

Something like that.

All the user has to supply here is the mpg123_c_type_descriptions.ml file and the mpg123_c_function_descriptions.ml file (basically, filling in functors required by ctypes) and then they get a neatly wrapped module mpg123_c where they can access all of their wrapped raw Types and Functions and build their more cooked interface on top of that.

Probably pkg_config would actually be more like lib_resolver:

type lib_resolver =
  | Pkg_config of string
  | My_library of (c_flags * c_library_flags)

so you can use either the system pkg-config if your library is standard enough, or manually provide the compile and link flags if you've vendored your library or are using a proprietary system library.

Probably c_headers should change to something like:

type c_headers =
  | Include_h of string
  | Preamble of string list
  | Preamble_file of Path.t

[mbacarella]

@rgrinberg Instead of using the rule stanza, you can use our rule api - see examples in simple_rules.ml.

I'm not quite following. I'm looking at Simple_rules expecting to find utilities that will make it easy for me to construct a Stanza.Rule.t, but it seems to operate at a different level so I'm probably misunderstanding something fundamental.

Let me sketch out more of what I'm trying to do in pseudocode here ec9e8c1

[nojb]

From a brief glance, I feel you should be looking at a ctypes stanza along the lines of the menhir stanza which takes care of setting up rules for the generated files, and leave it to the user to use the generated modules (and C stubs) as she sees fit (in libraries, executables, etc).

This is semantically simpler and easy to reason about, instead of trying to embed ctypes directly into the library stanza.

[rgrinberg]

This is semantically simpler and easy to reason about, instead of trying to embed ctypes directly into the library stanza.

This might be my fault. I suggested embedding ctypes as a field to library because it seems to share almost all fields with the library field.

[rgrinberg]

Why does this need to be in a separate library? Can't it be a part of the user defined library? I think I forgot why we need this mpg123 vs. mpg123.c split.

[mbacarella]

This is actually what luv does, which I adapted. I didn't really question it but yeah, I'm not seeing why it can't simply set up in the library that contains the ctypes stanza.

[rgrinberg]

I also suggest that you start with writing some tests. It helps flesh out what the stanza is going to look like. We have the beginning of a small developer focused manual here which you can refer to. You can also just look at some of the other blackbox tests to get an idea.

[rgrinberg]

I've expanded the manual above with a small section about rules. It's pretty small right now, but if you have any more questions, I'd be happy to document things further.

[mbacarella]

Thanks for the hand-holding, @rgrinberg

My plan was to lexically blow a ctypes stanza up into library, executable and rule stanzas at Dune_load.parse, but I take it from your suggestions to use helper functions that this is better moved into Gen_rules.gen_rules because that's where the context data is available?

[rgrinberg]

but I take it from your suggestions to use helper functions that this is better moved into Gen_rules.gen_rules because that's where the context data is available?

That's not the only reason. Another reason is that you don't want to do this step too eagerly, you want to wait until loading those rules is needed. So you should a clause for your ctypes stanza in gen_rules that will call Ctypes_rules.gen.

[rgrinberg]

Note that the above advice is only relevant if ctypes is a stanza and not a field. If it's a field of library, all the rules should be called from lib_rules. This is where adding some cram tests to demonstrate how you want the stanza to work would be helpful :)

[mbacarella]

This is where adding some cram tests to demonstrate how you want the stanza to work would be helpful :)

Right-o, planning to work some into my next push.

[mbacarella]

Hmm, it's been awhile. Let me push what I have so far.

This version is a naive translation of the by-hand dune files you might write. One problem I'm having is getting the generated modules to resolve when the library stanzas are processed.

I'm actually a bit confused by this, because I think the rules to produce the generated modules are being properly run before the library stanzas are evaluated, but they fail with (e.g.) Module "Foo__c_types" does not exist though this seems to work fine if you ask dune to process a normal dune file boilerplate.

Am I supposed to get them populated into Dir_contents?

[rgrinberg]

I'm actually a bit confused by this, because I think the rules to produce the generated modules are being properly run before the library stanzas are evaluated, but they fail with (e.g.) Module "Foo__c_types" does not exist though this seems to work fine if you ask dune to process a normal dune file boilerplate.

From what I see, you're doing a lot of Module.of_string "foo__bar". That's not the way to go and you should do it Ml_sources. The name mangling of the module names should be done automatically for you so that it doesn't matter if the library is wrapped or not.

[mbacarella]

To be clear, I'm doing Module.of_string "foo__bar" instead of simply calling it "bar" to avoid having the generated modules possibly clobber any other user modules, not to reproduce the behavior of wrapped.

What do you mean by use Ml_sources? The only way to create an Ml_sources.t appears to be by giving it a directory via make.

Asked a different way, when I run Ctypes_rules.gen_rules, it generates some .ml files that should satisfy dependencies of Library.t stanzas that were synthesized earlier. Those files aren't appearing in dir_contents though so it blows up with module not found. Am I doing this rule generation way too late?

[mbacarella]

EDIT: ah, actually, I see there's a spot in Dir_contents now where extensions can specify that they provide generated .ml files. Missed this somehow the first time.

[mbacarella]

When you deem the PR ready, remove the draft label and i will consider for merge.

Alright, fleshed out the documentation and added more test coverage. Please kindly consider.

[rgrinberg]

Manually merged to master.