Improving the external function feature

[lpil]

Gleam features FFI in the form of "external functions". It also has conditional compilation (if erlang {) so that Gleam projects that support multiple targets can use FFI with one external function definition per target.

/// A function that prints a string.
pub fn print(s: String) -> Nil {
  do_print(s)
}

if erlang {
  external fn do_print(String) -> Nil =
    "app_ffi" "print"
}

if javascript {
  external fn do_print(String) -> Nil =
    "./app_ffi.mjs" "print"
}

Problems with the current design

There are several small annoyances with the current design.

No way to infer and communicate the supported targets

If writing a library that only supports one target there are two options, either wrap all the code in each module in conditional compilation, or to not use conditional compilation at all. Both approaches have downsides.

Adding conditional compilation to each module is tedious and error prone, and if the package is added as a dependency to a project using an unsupported target then when it is compiled a cryptic error will be produced about a random value or type not existing for modules imported from that package.

Not using conditional compilation is less irritating to the package author, but using the package with an unsupported target will result in either a runtime error for JavaScript, or a compile time error for Erlang from the Erlang compiler.

In both instances the user gets a very confusing error, and we don't have a good way to identify when a package is not compatible with a target.

Diverging module interfaces

With the current system is it possible and easy for modules to expose different functions and types depending on the target, either intentionally or by accident.

if erlang {
  pub external fn read_file(String) -> Result(BitString, Nil) =
    "app_ffi" "read_file"
}

if javascript {
  pub external fn read_file(String) -> Result(String, Nil) =
    "./app_ffi.js" "read_file"
}

In this example one implementation returns a BitString and the other returns a String. In a real module with much more code it would be harder to spot this, and there is no aid from the compiler here.

We want to discourage modules having different interfaces for different targets as it makes them harder and more confusing to use, and we want it to be easy for the author to keep the interfaces consistent.

Verbosity

Ignoring comments, in the first example there 13 lines, and only 3 of them convey any information about the code (the function signature, and the source of the external implementations).

The function signature is written 3 times, and the function name is written 4 times (albeit with the meaningless do_ prefix for 3 of those times).

No good home for documentation

Documentation is written in /// comments in Gleam on the line above the item they document.

Due to there being multiple implementations of the same function there is no one place for the documentation to be placed, so it has to be duplicated, or a wrapper function has to be written for the documentation to be attached to, as in the example above.

Duplicating documentation is verbose and tedious, and is it easy for each comment to get out-of-sync. Wrapping functions is easier to manage but also verbose compared to documenting one function, and it has a small runtime performance cost.

Labelled argument syntax confusion

This one is not specific to multi-target projects, but a common mistake with external functions is to give the arguments labels unintentionally.

pub external fn print(s: String) -> Nil =
  "app_ffi" "print"

In this code the argument has been given the label s, but often the programmer intended to give the argument the name s and leave it unlabelled. This confusion likely stems from the syntax for an external function's argument labels is the same as a regular function's argument name.

The proposal

The proposal is to remove the dedicated "external" syntax, and instead have a single function declaration syntax that can be used for both regular functions and external functions, with explicitly listed targets. This syntax would be a replacement for both the external function syntax and conditional compilation blocks.

Note: The syntax here is a placeholder. We can determine the exact syntax later if we decide this new approach is a good idea.

Here is the first example in the current system and then the new proposed system.

if erlang {
  pub external fn read_file(String) -> Result(BitString, Nil) =
    "app_ffi" "read_file"
}

if javascript {
  pub external fn read_file(String) -> Result(String, Nil) =
    "./app_ffi.js" "read_file"
}

/// A function that prints a string.
@external(erlang, "app_ffi", "print")
@external(javascript, "./app_ffi.mjs", "print")
pub fn print(s: String) -> Nil 

A function header is used to declare the function's name and its type signature, but it is absent a body. Attributes are used to declare the external implementations for the function, each explicitly listing the target.

The syntax is much more concise, and it is clear where the documentation should be placed. There is no longer a difference in syntax between regular functions and external functions, so the confusion around labelled arguments is removed.

When compiling this code we can check the given implementations and return a helpful error if the current target is not supported.

We can determine the supported targets for a package by scanning the source code and keeping track of the targets that have been declared for all FFI. This information could be presented on https://packages.gleam.run/ and permit uses to search for packages that support their desired targets.

Mixing regular and external function definitions

In some cases we wish to use an external definition only for some targets, using a pure Gleam implementation for others. One such example is the list.length function, which is implemented in Gleam for JavaScript, but uses runtime's length function for Erlang as it is implemented in C and is faster.

@external(erlang, "erlang", "length")
pub fn length(xs: List(a)) -> Int {
  fold(xs, 0, fn(_, acc) { acc + 1 })
}

Here the function has a body as well as an external implementation for Erlang. For the Erlang target the external implementation will be used, and for other targets the pure Gleam implementation will be used.

External types

Given external would no longer a keyword for function I think we would also want to remove it as a keyword for types, instead specifying the type with no constructors.

// Current syntax
pub external type Timer

// Proposed syntax
pub type Timer

Future uses for the attribute syntax

An attribute syntax as proposed here can also be used for other features such as marking functions as deprecated, and supplying aliases for use in generated HTML documentation's search.

Problems with the proposal

As raised by @tynanbe sometimes platform specific code isn't directly using native target code via external functions, instead it is using different target-specific Gleam modules.

if erlang {
  import gleam/erlang/os.{start_arguments as argv}
}

if javascript {
  import gleam/nodejs.{argv}
}

pub fn main() {
  let arguments = argv()
}

This proposal is currently insufficient to support this. What could we do here?

[tynanbe]

Perhaps @ (or whatever easily spotted symbol) should always prefix a compiler-level keyword. These could be expanded upon later (e.g. @macro, @pure, etc.).

If conditional compilation is to remain, it could then use the @target x expr form, (or @erlang expr, etc.).

Then the @target keyword could work for both conditional blocks and function attributes.

const target =
  @erlang "Erlang"
  @javascript "Javascript"
  @_ "Unknown"
  
pub type NoReturn

/// Halts the runtime.
///
@erlang external "erlang" "halt"
@javascript external "" "process.exit"
pub fn exit() -> NoReturn {
  NoReturn
}

or

@target erlang import gleam/erlang/os

/// Returns a list of argument strings passed when invoking the program.
///
@target javascript external "./mod.mjs" "arguments"
pub fn arguments() -> List(String) {
  @target erlang { os.start_arguments() }
  @target _ { [] }
}

I also wonder about the distinction between function attributes and current language keywords. Perhaps all functions should use only the fn keyword, and anything else that modifies them should be in the form of function attributes (e.g. @public).

[lpil]

Using @target or such for conditional compilation is a good idea I think, the current approach is a bit unusual. @target would look similar to Rust's #[cfg(target = ...)], C's #ifdef ..., Erlang's -ifdef(...) etc.

I don't want to change the syntax for making values public, the current one is picked for it being generally familiar to people.

I'd like to surround arguments with (), to keep it in keeping with Gleam and other C styled languages. I think this too will be more familiar to people.

On the naming: Keywords are bare words, so if it has a prefix it wouldn't be a keyword. Attribute is the name for metadata attached to code in all the languages I checked so I think it makes sense to go with the general convention there.

[pwtail]

Hi @lpil,

felt like adding my 5 cents here. Why is it a problem to call "external" functions? Is it because of 2 different targets supported - erlang and javascript? Because if we're speaking about the erlang target only, as far as I know, both erlang and gleam are compatible without any intermediate layer required.

I think it would be best if one could just import whatever function from an erlang module and call it. Without any "external" declarations needed. Are there any troubles with it?

[lpil]

Hello! As you say we certainly want to be able to import and call any Erlang or JavaScript function, and that is possible both with the current system and the proposed one. There are some problems with the current system, as detailed in my post above. An alternative design may be able to resolve the issues, which is what is attempted here with this proposal.

[pwtail]

What I meant is making erlang imports absolutely transparent, so that @erlang external wasn't needed. Why declare an exit function that will be the same as erlang's halt function if we can import it

import erlang/stdlib.{halt as exit}. Btw, does gleam support this syntax - importing a function under a different name?

[lpil]

Yup that syntax is supported!

External functions are regular imports, the only difference is that you have to give the functions type annotations as it is not possible to type check Erlang code.

[pwtail]

Ok I get it.

[NthTensor]

I really like the way this unifies external and regular functions. I guess we could do something similar with types.

@external(javascript)
pub type Date {
  Date(milliseconds: Int)
}

@external(javascript, "./date_ffi.mjs", "parse")
pub fn parse(str: String) -> Date {
  // implement 8601 date parsing in gleam
}

This is safe if you treat Date as opaque except in the body of external functions (right?).

Allowing different imports on different platforms seems like a headache. I would prefer platform-specific blocks, maybe something like the following:

import gleam/erlang/os.{start_arguments as erlang_argv}
import gleam/nodejs.{argv as node_argv}

pub fn main() {
  let arguments = @target {
    javascript -> node_argv()
    erlang -> erlang_argv()
    _ -> []
  } 
}

A case style conditional compilation block would let us enforce the general principle that all type signatures must be the same on all platforms, even if the constructors and specialized implementations are not.

[lpil]

Not allowing different imports per target would mean it's not possible to write a package that unifies two different target specific packages, and not being able to implement platform specific private functions would mean that certain programs cannot be written. For example those that use a helper function to implement a tail call optimised recursive function. I think we would need to be less restrictive than that.