Skip to content

Commit

Permalink
Updated based on feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
@nrc
nrc committed Oct 7, 2016
1 parent e28f7a9 commit 1c2a50d
Showing 1 changed file with 101 additions and 70 deletions.
171 changes: 101 additions & 70 deletions text/0000-proc-macros.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,16 @@ This RFC specifies the architecture of the procedural macro system. It relies on
[RFC 1561](https://github.com/rust-lang/rfcs/pull/1561) which specifies the
naming and modularisation of macros. It leaves many of the details for further
RFCs, in particular the details of the APIs available to macro authors
(tentatively called `libmacro`). See this [blog post](http://ncameron.org/blog/libmacro/)
for some ideas of how that might look.
(tentatively called `libproc_macro`, formerly `libmacro`). See this
[blog post](http://ncameron.org/blog/libmacro/) for some ideas of how that might
look.

[RFC 1681](https://github.com/rust-lang/rfcs/pull/1681) specified a mechanism
for custom derive using 'macros 1.1'. That RFC is essentially a subset of this
one. Changes and differences are noted throughout the text.

At the highest level, macros are defined by implementing functions marked with
a `#[macro]` attribute. Macros operate on a list of tokens provided by the
a `#[proc_macro]` attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.
Expand Down Expand Up @@ -57,13 +62,18 @@ like macro is used with syntax `foo!(...)`, and an attribute-like macro with
`#[foo(...)] ...`. Macros may be used in the same places as `macro_rules` macros
and this remains unchanged.

There is also a third kind, custom derive, which are specified in [RFC
1681](https://github.com/rust-lang/rfcs/pull/1681). This RFC extends the
facilities open to custom derive macros beyond the string-based system of RFC
1681.

To define a procedural macro, the programmer must write a function with a
specific signature and attribute. Where `foo` is the name of a function-like
macro:

```
#[macro]
pub fn foo(TokenStream, &mut MacroContext) -> TokenStream;
#[proc_macro]
pub fn foo(TokenStream) -> TokenStream;
```

The first argument is the tokens between the delimiters in the macro use.
Expand All @@ -75,8 +85,8 @@ The value returned replaces the macro use.
Attribute-like:

```
#[macro_attribute]
pub fn foo(Option<TokenStream>, TokenStream, &mut MacroContext) -> TokenStream;
#[prco_macro_attribute]
pub fn foo(Option<TokenStream>, TokenStream) -> TokenStream;
```

The first argument is a list of the tokens between the delimiters in the macro
Expand All @@ -101,54 +111,93 @@ by another macro without parsing, in which case they do not need to parse. The
distinction is not statically enforced. It could be, but I don't think the
overhead would be justified.

We also introduce a special configuration option: `#[cfg(macro)]`. Items with
Custom derive:

```
#[proc_macro_derive]
pub fn foo(TokenStream) -> TokenStream;
```

Similar to attribute-like macros, the item a custom derive applies to must
parse. Custom derives may on be applied to the items that a built-in derive may
be applied to (structs and enums).

Currently, macros implementing custom derive only have the option of converting
the `TokenStream` to a string and converting a result string back to a
`TokenStream`. This option will remain, but macro authors will also be able to
operate directly on the `TokenStream` (which should be preferred, since it
allows for hygiene and span support).

Procedural macros which take an identifier before the argument list (e.g, `foo!
bar(...)`) will not be supported (at least initially).

My feeling is that this macro form is not used enough to justify its existence.
From a design perspective, it encourages uses of macros for language extension,
rather than syntactic abstraction. I feel that such macros are at higher risk of
making programs incomprehensible and of fragmenting the ecosystem).

Behind the scenes, these functions implement traits for each macro kind. We may
in the future allow implementing these traits directly, rather than just
implementing the above functions. By adding methods to these traits, we can
allow macro implementations to pass data to the compiler, for example,
specifying hygiene information or allowing for fast re-compilation.

## `proc-macro` crates

[Macros 1.1](https://github.com/rust-lang/rfcs/pull/1681) added a new crate
type: proc-macro. This both allows procedural macros to be declared within the
crate, and dictates how the crate is compiled. Procedural macros must use
this crate type.

We introduce a special configuration option: `#[cfg(proc_macro)]`. Items with
this configuration are not macros themselves but are compiled only for macro
uses.

Initially, it will only be legal to apply `#[cfg(macro)]` to a whole crate and
the `#[macro]` and `#[macro_attribute]` attributes may only appear within a
`#[cfg(macro)]` crate. This has the effect of partitioning crates into macro-
defining and non-macro defining crates. Macros may not be used in the crate in
which they are defined, although they may be called as regular functions. In the
future, I hope we can relax these restrictions so that macro and non-macro code
can live in the same crate.
If a crate is a `proc-macro` crate, then the `proc_macro` cfg variable is true
for the whole crate. Initially it will be false for all other crates. This has
the effect of partitioning crates into macro- defining and non-macro defining
crates. In the future, I hope we can relax these restrictions so that macro and
non-macro code can live in the same crate.

Importing macros for use means using `extern crate` to make the crate available
and then using `use` imports or paths to name macros, just like other items.
Again, see [RFC 1561](https://github.com/rust-lang/rfcs/pull/1561) for more
details.

When a `#[cfg(macro)]` crate is `extern crate`ed, it's items (even public ones)
are not available to the importing crate; only macros declared in that crate.
There should be a lint to warn about public items which will not be visible due
to `#[cfg(macro)]`. The crate is dynamically linked with the compiler at
compile-time, rather than with the importing crate at runtime.
When a `proc-macro` crate is `extern crate`ed, it's items (even public ones) are
not available to the importing crate; only macros declared in that crate. There
should be a lint to warn about public items which will not be visible due to
`proc_macro`. The crate is used by the compiler at compile-time, rather than
linked with the importing crate at runtime.

[Macros 1.1](https://github.com/rust-lang/rfcs/pull/1681) required `#[macro_use]`
on `extern crate` which imports procedural macros. This will not be required
and should be deprecated.


## Writing procedural macros

Procedural macro authors should not use the compiler crates (libsyntax, etc.).
Using these will remain unstable. We will make available a new crate, libmacro,
which will follow the usual path to stabilisation, will be part of the Rust
distribution, and will be required to be used by procedural macros (because, at
the least, it defines the types used in the required signatures).
Using these will remain unstable. We will make available a new crate,
libproc_macro, which will follow the usual path to stabilisation, will be part
of the Rust distribution, and will be required to be used by procedural macros
(because, at the least, it defines the types used in the required signatures).

The details of libmacro will be specified in a future RFC. In the meantime, this
[blog post](http://ncameron.org/blog/libmacro/) gives an idea of what it might
contain.
The details of libproc_macro will be specified in a future RFC. In the meantime,
this [blog post](http://ncameron.org/blog/libmacro/) gives an idea of what it
might contain.

The philosophy here is that libmacro will contain low-level tools for
The philosophy here is that libproc_macro will contain low-level tools for
constructing macros, dealing with tokens, hygiene, pattern matching, quasi-
quoting, interactions with the compiler, etc. For higher level abstractions
(such as parsing and an AST), macros should use external libraries (there are no
restrictions on `#[cfg(macro)]` crates using other crates).
restrictions on `#[cfg(proc_macro)]` crates using other crates).

The `MacroContext` is an object passed to all procedural macro definitions. It
is the main entry point to the libmacro API and for interaction with the
compiler. Via the `MacroContext`, a procedural macro can access information
about the context in which it is used and defined, and perform operations which
rely on the state of the compiler. It will be more fully defined in the upcoming
RFC proposing libmacro.
A `MacroContext` is an object placed in thread-local storage when a macro is
expanded. It contains data about how the macro is being used and defined. It is
expected that for most uses, macro authors will not use the `MacroContext`
directly, but it will be used by library functions. It will be more fully
defined in the upcoming RFC proposing libproc_macro.

Rust macros are hygienic by default. Hygiene is a large and complex subject, but
to summarise: effectively, naming takes place in the context of the macro
Expand All @@ -157,8 +206,8 @@ definition, not the expanded macro.
Procedural macros often want to bend the rules around macro hygiene, for example
to make items or variables more widely nameable than they would be by default.
Procedural macros will be able to take part in the application of the hygiene
algorithm via libmacro. Again, full details must wait for the libmacro RFC and a
sketch is available in this [blog post](http://ncameron.org/blog/libmacro/).
algorithm via libproc_macro. Again, full details must wait for the libproc_macro
RFC and a sketch is available in this [blog post](http://ncameron.org/blog/libmacro/).


## Tokens
Expand Down Expand Up @@ -219,12 +268,12 @@ pub enum TokenKind {
// The content of the comment can be found from the span.
Comment(CommentKind),
// Symbol is the string contents, not including delimiters. It would be nice
// `text` is the string contents, not including delimiters. It would be nice
// to avoid an allocation in the common case that the string is in the
// source code. We might be able to use `&'Codemap str` or something.
// `Option<usize> is for the count of `#`s if the string is a raw string. If
// source code. We might be able to use `&'codemap str` or something.
// `raw_markers` is for the count of `#`s if the string is a raw string. If
// the string is not raw, then it will be `None`.
String(Symbol, Option<usize>, StringKind),
String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },
// char literal, span includes the `'` delimiters.
Char(char),
Expand Down Expand Up @@ -269,6 +318,10 @@ pub enum StringKind {
pub struct Symbol { ... }
```

Note that although tokens exclude whitespace, by examining the spans of tokens,
a procedural macro can get the string representation of a `TokenStream` and thus
has access to whitespace information.

### Open question: `Punctuation(char)` and multi-char operators.

Rust has many compound operators, e.g., `<<`. It's not clear how best to deal
Expand All @@ -294,14 +347,14 @@ Some solutions:
## Staging

1. Implement [RFC 1561](https://github.com/rust-lang/rfcs/pull/1561).
2. Implement `#[macro]` and `#[cfg(macro)]` and the function approach to
2. Implement `#[proc_macro]` and `#[cfg(proc_macro)]` and the function approach to
defining macros. However, pass the existing data structures to the macros,
rather than tokens and `MacroContext`.
3. Implement libmacro and make this available to macros. At this stage both old
3. Implement libproc_macro and make this available to macros. At this stage both old
and new macros are available (functions with different signatures). This will
require an RFC and considerable refactoring of the compiler.
4. Implement some high-level macro facilities in external crates on top of
libmacro. It is hoped that much of this work will be community-led.
libproc_macro. It is hoped that much of this work will be community-led.
5. After some time to allow conversion, deprecate the old-style macros. Later,
remove old macros completely.

Expand Down Expand Up @@ -345,7 +398,10 @@ latter can be written today, the former require more work on an interface to the
compiler to be practical).

We could use the `macro` keyword rather than the `fn` keyword to declare a
macro. We would then not require a `#[macro]` attribute.
macro. We would then not require a `#[proc_macro]` attribute.

We could use `#[macro]` instead of `#[proc_macro]` (and similarly for the other
attributes). This would require making `macro` a contextual keyword.

We could have a dedicated syntax for procedural macros, similar to the
`macro_rules` syntax for macros by example. Since a procedural macro is really
Expand All @@ -365,31 +421,6 @@ would require additional rules on token trees and may not be possible.
# Unresolved questions
[unresolved]: #unresolved-questions

### macros with an extra identifier

We currently allow procedural macros to take an extra ident after the macro name
and before the arguments, e.g., `foo! bar(...)` where `foo` is the macro name
and `bar` is the extra identifier. This is used for `macro_rules` and is useful
for macros which define classes of items, rather than instances of items. E.g.,
a `struct!` macro might be used similarly to the `struct` keyword.

My feeling is that this macro form is not used enough to justify its existence.
From a design perspective, it encourages uses of macros for language extension,
rather than syntactic abstraction. I feel that such macros are at higher risk of
making programs incomprehensible and of fragmenting the ecosystem).

Therefore, I would like to remove them from the language. Alternatively, they
could be incorporated into the new design by having another kind of macro
function:

```
#[macro_with_ident]
pub fn foo(&Token, TokenStream, &mut MacroContext) -> TokenStream;
```

where the first argument is the extra identifier.


### Linking model

Currently, procedural macros are dynamically linked with the compiler. This
Expand Down

0 comments on commit 1c2a50d

Please sign in to comment.