Allow macros in types

[ghost]

Allow macros in types (accepted)

rendered text/

---

Rendered draft

cc @aturon @huonw @jroesch

[jroesch]

I think this would be very powerful. Making this type level fragment of the language more ergonomic would not only allow powerful functional programming constructs but the ability to compute information, and enforce powerful invariants at compile time.

You could now imagine parametrizing data types containing statically sized arrays like so:

struct Data<N: Nat> {
   internal_array: Array![Expr!(N + 1)] // or whatever the final syntax would be
}

I have recently wrote a file system in Rust and it would of been very cool to have the type system compute the necessary sizes of differing internal buffers based on parameters like BlockSize, # of blocks, # of INodes and so on.

[ghost]

Following the discussion on IRC, I added a little note about the possibility for optimization for Expr! using partial evaluation or other techniques.

[quantheory]

After a couple of weeks of discussion, seeing this PR finally spurred me to open #884, which GitHub helpfully mentions right above this post. ^^^

That's not an alternative to this proposal per se, but I did want to point out that some of us have been looking at the approach of extending the type system, rather than relying on macros to write code that's generic over type-level numbers.

[ghost]

@quantheory Thanks for pointing out the other RFC. I've added a note to the type-level naturals example with a link to your RFC.

Essentially, I think these proposals are orthogonal, as you suggest. It just happens to be the case that type-level arithmetic (or type-level programming generally) that is currently possible in Rust becomes easier with macros in types. But I don't really intend for this to be a solution (not in any proper sense) to the more specific issue of parameterizing types by numeric values. I think allowing types in macros should just be generally useful, even if a more targeted solution gets implemented for that.

[jroesch]

@quanttheory I'm reading over the discuss thread and the RFC this afternoon, but my initial thought is that by baking in arithmetic into the type system we add a lot of complexity without any generality (other type level values). It may very well be the most pragmatic way forward, but I think it is worth thinking about alternatives.

[quantheory]

@jroesch My short answer is that if you add up the cases that already exist or are planned (constant expressions in arrays, associated const, sizeof) a significant chunk of the complexity is already present or expected to be brought in already for a much less general set of cases. So in that context I would consider the proposal an improvement in the overall complexity/expressiveness ratio, though that doesn't mean that there couldn't be something better.

That said, I do need some time to properly consider this and @freebroccolo's considered response. I would prefer to do any more discussion specific to the const parameter RFC on that request, just to avoid cluttering this PR with anything too off-topic.

[nrc]

What are the effects on hygiene? As I understand it there is no hygienic handling of types in macros at the moment, would allowing macros in type position make the hygiene situation worse? (I have no idea if there is some effect or not, but it seems worth thinking about).

[aturon]

@freebroccolo Just wanted to say, thanks for writing up this RFC! I've been busy with alpha2 but will make sure it gets attention.

cc @nikomatsakis

[ghost]

What are the effects on hygiene? As I understand it there is no hygienic handling of types in macros at the moment, would allowing macros in type position make the hygiene situation worse? (I have no idea if there is some effect or not, but it seems worth thinking about).

@nick29581 This is something I don't have a good answer for since I don't really understand the hygiene stuff in Rust that well. I did attempt to read some of the documentation and literature pointed to but I think someone who is more familiar with the internals will have to comment on that aspect.

[aturon]

ping @nikomatsakis @pnkfelix

[pnkfelix]

i will shepherd

[nikomatsakis]

Hear ye, hear ye. This RFC was moved into Final Comment Period as of Wed, June 10th. (Sorry, forgot to add this notice!)

[gsingh93]

I think you mean Wednesday, June 10th.

On Thu, Jun 11, 2015, 10:22 AM Niko Matsakis [email protected]
wrote:

Hear ye, hear ye. This RFC was moved into Final Comment Period as of
Wed, June 11th. (Sorry, forgot to add this notice!)

—
Reply to this email directly or view it on GitHub
#873 (comment).

[Mr-Byte]

Seeing as how the current system is considered stable and the fact that this RFC would make current macros more useful in more places without really introducing new features to macros themselves (this is really just where they can be used) I'm in favor of this RFC.

I've had several use cases for needing macros in place of types, of which I've either hacked around with much nastier, more complex macros or I've given up on entirely and just wrote the code by hand. So I very much welcome this change.

[aturon]

While I share @brson's concern about feature creep (and also pity the type checker for the horrors this change is likely to unleash), I agree with others that this is a relatively minor change given the extant macro system. I don't think, for example, that it appreciably changes the difficulty that external tools already have to face when dealing with macros. And it's certainly a natural feature to want in conjunction with expression/item-level macros.

So on the whole, 👍 -- let's try this behind a feature gate, get some experience, and see what it looks like.

[jroesch]

@aturon I am in support of this change as well and would be happy to help getting it in shape and merged assuming @freebroccolo doesn't want to do the work himself.

[ghost]

@jroesch if the change is accepted I would be glad for you work on the merge if you like. I kept my branch up to date with master for a few months but I haven't tried to build it recently so there's probably some breakage by now.

[paholg]

I've been trying to figure out syntax extensions to make a convenient way to perform many type operations at once. If this goes through, I could do it in a couple lines in a regular macro.

I'm all for it.

[nrc]

AIUI, the argument for not doing this in the current macro system is that we could then allow it in a future one and that could be more of a carrot for using future macros rather than current ones. That is particularly appealing because future macros should be hygienic wrt to types and lifetimes and thus mean that this feature does not allow for the kind of badness that @nikomatsakis pointed out.

OTOH, I think it is a small and welcome extension to macro_rules and doesn't do any harm.

I do feel with every step like this we are improving things atm, at the expense of making adoption of some future macros system harder. That might be an OK trade off, but it is one we should make consciously.

[nikomatsakis]

I just realized something about hygiene and lifetimes. This realization makes me feel both better and worse about the idea of macros that expand to types. Previously, I described how the lack of lifetime hygiene meant that if a macro introduced lifetimes binders in the form of a for<'a>... type, those names might interfere with other lifetime names in scope. But what I just remembered is shadowing lifetime names is currently a hard error: http://is.gd/oDoH6V

So this is not really a concern when it comes to forward compatibility, though it may be a usability concern for the macro author. I assume people will just limit themselves to expanding to awful things like for<'__foo_a> to achieve some primitive namespacing, but that won't work for recursive use of type macros. (Annoyingly, we also have no "lifetime" fragment specifier like $t:lifetime, so it's awkward to write macros that accept lifetimes and then use them within types.)

Just for completeness, the other interaction with hygiene is that type-expanding macros will likely want to refer to well-known types. e.g. if I wanted a macro myvec!{X} that expanded to MyVec<X>, I would have to write it to use something like $crate::MyVec<$X>, but I think this sort of thing is just a fact of life with today's macro system. It just happens that naming top-level items is going to be very, very common in macros that expand to types, and is somewhat less common for macros that expand to expressions.

[jroesch]

@nikomatsakis would you be interested in expanding this to provide a fragment specifier for lifetimes? If it isn't out of scope I think it would be a useful/nice for type macros.

[retep998]

Just wondering, would this allow macros in ident positions? For example struct SomeMacro!() {.

[nikomatsakis]

@nrc

I do feel with every step like this we are improving things atm, at the expense of making adoption of some future macros system harder.

I'm not sure I fully understand what you mean. Do you mean "the more limited the current macro system is, the more tempting a new macro system will be to use?" That seems likely true -- if the current macro system is fulfilling most use cases people have in practice, it'll be more work to get them to migrate away.

That said, even if we did a great job selling a new, replacement macro system, I doubt we'll be able to remove the one we have. It is very widely used, that's why it's stable in the first place. I guess it's plausible that if the macro system is only being used in narrow ways, we might be able to keep some more restricted form around forever, but it's not clear what that subset is, or whether it would simplify the maintenance burden.

Or maybe you were thinking that we might be able to improve the existing macro system in place? For example, by adding lifetime hygiene? This also seems to get harder if macros have more use, but I strongly suspect the ship has largely sailed here, and the only way to do improve macros "in place" is with some sort of opt-in, whether that be a 2.0 release, a different form of macro_rules declaration, or something else.

For example, I know that the rusty_peg macro takes advantage of the lack of lifetime hygiene (admittedly, I only wrote things that way to work around the lack of lifetime hygiene). And I know of others that generate items parametrized by a few extra lifetimes (like 'a, 'b, 'c) "just in case" they are needed by the types the user writes. So I am quite confident that adding lifetime hygiene would break a reasonable number of existing macros.

[yongqli]

I would find this useful as well. My current use case is hacking around the lack of type level integers by asserting all the possible ways to multiply together matrices. For example:

impl<O: SquareMat, M, S: SquareMat> OpOn<O, S> for M
    where M: Mat<Row=O::Col, Col=S::Row> + Mul<S::Row, Output=O::Col> +
             Mul<S, Output=M> + Mul<<M as Mat>::Transpose, Output=O> {

The problem is that I have to assert all the possible ways to operate on all the matrices every place where I want to use a rectangular matrix. Being able to refactor them into a macro would be very nice. Ideally I would be able to write something like: op_on<O, M, S>(..) where rectangular_mat!(O, M, S) ...

[pythonesque]

Given that it's the final comment period: very much in favor of this (I thought I'd already commented, but I guess not). It enables some really neat stuff.

[dylanede]

Very much in favour of this. However, @yongqli, I don't think you would be able to do what you suggest with impl bounds with this feature. This feature only enables macros in type positions, not where clauses.

[nikomatsakis]

@jroesch

would you be interested in expanding this to provide a fragment specifier for lifetimes? If it isn't out of scope I think it would be a useful/nice for type macros.

I would be interested in having a fragment specific for lifetimes, yes. It does seem relevant to this RFC, but could also be separate.

[ghost]

I have the same question as @retep998 - would this permit struct some_macro!() {?

[ghost]

@retep998 @Toby-s no, this doesn't change anything about top level identifier fragments in that regard.

@dylanede Actually, the implementation I provided was usable for where clauses, although there were some things that didn't quite work due (I assume) to somewhat incomplete/different handling of that part of the AST. In fact, I used it to make a fairly reasonable implementation of a haskell type-families style macro (e.g., it would derive implementations from lists of clauses and patterns).

[pnkfelix]

Merged as https://github.com/rust-lang/rfcs/commits/master/text/0873-type-macros.md

[pnkfelix]

(sorry that i rebased it to my own branch rather than merging, i'll try to fix that now so that the colors are nicer)