Initial implementation of declarative macros 2.0 #40847

jseyfried · 2017-03-27T00:53:08Z

Implement declarative macros 2.0 (rust-lang/rfcs#1584) behind #![feature(decl_macro)].
Differences from macro_rules! include:

new syntax: macro m(..) { .. } instead of macro_rules! m { (..) => { .. } }
declarative macros are items:

// crate A:
pub mod foo {
    m!(); // use before definition; declaration order is irrelevant
    pub macro m() {} // `pub`, `pub(super)`, etc. work
}
fn main() {
    foo::m!(); // named like other items
    { use foo::m as n; n!(); } // imported like other items
}
pub use foo::m; // re-exported like other items

// crate B:
extern crate A; // no need for `#[macro_use]`
A::foo::m!(); A::m!();

Racket-like hygiene for items, imports, methods, fields, type parameters, privacy, etc.
- Intuitively, names in a macro definition are resolved in the macro definition's scope, not the scope in which the macro is used.
- This explaination of hygiene for Racket applies here (except for the "Breaking Hygiene" section). I wrote a similar explanation for Rust.
- Generally speaking, if fn f() { <body> } resolves, pub macro m() { <body> } ... m!() also resolves, even if m!() is in a separate crate.
- ::foo::bar in a macro behaves like $crate::foo::bar in a macro_rules!, except it can access everything visible from the macro (thus more permissive).
- See src/test/{run-pass, compile-fail}/hygiene for examples. Small example:

mod foo {
    fn f() { println!("hello world"); }
    pub macro m() { f(); }
}
fn main() { foo::m!(); }

Limitations:

This does not address planned changes to matchers (expr,ty, etc.), c.f. Macro matching is incorrect for macros invoked in macro expansions #26361.
Lints (including stability and deprecation) and unsafe are not hygienic.
- adding hygiene here will be mostly or entirely backwards compatible
Nested macro definitions (a macro inside another macro) don't always work correctly when invoked from external crates.
- pending improvements in how we encode macro definitions in crate metadata
There is no way to "escape" hygiene without using a procedural macro.

r? @nrc

jseyfried · 2017-03-27T05:06:42Z

cc @rust-lang/compiler @durka @Ericson2314 @SergioBenitez @tikue

petrochenkov · 2017-03-27T22:53:07Z

(I'm going to review this too in a few days.)

petrochenkov · 2017-03-28T21:58:08Z

src/libsyntax/parse/parser.rs

+            token::Ident(ident) if ident.name == keywords::Macro.name() => {
+                self.bump();
+                let ident = self.parse_ident()?;
+                let tokens = if self.token == token::OpenDelim(token::Brace) {


It's usually better to use self.check(&token::OpenDelim(token::Brace)) to populate expected tokens.

Good point, will do.

petrochenkov · 2017-03-28T21:58:27Z

src/libsyntax/parse/parser.rs

+                        TokenTree::Delimited(_, ref delimited) => delimited.stream(),
+                        _ => unreachable!(),
+                    }
+                } else if self.token == token::OpenDelim(token::Paren) {


self.check(...) as well

petrochenkov · 2017-03-28T21:59:57Z

src/libsyntax/parse/parser.rs

+                    }
+                } else if self.token == token::OpenDelim(token::Paren) {
+                    let args = self.parse_token_tree();
+                    let body = if self.token == token::OpenDelim(token::Brace) {


And here, plus self.unexpected() can be used for the else clause.

petrochenkov · 2017-03-28T22:10:44Z

src/librustc_resolve/resolve_imports.rs

@@ -783,7 +783,7 @@ impl<'a, 'b:'a> ImportResolver<'a, 'b> {
            };

            if binding.vis == ty::Visibility::Public &&
-               (binding.is_import() || binding.is_extern_crate()) {
+               (binding.is_import() || binding.is_macro_def()) {


Not sure what this change does.

Removing binding.is_extern_crate() is a no-op (it's implied by binding.is_import()), adding binding.is_macro_def() allows the binding to be accessed from external crates.

petrochenkov · 2017-03-28T22:14:18Z

src/librustc_resolve/lib.rs

@@ -1107,7 +1107,7 @@ pub struct Resolver<'a> {

    prelude: Option<Module<'a>>,

-    trait_item_map: FxHashMap<(DefId, Name, Namespace), (Def, bool /* has self */)>,
+    has_self: FxHashSet<DefId>,


Could you leave a comment that this is used only for better diagnostics, not for the resolution algorithm.
(Maybe group all resolver fields of this kind together.)

petrochenkov · 2017-03-28T22:36:40Z

The RFC draft contains this exception from hygiene rules - "except when matching against names in patterns".
What is this about? Could you give an example?
(I haven't read the few last commits yet, maybe they already contain the answer.)

jseyfried · 2017-03-28T22:46:54Z

@petrochenkov I mean something like this:

macro n($i:ident) {
    m!($i, x); // The resolution of `x` depends on whether `$i` is `foo` or `bar`
}

// Here, we are "matching against names in patterns" with `foo` and `bar`:
macro m {
    (foo, $j:ident) => { let $j = 0; $j },
    (bar, $j:ident) => { $j },
}

petrochenkov · 2017-03-28T23:14:34Z

src/libsyntax_pos/hygiene.rs

+        scope
+    }
+
+    pub fn glob_adjust(&mut self, expansion: Mark, mut glob_ctxt: SyntaxContext)


Could you leave some, maybe minimal, documentation for the new public interface of SyntaxContext? What is the intent of these adjustments, where are they used, how do they correspond to Rust code.

Yeah, added comments.

petrochenkov · 2017-03-28T23:15:01Z

src/libsyntax/ext/base.rs

@@ -530,12 +530,15 @@ pub enum SyntaxExtension {

    /// An attribute-like procedural macro that derives a builtin trait.
    BuiltinDerive(BuiltinDeriveFn),
+
+    DeclMacro(Box<TTMacroExpander>, Option<Span>),


Doc comment here as well, please.

jseyfried · 2017-03-29T01:41:09Z

@petrochenkov I documented the adjust functions and addressed your other comments.

petrochenkov · 2017-03-29T07:41:09Z

@jseyfried
This is only the beginning! I haven't read the "hygienize" and "improve" commits yet 😄

bors · 2017-03-30T16:58:33Z

☔ The latest upstream changes (presumably #40224) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · 2017-03-30T23:51:32Z

FWIW, I finished reading (more or less).
I don't see critical problems, but there is lot of code that I will never understand without weeks of experimenting which I don't have unfortunately, so I can't guarantee anything. I also didn't bother reporting nits, including naming issues.

I'm unhappy with greatly raised complexity and leakage of macros and hygiene into later stages of compilation, I can only hope it will worth it, because the general idea of per-identifier hygiene/resolution/privacy is very attractive.

It looks like private-in-public checks are left alone. They are still unhygienic, right?
This is good for now, "semantic" type privacy seems incompatible with identifier-based privacy implemented here.

The embargo visitor is terribly pessimistic, if I understand it correctly, but it all can be improved in the future.

@petrochenkov I mean something like this:

I see, I though about pattern-patterns, not macro-patterns.

jseyfried · 2017-03-31T09:21:55Z

@petrochenkov

I'm unhappy with greatly raised complexity and leakage of macros and hygiene into later stages of compilation

I'm afraid this is unavoidable if we want names in macro definitions to have reliable resolutions (as defined in the draft RFC).

It looks like private-in-public checks are left alone. They are still unhygienic, right?

Yeah, this doesn't address private-in-public hygiene. I'm currently leaning towards:

macro m($i:ident) {
    fn $i(_: <type>) {} // <type> must be accessible from the use site (enforced today)
    fn f(_: <type>){} // <type> must be accessible from the use site *or* the def site
}

After we address #30476, inferred types would also need to be accessible from the use site or the def site.

The embargo visitor is terribly pessimistic

Indeed. However, this pessimism is needed with the current implementation of procedural macros (#40939), since a procedural macro can rename an identifier and leave the hygiene information (in the span) unchanged. Thus, we must be prepared to resolve arbitrary names at the def site of exported declarative macros.

jseyfried · 2017-05-31T18:19:10Z

@tikue Yeah, in general it's a bug if invoking a macro in an external crate behaves differently than invoking it locally. Could you open an issue (just copy the comment) and cc me?

tikue · 2017-05-31T18:32:33Z

Thanks @jseyfried, will do.

@jseyfried

Extend the unused macro lint to macros 2.0 Extends the unused macro lint (added in PR #41907) to macros 2.0 (added in PR #40847). r? @jseyfried

@jseyfried

Check types for privacy This PR implements late post factum checking of type privacy, as opposed to early preventive "private-in-public" checking. This will allow to turn private-in-public checks into a lint and make them more heuristic-based, and more aligned with what people may expect (e.g. reachability-based behavior). Types are privacy-checked if they are written explicitly, and also if they are inferred as expression or pattern types. This PR checks "semantic" types and does it unhygienically, this significantly restricts what macros 2.0 (as implemented in #40847) can do (sorry @jseyfried) - they still can use private *names*, but can't use private *types*. This is the most conservative solution, but hopefully it's temporary and can be relaxed in the future, probably using macro contexts of expression/pattern spans. Traits are also checked in preparation for [trait aliases](#41517), which will be able to leak private traits, and macros 2.0 which will be able to leak pretty much anything. This is a [breaking-change], but the code that is not contrived and can be broken by this patch should be guarded by `private_in_public` lint. [Previous crater run](#34537 (comment)) discovered a few abandoned crates that weren't updated since `private_in_public` has been introduced in 2015. cc #34537 https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504 Fixes #30476 Fixes #33479 cc @nikomatsakis r? @eddyb

ghost · 2017-06-24T21:37:03Z

I'm not entirely sure if this should be happening but it seems that you can't actually implement a trait from outside of a macro while inside a macro unless the Trait doesn't have anything to implement. While the code in #40847 (comment) seems to compile, it only compiles because fn f(&self) { } has a default implementation. If you change it to fn f(&self); then it no longer works and it complains that f is missing. It's also very misleading because the function that you attempt to implement it with is silently ignored, and the default implementation will run instead.

#![feature(decl_macro)]

trait Tr {
    fn f(&self) {
        println!("This shouldn't happen");
    }
}

pub macro m($t:ty) {
    impl Tr for $t {
        fn f(&self) {
            println!("Run me");
        }
    }
}

struct S;
m!(S);

fn main() {
    S.f(); // "This shouldn't happen"
}

petrochenkov · 2017-06-24T22:16:56Z

@whataloadofwhat
This is certainly a bug.
Ancestors::defs in specialization_graph.rs and its uses seem to not be updated for macro 2.0 hygiene.

@nrc

proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API All new API is gated behind `#![feature(proc_macro)]` and may be used with `#[proc_macro]`, `#[proc_macro_attribute]`, and `#[proc_macro_derive]` procedural macros. More specifically, this PR adds the following in `proc_macro`: ```rust // `TokenStream` constructors: impl TokenStream { fn empty() -> TokenStream { ... } } impl From<TokenTree> for TokenStream { ... } impl From<TokenKind> for TokenStream { ... } impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... } macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter // `TokenStream` destructuring: impl TokenStream { fn is_empty(&self) -> bool { ... } } impl IntoIterator for TokenStream { type Item = TokenTree; ... } struct TokenTree { span: Span, kind: TokenKind } impl From<TokenKind> for TokenTree { ... } impl Display for TokenTree { ... } struct Span { ... } // a region of source code along with expansion/hygiene information impl Default for Span { ... } // a span from the current procedural macro definition impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion fn quote_span(span: Span) -> TokenStream; enum TokenKind { Group(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )` Term(Term), // a unicode identifier, lifetime ('a), or underscore Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.). Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`) } enum Delimiter { Parenthesis, // `( ... )` Brace, // `[ ... ]` Bracket, // `{ ... }` None, // an implicit delimiter, e.g. `$var`, where $var is `...`. } struct Term { ... } // An interned string impl Term { fn intern(string: &str) -> Symbol { ... } fn as_str(&self) -> &str { ... } } enum Spacing { Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`. Joint, // immediately followed by another `Op`, e.g. `+` in `+=` } struct Literal { ... } impl Display for Literal { ... } impl Literal { fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64 fn string(string: &str) -> Literal { ... } fn character(ch: char) -> Literal { ... } fn byte_string(bytes: &[u8]) -> Literal { ... } } ``` For details on `quote!` hygiene, see [this example](20a9048) and [declarative macros 2.0](#40847). r? @nrc

@jseyfried

Check types for privacy This PR implements late post factum checking of type privacy, as opposed to early preventive "private-in-public" checking. This will allow to turn private-in-public checks into a lint and make them more heuristic-based, and more aligned with what people may expect (e.g. reachability-based behavior). Types are privacy-checked if they are written explicitly, and also if they are inferred as expression or pattern types. This PR checks "semantic" types and does it unhygienically, this significantly restricts what macros 2.0 (as implemented in #40847) can do (sorry @jseyfried) - they still can use private *names*, but can't use private *types*. This is the most conservative solution, but hopefully it's temporary and can be relaxed in the future, probably using macro contexts of expression/pattern spans. Traits are also checked in preparation for [trait aliases](#41517), which will be able to leak private traits, and macros 2.0 which will be able to leak pretty much anything. This is a [breaking-change], but the code that is not contrived and can be broken by this patch should be guarded by `private_in_public` lint. [Previous crater run](#34537 (comment)) discovered a few abandoned crates that weren't updated since `private_in_public` has been introduced in 2015. cc #34537 https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504 Fixes #30476 Fixes #33479 cc @nikomatsakis r? @eddyb

@jseyfried

Improve resolution of associated types in declarative macros 2.0 Make various identifier comparisons for associated types (and sometimes other associated items) hygienic. Now declarative macros 2.0 can use `Self::AssocTy`, `TyParam::AssocTy`, `Trait<AssocTy = u8>` where `AssocTy` is an associated type of a trait `Trait` visible from the macro. Also, `Trait` can now be implemented inside the macro and specialization should work properly (fixes #40847 (comment)). r? @jseyfried or @eddyb

@jseyfried

Improve resolution of associated types in declarative macros 2.0 Make various identifier comparisons for associated types (and sometimes other associated items) hygienic. Now declarative macros 2.0 can use `Self::AssocTy`, `TyParam::AssocTy`, `Trait<AssocTy = u8>` where `AssocTy` is an associated type of a trait `Trait` visible from the macro. Also, `Trait` can now be implemented inside the macro and specialization should work properly (fixes #40847 (comment)). r? @jseyfried or @eddyb

rust-highfive assigned nrc Mar 27, 2017

jseyfried force-pushed the decl_macro branch 5 times, most recently from 0b2df87 to d82defb Compare March 27, 2017 04:03

jseyfried mentioned this pull request Mar 27, 2017

Tracking issue: declarative macros 2.0 #39412

Open

19 tasks

petrochenkov reviewed Mar 28, 2017

View reviewed changes

jseyfried force-pushed the decl_macro branch 2 times, most recently from fdfd619 to eaa9bb4 Compare March 29, 2017 00:39

jseyfried force-pushed the decl_macro branch 2 times, most recently from ca341e3 to afe7d89 Compare March 30, 2017 12:06

jseyfried force-pushed the decl_macro branch from afe7d89 to fbf18a2 Compare March 30, 2017 22:16

jseyfried mentioned this pull request Mar 31, 2017

proc_macro: implement TokenTree, TokenKind, hygienic quote!, and other API #40939

Merged

jseyfried mentioned this pull request Apr 1, 2017

self and hygiene interact in unfortunate ways #39922

Closed

bors added a commit that referenced this pull request Jun 3, 2017

Auto merge of #42334 - est31:master, r=jseyfried

a8dc246

Extend the unused macro lint to macros 2.0 Extends the unused macro lint (added in PR #41907) to macros 2.0 (added in PR #40847). r? @jseyfried

jseyfried mentioned this pull request Jun 5, 2017

decl_macro: scoping and other issues #42448

Closed

retep998 mentioned this pull request Jun 8, 2017

rustc crashes with exit code 3221225501 #42544

Closed

petrochenkov mentioned this pull request Jul 4, 2017

Disable macro hygiene rust-lang/rfcs#2054

Closed

ibabushkin mentioned this pull request Aug 13, 2017

Add expansion info to crate metadata #43847

Closed

TimNN mentioned this pull request Aug 17, 2017

Nightly regression on x86_64_unknown_linux_gnu #43126

Closed

petrochenkov mentioned this pull request Sep 2, 2017

RFC: Clarify and streamline paths and visibility rust-lang/rfcs#2126

Merged

valff mentioned this pull request Sep 6, 2017

Declarative Macros 2.0 and Source Corruption rust-lang/rustfmt#1949

Closed

petrochenkov mentioned this pull request Sep 24, 2017

Improve resolution of associated types in declarative macros 2.0 #44818

Merged

jseyfried mentioned this pull request Nov 28, 2017

incr.comp.: Fix Span hashing in various ways #46301

Closed

Geal mentioned this pull request Dec 2, 2017

follow and test macros 2.0 rust-bakery/nom#632

Closed

petrochenkov mentioned this pull request Apr 21, 2018

Visibilities of "private" items from macros 1.2 #50130

Closed

c410-f3r mentioned this pull request Feb 9, 2021

Move some tests to more reasonable directories - 4 #81941

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation of declarative macros 2.0 #40847

Initial implementation of declarative macros 2.0 #40847

jseyfried commented Mar 27, 2017 •

edited

Loading

jseyfried commented Mar 27, 2017

petrochenkov commented Mar 27, 2017

petrochenkov Mar 28, 2017

jseyfried Mar 28, 2017

petrochenkov Mar 28, 2017

petrochenkov Mar 28, 2017

petrochenkov Mar 28, 2017

jseyfried Mar 28, 2017 •

edited

Loading

petrochenkov Mar 28, 2017

jseyfried Mar 28, 2017

petrochenkov commented Mar 28, 2017

jseyfried commented Mar 28, 2017 •

edited

Loading

petrochenkov Mar 28, 2017

jseyfried Mar 28, 2017 •

edited

Loading

petrochenkov Mar 28, 2017

jseyfried Mar 28, 2017 •

edited

Loading

jseyfried commented Mar 29, 2017

petrochenkov commented Mar 29, 2017

bors commented Mar 30, 2017

petrochenkov commented Mar 30, 2017 •

edited

Loading

jseyfried commented Mar 31, 2017 •

edited

Loading

jseyfried commented May 31, 2017

tikue commented May 31, 2017

ghost commented Jun 24, 2017

petrochenkov commented Jun 24, 2017

Initial implementation of declarative macros 2.0 #40847

Initial implementation of declarative macros 2.0 #40847

Conversation

jseyfried commented Mar 27, 2017 • edited Loading

jseyfried commented Mar 27, 2017

petrochenkov commented Mar 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jseyfried Mar 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov commented Mar 28, 2017

jseyfried commented Mar 28, 2017 • edited Loading

Choose a reason for hiding this comment

jseyfried Mar 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jseyfried Mar 28, 2017 • edited Loading

Choose a reason for hiding this comment

jseyfried commented Mar 29, 2017

petrochenkov commented Mar 29, 2017

bors commented Mar 30, 2017

petrochenkov commented Mar 30, 2017 • edited Loading

jseyfried commented Mar 31, 2017 • edited Loading

jseyfried commented May 31, 2017

tikue commented May 31, 2017

ghost commented Jun 24, 2017

petrochenkov commented Jun 24, 2017

jseyfried commented Mar 27, 2017 •

edited

Loading

jseyfried Mar 28, 2017 •

edited

Loading

jseyfried commented Mar 28, 2017 •

edited

Loading

jseyfried Mar 28, 2017 •

edited

Loading

jseyfried Mar 28, 2017 •

edited

Loading

petrochenkov commented Mar 30, 2017 •

edited

Loading

jseyfried commented Mar 31, 2017 •

edited

Loading