proc_macro: Reorganize public API

This commit is a reorganization of the `proc_macro` crate's public user-facing
API. This is the result of a number of discussions at the recent Rust All-Hands
where we're hoping to get the `proc_macro` crate into ship shape for
stabilization of a subset of its functionality in the Rust 2018 release.

The reorganization here is motivated by experiences from the `proc-macro2`,
`quote`, and `syn` crates on crates.io (and other crates which depend on them).
The main focus is future flexibility along with making a few more operations
consistent and/or fixing bugs. A summary of the changes made from today's
`proc_macro` API is:

* The `TokenNode` enum has been removed and the public fields of `TokenTree`
  have also been removed. Instead the `TokenTree` type is now a public enum
  (what `TokenNode` was) and each variant is an opaque struct which internally
  contains `Span` information. This makes the various tokens a bit more
  consistent, require fewer wrappers, and otherwise provides good
  future-compatibility as opaque structs are easy to modify later on.

* `Literal` integer constructors have been expanded to be unambiguous as to what
  they're doing and also allow for more future flexibility. Previously
  constructors like `Literal::float` and `Literal::integer` were used to create
  unsuffixed literals and the concrete methods like `Literal::i32` would create
  a suffixed token. This wasn't immediately clear to all users (the
  suffixed/unsuffixed aspect) and having *one* constructor for unsuffixed
  literals required us to pick a largest type which may not always be true. To
  fix these issues all constructors are now of the form
  `Literal::i32_unsuffixed` or `Literal::i32_suffixed` (for all integral types).
  This should allow future compatibility as well as being immediately clear
  what's suffixed and what isn't.

* Each variant of `TokenTree` internally contains a `Span` which can also be
  configured via `set_span`. For example `Literal` and `Term` now both
  internally contain a `Span` rather than having it stored in an auxiliary
  location.

* Constructors of all tokens are called `new` now (aka `Term::intern` is gone)
  and most do not take spans. Manufactured tokens typically don't have a fresh
  span to go with them and the span is purely used for error-reporting
  **except** the span for `Term`, which currently affects hygiene. The default
  spans for all these constructed tokens is `Span::call_site()` for now.

  The `Term` type's constructor explicitly requires passing in a `Span` to
  provide future-proofing against possible hygiene changes. It's intended that a
  first pass of stabilization will likely only stabilize `Span::call_site()`
  which is an explicit opt-in for "I would like no hygiene here please". The
  intention here is to make this explicit in procedural macros to be
  forwards-compatible with a hygiene-specifying solution.

* Some of the conversions for `TokenStream` have been simplified a little.

* The `TokenTreeIter` iterator was renamed to `token_stream::IntoIter`.

Overall the hope is that this is the "final pass" at the API of `TokenStream`
and most of `TokenTree` before stabilization. Explicitly left out here is any
changes to `Span`'s API which will likely need to be re-evaluated before
stabilization.

All changes in this PR have already been reflected to the [`proc-macro2`],
`quote`, and `syn` crates. New versions of all these crates have also been
published to crates.io.

Once this lands in nightly I plan on making an internals post again summarizing
the changes made here and also calling on all macro authors to give the APIs a
spin and see how they work. Hopefully pending no major issues we can then have
an FCP to stabilize later this cycle!

[`proc-macro2`]: https://docs.rs/proc-macro2/0.3.1/proc_macro2/

src/libproc_macro/quote.rs

@@ -14,7 +14,7 @@
 //! This quasiquoter uses macros 2.0 hygiene to reliably access
 //! items from `proc_macro`, to build a `proc_macro::TokenStream`.
 
-use {Delimiter, Literal, Spacing, Span, Term, TokenNode, TokenStream, TokenTree};
+use {Delimiter, Literal, Spacing, Span, Term, Op, Group, TokenStream, TokenTree};
 
 use syntax::ext::base::{ExtCtxt, ProcMacro};
 use syntax::parse::token;
@@ -23,47 +23,59 @@ use syntax::tokenstream;
 pub struct Quoter;
 
 pub fn unquote<T: Into<TokenStream> + Clone>(tokens: &T) -> TokenStream {
-    T::into(tokens.clone())
+    tokens.clone().into()
 }
 
 pub trait Quote {
     fn quote(self) -> TokenStream;
 }
 
+macro_rules! tt2ts {
+    ($e:expr) => (TokenStream::from(TokenTree::from($e)))
+}
+
 macro_rules! quote_tok {
-    (,) => { TokenNode::Op(',', Spacing::Alone) };
-    (.) => { TokenNode::Op('.', Spacing::Alone) };
-    (:) => { TokenNode::Op(':', Spacing::Alone) };
+    (,) => { tt2ts!(Op::new(',', Spacing::Alone)) };
+    (.) => { tt2ts!(Op::new('.', Spacing::Alone)) };
+    (:) => { tt2ts!(Op::new(':', Spacing::Alone)) };
+    (|) => { tt2ts!(Op::new('|', Spacing::Alone)) };
     (::) => {
         [
-            TokenNode::Op(':', Spacing::Joint),
-            TokenNode::Op(':', Spacing::Alone)
-        ].iter().cloned().collect::<TokenStream>()
+            TokenTree::from(Op::new(':', Spacing::Joint)),
+            TokenTree::from(Op::new(':', Spacing::Alone)),
+        ].iter()
+            .cloned()
+            .map(|mut x| {
+                x.set_span(Span::def_site());
+                x
+            })
+            .collect::<TokenStream>()
     };
-    (!) => { TokenNode::Op('!', Spacing::Alone) };
-    (<) => { TokenNode::Op('<', Spacing::Alone) };
-    (>) => { TokenNode::Op('>', Spacing::Alone) };
-    (_) => { TokenNode::Op('_', Spacing::Alone) };
-    (0) => { TokenNode::Literal(::Literal::integer(0)) };
-    (&) => { TokenNode::Op('&', Spacing::Alone) };
-    ($i:ident) => { TokenNode::Term(Term::intern(stringify!($i))) };
+    (!) => { tt2ts!(Op::new('!', Spacing::Alone)) };
+    (<) => { tt2ts!(Op::new('<', Spacing::Alone)) };
+    (>) => { tt2ts!(Op::new('>', Spacing::Alone)) };
+    (_) => { tt2ts!(Op::new('_', Spacing::Alone)) };
+    (0) => { tt2ts!(Literal::i8_unsuffixed(0)) };
+    (&) => { tt2ts!(Op::new('&', Spacing::Alone)) };
+    ($i:ident) => { tt2ts!(Term::new(stringify!($i), Span::def_site())) };
 }
 
 macro_rules! quote_tree {
     ((unquote $($t:tt)*)) => { $($t)* };
     ((quote $($t:tt)*)) => { ($($t)*).quote() };
-    (($($t:tt)*)) => { TokenNode::Group(Delimiter::Parenthesis, quote!($($t)*)) };
-    ([$($t:tt)*]) => { TokenNode::Group(Delimiter::Bracket, quote!($($t)*)) };
-    ({$($t:tt)*}) => { TokenNode::Group(Delimiter::Brace, quote!($($t)*)) };
+    (($($t:tt)*)) => { tt2ts!(Group::new(Delimiter::Parenthesis, quote!($($t)*))) };
+    ([$($t:tt)*]) => { tt2ts!(Group::new(Delimiter::Bracket, quote!($($t)*))) };
+    ({$($t:tt)*}) => { tt2ts!(Group::new(Delimiter::Brace, quote!($($t)*))) };
     ($t:tt) => { quote_tok!($t) };
 }
 
 macro_rules! quote {
     () => { TokenStream::empty() };
     ($($t:tt)*) => {
-        [
-            $(TokenStream::from(quote_tree!($t)),)*
-        ].iter().cloned().collect::<TokenStream>()
+        [$(quote_tree!($t),)*].iter()
+            .cloned()
+            .flat_map(|x| x.into_iter())
+            .collect::<TokenStream>()
     };
 }
 
@@ -97,72 +109,81 @@ impl Quote for TokenStream {
         let tokens = self.into_iter().filter_map(|tree| {
             if after_dollar {
                 after_dollar = false;
-                match tree.kind {
-                    TokenNode::Term(_) => {
+                match tree {
+                    TokenTree::Term(_) => {
+                        let tree = TokenStream::from(tree);
                         return Some(quote!(::__internal::unquote(&(unquote tree)),));
                     }
-                    TokenNode::Op('$', _) => {}
+                    TokenTree::Op(ref tt) if tt.op() == '$' => {}
                     _ => panic!("`$` must be followed by an ident or `$` in `quote!`"),
                 }
-            } else if let TokenNode::Op('$', _) = tree.kind {
-                after_dollar = true;
-                return None;
+            } else if let TokenTree::Op(tt) = tree {
+                if tt.op() == '$' {
+                    after_dollar = true;
+                    return None;
+                }
             }
 
             Some(quote!(::TokenStream::from((quote tree)),))
-        }).collect::<TokenStream>();
+        }).flat_map(|t| t.into_iter()).collect::<TokenStream>();
 
         if after_dollar {
             panic!("unexpected trailing `$` in `quote!`");
         }
 
-        quote!([(unquote tokens)].iter().cloned().collect::<::TokenStream>())
+        quote!(
+            [(unquote tokens)].iter()
+                .cloned()
+                .flat_map(|x| x.into_iter())
+                .collect::<::TokenStream>()
+        )
     }
 }
 
 impl Quote for TokenTree {
     fn quote(self) -> TokenStream {
-        quote!(::TokenTree { span: (quote self.span), kind: (quote self.kind) })
+        match self {
+            TokenTree::Op(tt) => quote!(::TokenTree::Op( (quote tt) )),
+            TokenTree::Group(tt) => quote!(::TokenTree::Group( (quote tt) )),
+            TokenTree::Term(tt) => quote!(::TokenTree::Term( (quote tt) )),
+            TokenTree::Literal(tt) => quote!(::TokenTree::Literal( (quote tt) )),
+        }
     }
 }
 
-impl Quote for TokenNode {
+impl Quote for char {
     fn quote(self) -> TokenStream {
-        macro_rules! gen_match {
-            ($($i:ident($($arg:ident),+)),*) => {
-                match self {
-                    $(TokenNode::$i($($arg),+) => quote! {
-                        ::TokenNode::$i($((quote $arg)),+)
-                    },)*
-                }
-            }
-        }
+        TokenTree::from(Literal::character(self)).into()
+    }
+}
 
-        gen_match! { Op(op, kind), Group(delim, tokens), Term(term), Literal(lit) }
+impl<'a> Quote for &'a str {
+    fn quote(self) -> TokenStream {
+        TokenTree::from(Literal::string(self)).into()
     }
 }
 
-impl Quote for char {
+impl Quote for usize {
     fn quote(self) -> TokenStream {
-        TokenNode::Literal(Literal::character(self)).into()
+        TokenTree::from(Literal::usize_unsuffixed(self)).into()
     }
 }
 
-impl<'a> Quote for &'a str {
+impl Quote for Group {
     fn quote(self) -> TokenStream {
-        TokenNode::Literal(Literal::string(self)).into()
+        quote!(::Group::new((quote self.delimiter()), (quote self.stream())))
     }
 }
 
-impl Quote for usize {
+impl Quote for Op {
     fn quote(self) -> TokenStream {
-        TokenNode::Literal(Literal::integer(self as i128)).into()
+        quote!(::Op::new((quote self.op()), (quote self.spacing())))
     }
 }
 
 impl Quote for Term {
     fn quote(self) -> TokenStream {
-        quote!(::Term::intern((quote self.as_str())))
+        quote!(::Term::new((quote self.as_str()), (quote self.span())))
     }
 }
 
@@ -182,31 +203,38 @@ macro_rules! literals {
         impl LiteralKind {
             pub fn with_contents_and_suffix(self, contents: Term, suffix: Option<Term>)
                                             -> Literal {
-                let contents = contents.0;
-                let suffix = suffix.map(|t| t.0);
+                let sym = contents.sym;
+                let suffix = suffix.map(|t| t.sym);
                 match self {
                     $(LiteralKind::$i => {
-                        Literal(token::Literal(token::Lit::$i(contents), suffix))
+                        Literal {
+                            token: token::Literal(token::Lit::$i(sym), suffix),
+                            span: contents.span,
+                        }
                     })*
                     $(LiteralKind::$raw(n) => {
-                        Literal(token::Literal(token::Lit::$raw(contents, n), suffix))
+                        Literal {
+                            token: token::Literal(token::Lit::$raw(sym, n), suffix),
+                            span: contents.span,
+                        }
                     })*
                 }
             }
         }
 
         impl Literal {
             fn kind_contents_and_suffix(self) -> (LiteralKind, Term, Option<Term>) {
-                let (lit, suffix) = match self.0 {
+                let (lit, suffix) = match self.token {
                     token::Literal(lit, suffix) => (lit, suffix),
-                    _ => panic!("unsupported literal {:?}", self.0),
+                    _ => panic!("unsupported literal {:?}", self.token),
                 };
 
                 let (kind, contents) = match lit {
                     $(token::Lit::$i(contents) => (LiteralKind::$i, contents),)*
                     $(token::Lit::$raw(contents, n) => (LiteralKind::$raw(n), contents),)*
                 };
-                (kind, Term(contents), suffix.map(Term))
+                let suffix = suffix.map(|sym| Term::new(&sym.as_str(), self.span()));
+                (kind, Term::new(&contents.as_str(), self.span()), suffix)
             }
         }
 

src/test/compile-fail-fulldeps/proc-macro/auxiliary/attributes-included.rs

@@ -16,7 +16,7 @@
 
 extern crate proc_macro;
 
-use proc_macro::{TokenStream, TokenTree, TokenNode, Delimiter, Literal, Spacing};
+use proc_macro::{TokenStream, TokenTree, Delimiter, Literal, Spacing, Group};
 
 #[proc_macro_attribute]
 pub fn foo(attr: TokenStream, input: TokenStream) -> TokenStream {
@@ -52,24 +52,30 @@ pub fn bar(attr: TokenStream, input: TokenStream) -> TokenStream {
 }
 
 fn assert_inline(slice: &mut &[TokenTree]) {
-    match slice[0].kind {
-        TokenNode::Op('#', _) => {}
+    match &slice[0] {
+        TokenTree::Op(tt) => assert_eq!(tt.op(), '#'),
         _ => panic!("expected '#' char"),
     }
-    match slice[1].kind {
-        TokenNode::Group(Delimiter::Bracket, _) => {}
+    match &slice[1] {
+        TokenTree::Group(tt) => assert_eq!(tt.delimiter(), Delimiter::Bracket),
         _ => panic!("expected brackets"),
     }
     *slice = &slice[2..];
 }
 
 fn assert_doc(slice: &mut &[TokenTree]) {
-    match slice[0].kind {
-        TokenNode::Op('#', Spacing::Alone) => {}
+    match &slice[0] {
+        TokenTree::Op(tt) => {
+            assert_eq!(tt.op(), '#');
+            assert_eq!(tt.spacing(), Spacing::Alone);
+        }
         _ => panic!("expected #"),
     }
-    let inner = match slice[1].kind {
-        TokenNode::Group(Delimiter::Bracket, ref s) => s.clone(),
+    let inner = match &slice[1] {
+        TokenTree::Group(tt) => {
+            assert_eq!(tt.delimiter(), Delimiter::Bracket);
+            tt.stream()
+        }
         _ => panic!("expected brackets"),
     };
     let tokens = inner.into_iter().collect::<Vec<_>>();
@@ -79,49 +85,55 @@ fn assert_doc(slice: &mut &[TokenTree]) {
         panic!("expected three tokens in doc")
     }
 
-    match tokens[0].kind {
-        TokenNode::Term(ref t) => assert_eq!("doc", t.as_str()),
+    match &tokens[0] {
+        TokenTree::Term(tt) => assert_eq!("doc", tt.as_str()),
         _ => panic!("expected `doc`"),
     }
-    match tokens[1].kind {
-        TokenNode::Op('=', Spacing::Alone) => {}
+    match &tokens[1] {
+        TokenTree::Op(tt) => {
+            assert_eq!(tt.op(), '=');
+            assert_eq!(tt.spacing(), Spacing::Alone);
+        }
         _ => panic!("expected equals"),
     }
-    match tokens[2].kind {
-        TokenNode::Literal(_) => {}
+    match tokens[2] {
+        TokenTree::Literal(_) => {}
         _ => panic!("expected literal"),
     }
 
     *slice = &slice[2..];
 }
 
 fn assert_invoc(slice: &mut &[TokenTree]) {
-    match slice[0].kind {
-        TokenNode::Op('#', _) => {}
+    match &slice[0] {
+        TokenTree::Op(tt) => assert_eq!(tt.op(), '#'),
         _ => panic!("expected '#' char"),
     }
-    match slice[1].kind {
-        TokenNode::Group(Delimiter::Bracket, _) => {}
+    match &slice[1] {
+        TokenTree::Group(tt) => assert_eq!(tt.delimiter(), Delimiter::Bracket),
         _ => panic!("expected brackets"),
     }
     *slice = &slice[2..];
 }
 
 fn assert_foo(slice: &mut &[TokenTree]) {
-    match slice[0].kind {
-        TokenNode::Term(ref name) => assert_eq!(name.as_str(), "fn"),
+    match &slice[0] {
+        TokenTree::Term(tt) => assert_eq!(tt.as_str(), "fn"),
         _ => panic!("expected fn"),
     }
-    match slice[1].kind {
-        TokenNode::Term(ref name) => assert_eq!(name.as_str(), "foo"),
+    match &slice[1] {
+        TokenTree::Term(tt) => assert_eq!(tt.as_str(), "foo"),
         _ => panic!("expected foo"),
     }
-    match slice[2].kind {
-        TokenNode::Group(Delimiter::Parenthesis, ref s) => assert!(s.is_empty()),
+    match &slice[2] {
+        TokenTree::Group(tt) => {
+            assert_eq!(tt.delimiter(), Delimiter::Parenthesis);
+            assert!(tt.stream().is_empty());
+        }
         _ => panic!("expected parens"),
     }
-    match slice[3].kind {
-        TokenNode::Group(Delimiter::Brace, _) => {}
+    match &slice[3] {
+        TokenTree::Group(tt) => assert_eq!(tt.delimiter(), Delimiter::Brace),
         _ => panic!("expected braces"),
     }
     *slice = &slice[4..];
@@ -132,22 +144,17 @@ fn fold_stream(input: TokenStream) -> TokenStream {
 }
 
 fn fold_tree(input: TokenTree) -> TokenTree {
-    TokenTree {
-        span: input.span,
-        kind: fold_node(input.kind),
-    }
-}
-
-fn fold_node(input: TokenNode) -> TokenNode {
     match input {
-        TokenNode::Group(a, b) => TokenNode::Group(a, fold_stream(b)),
-        TokenNode::Op(a, b) => TokenNode::Op(a, b),
-        TokenNode::Term(a) => TokenNode::Term(a),
-        TokenNode::Literal(a) => {
+        TokenTree::Group(b) => {
+            TokenTree::Group(Group::new(b.delimiter(), fold_stream(b.stream())))
+        }
+        TokenTree::Op(b) => TokenTree::Op(b),
+        TokenTree::Term(a) => TokenTree::Term(a),
+        TokenTree::Literal(a) => {
             if a.to_string() != "\"foo\"" {
-                TokenNode::Literal(a)
+                TokenTree::Literal(a)
             } else {
-                TokenNode::Literal(Literal::integer(3))
+                TokenTree::Literal(Literal::i32_unsuffixed(3))
             }
         }
     }