Token macros

[dtolnay]

I am not dead set on this, but I think it is an improvement. This commit introduces a type macro Token! and parser macros punct! and keyword! used as follows.

ItemExternCrate {
    pub attrs: Vec,
    pub vis: Visibility,
    pub extern_token: Token![extern],
    pub crate_token: Token![crate],
    pub ident: Ident,
    pub rename: Option<(Token![as], Ident)>,
    pub semi_token: Token![;],
}

impl_synom!(ItemExternCrate "extern crate item" do_parse!(
    attrs: many0!(call!(Attribute::parse_outer)) >>
    vis: syn!(Visibility) >>
    extern_: keyword!(extern) >>
    crate_: keyword!(crate) >>
    ident: syn!(Ident) >>
    rename: option!(tuple!(keyword!(as), syn!(Ident))) >>
    semi: punct!(;) >>
    (ItemExternCrate {
        attrs: attrs,
        vis: vis,
        extern_token: extern_,
        crate_token: crate_,
        ident: ident,
        rename: rename,
        semi_token: semi,
    })
));

There are a few advantages.

• I learn this scheme once and then I can be productive. In the previous scheme I frequently have to look up the names of RArrow, Colon2, Star, etc.

• No longer misleading for punctuation that has overloaded meanings. In the previous scheme for example TyRptr, which represents &T and &mut T, contained a tokens::And. But in this context the & symbol means "reference", not "and".

• No longer need to glob import synom::tokens::* which means token names no longer conflict with prelude types or syn types. As part of this commit, I have renamed tokens::Box_, tokens::Default_, and tokens::Fn_ to remove the underscores. I kept tokens::Self_ because that one would be a reserved word without underscore. This also frees up the name Type for Standardize on usage of abbreviations #224.

[mystor]

I like this a lot. I'd say that it's definitely a usability improvement over the old system.

[alexcrichton]

👍