Stabilize Span::mixed_site

[petrochenkov]

Closes #65049.
cc #54727 (comment)

Pre-requisite for #68717 ("Stabilize fn-like proc macros in expression, pattern and statement positions").

Stabilization report: #68716 (comment).

[rust-highfive]

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

Stabilization report

This PR stabilizes a proc macro API Span::mixed_site() exposing macro_rules hygiene to proc macro authors.

Tracking issue: #65049
Version target: 1.43

Technical specification

Procedural macro producing tokens with Span::mixed_site() spans should be equivalent to a macro_rules macro producing the same tokens.
Identifiers with this span will resolve at the macro's definition site for local variables, labels and $crate, and resolve at the call site for everything else.

Motivation

Proc macros do not have direct access to our oldest and most stable hygiene kind - macro_rules hygiene.

To emulate it macro authors have to go through two steps - first generate a temporary macro_rules item, then generate a macro call to that item. Popular crates like proc_macro_hack use this trick to generate hygienic identifiers from proc macros.

I'd say that these workarounds with nested macro definitions have more chances to hit some corner cases in our hygiene system, in which we don't have full confidence.
So, let's provide a direct access to macro_rules hygiene instead.
This access is provided with a method Span::mixed_site, which is similar to existing Span::call_site (stable, fully unhygienic) and Span::def_site (unstable, fully hygienic).

This API addition opens the way to stabilizing proc macros in expression, pattern and statement positions (#68717), for which use of call-site hygiene or workarounds with temporary items would be quite unfortunate.
macro_rules expanded in all these positions, on the other hand, are stable since 1.0 and widely used.
So, we are suggesting to stabilize such use of procedural macros and give a way to proc macro authors to make these macros reasonably hygienic with Span::mixed_site.

Tests

ui/proc-macro/mixed-site-span.rs makes sure that the API works.
Otherwise, all tests for macro_rules test the underlying mechanism.

History

The implementation (#64690) was merged about 4 months ago.

[petrochenkov]

r? @dtolnay

[Centril]

This API addition opens the way to stabilizing proc macros in expression, pattern and statement positions (#68717), for which use of call-site hygiene or workarounds with temporary items would be quite unfortunate.

One thing I'm wondering about here is the documentation ("how do we teach this") aspect and making it easy to "do the right thing" wrt. hygiene in expr/pat/stmt contexts. Hygiene is, in my experience, one of the less well known parts of the language and so it seems like this could be easy to get wrong without good teaching material. In the implementation PR, @dtolnay noted (#64690 (comment)) that:

I will try to test whether changing the default span of quote! from call_site to mixed_site on sufficiently new compilers can possibly break anything that's currently stable. The alternative would be adding a macro like quote_expr! that is shorthand for quote_spanned!(Span::mixed_site()=>...).

I'm wondering if more thought has been put into this. @CAD97 did note in #64690 (comment) that changing to mixed_site could trivially break some crates using quote!, which is now a 1.0.X crate, so it seems like the most practical path would be quote_expr! { ... } (although that name is sub-optimal for patterns & statements).

[petrochenkov]

@Centril
I'm not sure this is our problem.
AFAIK, most user-facing APIs are provided by third-party crates, proc_macro only provides primitives.

The recommendation on using Span::mixed_site is #64690 (comment) - use it everywhere unless you specifically want unhygienic local variables, labels or $crate, which is a rare case.

I don't understand why the @CAD97's example would be broken, it doesn't seem to involve local variables.

[Centril]

The teams should, in my view, still be interested in the wider ecosystem effects with respect to #68717 and not just the narrow technical aspect of mixed_site. We should make sure we're not inadvertently laying traps for users.

Specifically, I'm somewhat concerned that the recommendation won't be heeded by many due to the relative ergonomics of quote!(...) vs quote_spanned!(Span::mixed_site()=> ...). So having some sort of plan for how to address that would be good. If quote! can change to mixed_site then the issue is resolved, but if not, it might involve adding a second macro to quote (e.g. q!(...)) and deprecating the other macro. (This is primarily directed at @dtolnay.)

[dtolnay]

I would be interested in a crater run that changes proc_macro::Span::call_site() to call mixed_site(), as a way to mimic what would happen from replacing quote's default. I agree with #68716 (comment) that I don't understand why the example given in the other PR would break.

• If nothing breaks, that's great.
• If the only things that break are things using unstable proc_macro_hygiene, we can evaluate but I am open to breaking those.
• If more complicated, this may be a good time to introduce impl Default for Span which produces call_site or mixed_site depending on whether we are in the expansion of an attribute/item macro vs expr/pat/stmt macro.

[Centril]

A crater run sounds like a great idea. 👍

(I'm skeptical of Default for Span but we can have that discussion when necessary.)

[petrochenkov]

I would be interested in a crater run that changes proc_macro::Span::call_site() to call mixed_site(), as a way to mimic what would happen from replacing quote's default.

#68766 is the PR doing that.

[dtolnay]

#68766 (comment) has the experiment results.

I haven't looked into the regressions too closely, but it seems a large variety of derive macros and attribute macros no longer work. That suggests to me mixed_site may not be a reasonable default and that it should be reserved as a power tool. But I intend to take a closer look at what particular patterns exhibit different behavior (and I could use all of your help with this). It is interesting that serde_derive seems completely fine using mixed_site everywhere.

[petrochenkov]

I'm also interested in finding out the root regressions and patterns that they represent.
It's possible that a few root regressions in some popular crate cause the high total number of regressed crates.

[petrochenkov]

ping @dtolnay, this is unblocked and needs an official r+.

[dtolnay]

@bors r+

[bors]

📌 Commit f5223a3 has been approved by dtolnay