Add full Rust language grammar definition to docs

[icorderi]

Original issue that inspired this patch.

The reference.md has evolved past simple grammatical constructs, and it serves a different purpose.
The intent for the proposed grammar.md is to hold only the official reference for the language grammar. This document would keep track of grammatical changes to the language over time, facilitate discussions over proposed changes to the existing grammar, and serve as basis for building parsers by third-parties (IDE's, GitHub linguist, CodeMirror, etc.).

The current state of the PR contains all the grammars that were available in reference.md and nothing else.
There are still a lot of missing pieces that weren't available. The following are just a few of the definitions missing:

• Functions
• Structures
• Traits
• Implementations
• Operators
• Statements
• Expressions

We need help from people familiar with those grammatical constructs to fill in the missing pieces.

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

[ftxqxd]

I think something like this might work:

fn_decl: "pub"? "unsafe"? ("extern" string_lit?)? "fn" ident ('<' generics '>')? where_clause? '(' arg-list ')' ("->" (type | '!'))? '{' stmt* expr? '}';

(arg-list is defined later in the Closure types section.) Note that functions inside extern blocks have a slightly different grammar:

extern_fn_decl: "pub"? "fn" ident '(' extern_arg_list ')' ("->" (type | '!'))? ';';
extern_arg_list: ident ':' type | ident ':' type ',' "..." | ident ':' type ',' extern_arg_list;

And, furthermore, functions inside trait definitions can have their argument names and/or body omitted, cannot be declared pub, can have self parameters, and have the order of extern and unsafe switched around (#19398):

trait_fn_decl: ("extern" string_lit?)? "unsafe"? "fn" ident ('<' generics '>')? where_clause? '(' trait_arg_list ')' ("->" (type | '!'))? ('{' stmt* expr? '}' | ';');
trait_arg_list: (ident ':')? type | (`&` lifetime)? "mut"? "self" (',' trait_arg_list)? | (ident ':')? type ',' trait_arg_list;

Trait implementations are similar, but cannot have their body or parameter names omitted:

impl_fn_decl: ("extern" string_lit?)? "unsafe"? "fn" ident ('<' generics '>')? where_clause? '(' trait_arg_list ')' ("->" (type | '!'))? '{' stmt* expr? '}';
impl_arg_list: ident ':' type | (`&` lifetime)? "mut"? "self" (',' impl_arg_list)? | ident ':' type ',' impl_arg_list;

I think I’ve handled most things here, but I’m sure I’ve forgotten something. Functions are pretty complicated!

[ftxqxd]

There are a few inconsistencies here:

• Sometimes [ foo ] is used for grouping, sometimes ( foo ) is, and sometimes [ foo ] is used to denote foo being optional;
• Some names use underscores, and some use hyphens;
• Sometimes := is used to define rules, and sometimes : is.

[icorderi]

@P1start I think styling issues might be just due to different people writing pieces of it. They are all valid on some grammar notation or another. The entire documents was copied from the reference.md.
For what is worth, we do need consistency on the rules and the documents starts with a grammar for the grammars, which appears to be a slight modification over EBNF syntax.

grammar : rule + ;
rule    : nonterminal ':' productionrule ';' ;
productionrule : production [ '|' production ] * ;
production : term * ;
term : element repeats ;
element : LITERAL | IDENTIFIER | '[' productionrule ']' ;
repeats : [ '*' | '+' ] NUMBER ? | NUMBER ? | '?' ;

Where:

• Whitespace in the grammar is ignored.
• Square brackets are used to group rules.
• LITERAL is a single printable ASCII character, or an escaped hexadecimal ASCII > code of the form \xQQ, in single quotes, denoting the corresponding Unicode codepoint U+00QQ.
• IDENTIFIER is a nonempty string of ASCII letters and underscores.
• The repeat forms apply to the adjacent element, and are as follows:
  • ? means zero or one repetition
  • * means zero or more repetitions
  • + means one or more repetitions
• NUMBER trailing a repeat symbol gives a maximum repetition count
• NUMBER on its own gives an exact repetition count
  This EBNF dialect should hopefully be familiar to many readers.

Based on that ( foo ) doesn't seems to be part of the grammar.

Personally I prefer defining the grammar with ::= or := for productions. It's an extra key stroke but it makes the rule easier to read, at least to me.

In any case it seems we have to:
0. (pick a grammar...)

1. clean up the existing productions to make them consistent with the grammar we will be using to define them.
2. Complete the missing sections

The questions now is whether we do this collaboration against my fork or we merge this and do it against rust-lang. I don't know how you guys generally handle WIP from multiple sources.

[nodakai]

This is a great step towards a matured programming language! Let me see how I can contribute to it.

I made some remarks on string/char literals with newline characters at #19399 . Hope this helps.

[steveklabnik]

I wonder at what point this is good to just land, and fill them in as we go, rather than let this sit here.

[shepmaster]

Should the grammar sections in the Reference be removed, so they don't go (more) stale?

[icorderi]

@steveklabnik agreed. We should not wait for this to be complete to merge.

@shepmaster, I'm not sure how much having the formal grammar on the reference helps the reader. If this document makes it in and becomes the goto place for the rust grammar then the reference should not have grammar definitions. We can simply leave links on the reference.md to the corresponding grammar on each section in case the reader wants to take a look.

[steveklabnik]

Well, the idea is that the reference is a borderline-language spec, so having the grammar in it is useful. But until we have a full grammar, it doesn't make sense to have part of it there and the rest in this document. We should pull the grammar out of the reference and then link to this instead.

[Gankra]

Triage bump

[emberian]

See also https://github.com/bleibig/rust-grammar/ and https://github.com/rust-lang/rust/tree/master/src/grammar

[emberian]

My basic feeling is that none of the old grammar is worth keeping and we should instead, if we want, distribute a grammar as a separate appendix.

[steveklabnik]

I'm going to merge this as-is, and we can keep improving it, including the part about adding links from the reference. Thank you!

[nikomatsakis]

I think if we're going to distribute an official grammar, we really ought to have an automated testing mechanism. Not to say r- on this patch, just that this should be a priority. I think it'd be very helpful, personally, even if it's not what the actual rustc uses (and even if we don't check for correspondence between rustc and the official grammar, which of course would also be good).

[steveklabnik]

@nikomatsakis we had some of this in src/grammar, but yes, such a thing would be nice.

[steveklabnik]

Oh also, this isn't currently linked from anywhere yet, so while it's kind of official, people may not actually know about it 😉 maybe that's a good critiera for when to add the link.

[emberian]

I also really think that this particular partial grammar is worthless, but we can replace it later with https://github.com/bleibig/rust-grammar or https://github.com/ptgreen/rfront as the need arises.

[steveklabnik]

@bors: r+ ab24ffe

[bors]

⌛ Testing commit ab24ffe with merge a0f86de...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-32-nopt-t, auto-win-32-opt, auto-win-64-nopt-t, auto-win-64-opt

[brson]

Epic PR. Thanks for sticking with it.

[SimonSapin]

Does this new document shows up somewhere on http://doc.rust-lang.org? I couldn’t find it.

[steveklabnik]

@SimonSapin it does not yet, as it's not finished.