Add initial regex source generator

[stephentoub]

This adds the [RegexGenerator(...)] attribute and a new System.Text.RegularExpressions.Generator source generator for Regex. Given a method like:

[RegexGenerator("abc")]
private static partial Regex Abc();

it will generate C# code to implement Abc() that returns a Regex instance with code like you'd have gotten with RegexOptions.Compiled, but generated at build time in C# rather than at run time with reflection emit in MSIL. The generator is largely a port of RegexCompiler.cs; ideally we wouldn't duplicate all of the emit logic, once for MSIL and once for C#, and we should subsequently look for ways to avoid this duplication. The generator includes source from System.Text.RegularExpressions, in particular in support of parsing and analyzing regexes.

It includes a test project for the generator focused on testing the mechanics of the generator as well as integrating the source generator into the regex test suite, rolled out into many but not all of the tests so far.

There are a bunch of follow-ups to this for while I'll be opening issues, including rolling this out through the rest of the tests (and finding ways to make it go faster, as it's currently significantly increasing the time for the regex test suite), fixing a bunch of TODOs (e.g. figuring out what to do about culture), and improving perf further (it's now really easy to see opportunities for improved code generation).

Fixes #58880
Contributes to #44676

cc: @danmoseley, @pgovind, @tannergooding, @eerhardt, @veanes, @olsaarik

[ghost]

Tagging subscribers to this area: @eerhardt, @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This adds the [RegexGenerator(...)] attribute and a new System.Text.RegularExpressions.Generator source generator for Regex. Given a method like:

[RegexGenerator("abc")]
private static partial Regex Abc();

it will generate C# code to implement Abc() that returns a Regex instance with code like you'd have gotten with RegexOptions.Compiled, but generated at build time in C# rather than at run time with reflection emit in MSIL. The generator is largely a port of RegexCompiler.cs; ideally we wouldn't duplicate all of the emit logic, once for MSIL and once for C#, and we should subsequently look for ways to avoid this duplication. The generator includes source from System.Text.RegularExpressions, in particular in support of parsing and analyzing regexes.

It includes a test project for the generator focused on testing the mechanics of the generator as well as integrating the source generator into the regex test suite, rolled out into many but not all of the tests so far.

There are a bunch of follow-ups to this for while I'll be opening issues, including rolling this out through the rest of the tests (and finding ways to make it go faster, as it's currently significantly increasing the time for the regex test suite), fixing a bunch of TODOs (e.g. figuring out what to do about culture), and improving perf further (it's now really easy to see opportunities for improved code generation).

Fixes #58880
Contributes to #44676

cc: @danmoseley, @pgovind, @tannergooding, @eerhardt, @veanes, @olsaarik

Author:	stephentoub
Assignees:	-
Labels:	area-System.Text.RegularExpressions
Milestone:	7.0.0

[dotnet-issue-labeler]

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, to please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

[stephentoub]

@jaredpar, FYI.

[danmoseley]

can we not reuse the existing resx?

[stephentoub]

How? i.e. is there a mechanism for including an additional resx file and having them merged, or something like that?

[danmoseley]

I meant just use the existing one - of course not all the strings apply. Perhaps that would get complex given this one is localized.
No I don't know a way to include one in another.

[stephentoub]

I meant just use the existing one

And add the generator specific strings into the existing one?

[danmoseley]

If that's less wasteful than the duplication. Feel free to ignore.

[danmoseley]

Will actually be interesting now we can look at the C# generated for different regexes to more easily see how it works.

[Szer]

Will there be any warning if that library code (supposed to be language agnostic?) would be called NOT from Roslyn?

[stephentoub]

Will there be any warning if that library code (supposed to be language agnostic?) would be called NOT from Roslyn?

I don't understand the question. Regex is usable from all .NET languages. If you try to use RegexGenerator with something other than C#, you'll get errors because no code will be generated. Can you clarify?