[RFC FS-1001] String Interpolation (dotnet#8907)

* string interploation implementation

* string interploation tests

* escape {{ }}, test verbatim and triple quote, implement .NET specifiers

* fix tests

* string interpolation tests: internal representation corner cases

* string-interp tests should have --langversion:preview

* string interop tests: sprintf

* string interp tests: format specifier negative cases

* string interp tests: format specifier negative cases, .NET-style padding

* fix nested interp strings

* style cleanup

* lex: unify string interp stack and counter

* string-interp: add test cases

* fix mixed quote nested string interpolation

* string-interp: add test case for multiple interpolation points with different indentation

* lexfilter: push new CtxtParen at endPos for INTERP_STRING_PART and INTERP_STRING_BEGIN_PART

* lexfilter: do not check undentation limit for string interpolation tokens.

* FormattableString prototype

* add FormattableString support

* negative error checking

* remove diagnostics

* simpler FormattableString implementation

* fix test

* add testing for nested

* add IFormattable support

* tweak error message

* tests: StringInterpolation: fix case errors

* fix error message

* check number of values matches

* allow use of format strings with printf and friends

* update baselines

* fix baselines

* add Experimental attributes

* update string interp negative tests

* stringinterp test: add PrintFormat tests

* printf: fix empty interpolation string evaluates to null in printf env

* enable test corectly

* Revert "printf: fix empty interpolation string evaluates to null in printf env"

This reverts commit 7f39617.

* simplify codegen for interpolated strings

* fix build

* fix build

* Merge master to feature/string-interp (dotnet#9580)

* Update dependencies from https://github.com/dotnet/arcade build 20200626.2 (dotnet#9577)

Microsoft.DotNet.Arcade.Sdk
 From Version 1.0.0-beta.20302.3 -> To Version 1.0.0-beta.20326.2

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>

* Improve perf for String.filter up to 3x (dotnet#9509)

* Improve perf for String.filter 2-2.5x

* Cleanup: remove "foo" etc in tests

* Add tests for new execution path for LOH in String.filter

* Change test string

* String map performance improvement (dotnet#9470)

* Simplify and improve perf of String.length

* Improve performance of String.map

* Revert "Simplify and improve perf of String.length"

* Resolves dotnet#9470 (comment)

* Lingering space

* Change `String` to use `new` to clarify use of ctor

* Add some better tests for String.map, add side-effect test

* Add tests to ensure the mapping function is called a deterministically amount of times

* Fix typo

* Remove "foo" from String.map tests

* Perf: String.replicate from O(n) to O(log(n)), up to 12x speed improvement (dotnet#9512)

* Turn String.replicate from O(n) into O(log(n))

* Cleanup String.replicate tests by removing usages of "foo"

* String.replicate: add tests for missing cases, and for the new O(log(n)) cut-off points

* Improve String.replicate algorithm further

* Add tests for String.replicate covering all lines/branches of algo

* Fix accidental comment

Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Abel Braaksma <[email protected]>

* Re enable tests for operators: OperatorsModule1.fs and OperatorsModule2.fs (dotnet#9516) (dotnet#9589)

* Re-enabling tests from OperatorsModule1/2.fs (compile errors)

* Fix compile errors in OperatorsModule1/2.fs, fix tests. Note tanh test comment.

* Fix `tanh` test, ensure stable result on x86 vs x64 runtimes

* Stop using exception AssertionException, so that test window shows useful info

* Whitespace cleanup and redundant code removal

* Cleanup spelling etc

* Re-enabling int, int16, int32, int64, nativeint, incr, nullArg etc tests

* Special-case floating-point assertion messages for higher precision output

* Fix/update/add tests (some still failing)

* Separate Checked tests, add & fix others, differentiate framework/bitness for some tests

* Add branch for .NET Native (ignore cos test)

* Resorting to comparing floats with a delta using Assert.AreNearEqual

* Add some more tests

Co-authored-by: Abel Braaksma <[email protected]>

* Moved fsharpqa/Libraries/Core/Unchecked test cases to NUnit (dotnet#9576) (dotnet#9599)

Co-authored-by: Thorsten Reichert <[email protected]>

* Moved fsharpqa/Libraries/Core/Unchecked test cases to NUnit (dotnet#9576) (dotnet#9604)

Co-authored-by: Thorsten Reichert <[email protected]>

* Merge master to feature/string-interp (dotnet#9615)

* Moved fsharpqa/Libraries/Core/Unchecked test cases to NUnit (dotnet#9576)

* Moved fsharpqa/Libraries/Core/Reflectiontest cases to NUnit (dotnet#9611)

* Migrated PreComputedTupleConstructor01.fs test case

* Migrated PreComputedTupleConstructor02.fs test case

* Migrated DU.fs and Record.fs test cases

* Allow notebook to discover location of shared framework (dotnet#9596)

Co-authored-by: Thorsten Reichert <[email protected]>
Co-authored-by: Kevin Ransom (msft) <[email protected]>
Co-authored-by: Phillip Carter <[email protected]>

* Merge master to feature/string-interp (dotnet#9619)

* Moved fsharpqa/Libraries/Core/Unchecked test cases to NUnit (dotnet#9576)

* Moved fsharpqa/Libraries/Core/Reflectiontest cases to NUnit (dotnet#9611)

* Migrated PreComputedTupleConstructor01.fs test case

* Migrated PreComputedTupleConstructor02.fs test case

* Migrated DU.fs and Record.fs test cases

* Allow notebook to discover location of shared framework (dotnet#9596)

Co-authored-by: Thorsten Reichert <[email protected]>
Co-authored-by: Kevin Ransom (msft) <[email protected]>

* Text tweeks

* don't auto-resolve types from System.Runtime.WindowsRuntime (dotnet#9644) (dotnet#9648)

Co-authored-by: Brett V. Forsgren <[email protected]>

* yeet (dotnet#9657) (dotnet#9661)

yeet

Co-authored-by: Phillip Carter <[email protected]>

* yeet (dotnet#9657) (dotnet#9670)

yeet

Co-authored-by: Phillip Carter <[email protected]>

* fix up tokenizer tests

* fix code review things

* fix code review things

* fix code review things

* fix code review things

* add various testing

* correct continuations for interpolated strings

* fix lexer continuations and colorization for multi-line interpolated strings

* revert xlf changes

* fix assert

* completion and brace matching (not all tests passing yet)

* Fix rebuild

* fix various niggles and get tests working

* fix printf when '%a' in final position

* fix test case

* interpolated string specifer highlighting

* fix triple quote interpolated string specifer highlighting

* fix triple quote interpolated string specifer highlighting

* fix build

* fix missing error message

* fix % specifiers for interpolated strings

* fix % specifiers for interpolated strings

* fix FCS tests

* minor nits from code review

* code review feedback and use struct tuples in more places

* revert struct tuples

* use struct tuples where possible, byrefs for index

* fix byref for index

* fix ksprintf block size

* make recent cache entry more explicit (cleanup)

* improve performance

* remove unused code

* Move existing Compiler.ComponentTests to a new Compiler.fs framework (dotnet#9839) (dotnet#9848)

* Move existing Compiler.ComponentTests to a new Compiler.fs framework; Add 'parse' function

* Changed some wording in error messages

Co-authored-by: Vlad Zarytovskii <[email protected]>

* Move existing Compiler.ComponentTests to a new Compiler.fs framework (dotnet#9839)

* Move existing Compiler.ComponentTests to a new Compiler.fs framework; Add 'parse' function

* Changed some wording in error messages

* fix dotnet#9893

* fix unmantched right brace in interp string

Co-authored-by: Yatao Li <[email protected]>
Co-authored-by: Kevin Ransom (msft) <[email protected]>
Co-authored-by: dotnet bot <[email protected]>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Abel Braaksma <[email protected]>
Co-authored-by: Thorsten Reichert <[email protected]>
Co-authored-by: Phillip Carter <[email protected]>
Co-authored-by: Brett V. Forsgren <[email protected]>
Co-authored-by: Vlad Zarytovskii <[email protected]>

src/fsharp/CheckFormatStrings.fsi

@@ -9,9 +9,10 @@ module internal FSharp.Compiler.CheckFormatStrings
 
 open FSharp.Compiler 
 open FSharp.Compiler.NameResolution
+open FSharp.Compiler.Range
 open FSharp.Compiler.TypedTree 
 open FSharp.Compiler.TcGlobals
 
-val ParseFormatString : Range.range -> TcGlobals -> formatStringCheckContext: FormatStringCheckContext option -> fmt: string -> bty: TType -> cty: TType -> dty: TType -> (TType * TType) * (Range.range * int) list
+val ParseFormatString : m: range -> fragmentRanges: range list -> g: TcGlobals -> isInterpolated: bool -> isFormattableString: bool -> formatStringCheckContext: FormatStringCheckContext option -> fmt: string -> bty: TType -> cty: TType -> dty: TType -> TType list * TType * TType * TType[] * (range * int) list * string
 
-val TryCountFormatStringArguments : m:Range.range -> g:TcGlobals -> fmt:string -> bty:TType -> cty:TType -> int option
+val TryCountFormatStringArguments: m: range -> g: TcGlobals -> isInterpolated: bool -> fmt:string -> bty:TType -> cty:TType -> int option

src/fsharp/CompileOps.fs

@@ -1166,6 +1166,10 @@ let OutputPhasedErrorR (os: StringBuilder) (err: PhasedDiagnostic) (canSuggestNa
               | Parser.TOKEN_EOF -> getErrorString("Parser.TOKEN.EOF")
               | Parser.TOKEN_CONST -> getErrorString("Parser.TOKEN.CONST")
               | Parser.TOKEN_FIXED -> getErrorString("Parser.TOKEN.FIXED")
+              | Parser.TOKEN_INTERP_STRING_BEGIN_END -> getErrorString("Parser.TOKEN.INTERP.STRING.BEGIN.END")
+              | Parser.TOKEN_INTERP_STRING_BEGIN_PART -> getErrorString("Parser.TOKEN.INTERP.STRING.BEGIN.PART")
+              | Parser.TOKEN_INTERP_STRING_PART -> getErrorString("Parser.TOKEN.INTERP.STRING.PART")
+              | Parser.TOKEN_INTERP_STRING_END -> getErrorString("Parser.TOKEN.INTERP.STRING.END")
               | unknown ->           
                   Debug.Assert(false, "unknown token tag")
                   let result = sprintf "%+A" unknown
@@ -3588,13 +3592,13 @@ let ParseOneInputLexbuf (tcConfig: TcConfig, lexResourceManager, conditionalComp
     use unwindbuildphase = PushThreadBuildPhaseUntilUnwind BuildPhase.Parse
     try 
         let skip = true in (* don't report whitespace from lexer *)
-        let lightSyntaxStatus = LightSyntaxStatus (tcConfig.ComputeLightSyntaxInitialStatus filename, true) 
-        let lexargs = mkLexargs (filename, conditionalCompilationDefines@tcConfig.conditionalCompilationDefines, lightSyntaxStatus, lexResourceManager, [], errorLogger, tcConfig.pathMap)
+        let lightStatus = LightSyntaxStatus (tcConfig.ComputeLightSyntaxInitialStatus filename, true) 
+        let lexargs = mkLexargs (conditionalCompilationDefines@tcConfig.conditionalCompilationDefines, lightStatus, lexResourceManager, [], errorLogger, tcConfig.pathMap)
         let shortFilename = SanitizeFileName filename tcConfig.implicitIncludeDir 
         let input = 
             Lexhelp.usingLexbufForParsing (lexbuf, filename) (fun lexbuf ->
                 if verbose then dprintn ("Parsing... "+shortFilename)
-                let tokenizer = LexFilter.LexFilter(lightSyntaxStatus, tcConfig.compilingFslib, Lexer.token lexargs skip, lexbuf)
+                let tokenizer = LexFilter.LexFilter(lightStatus, tcConfig.compilingFslib, Lexer.token lexargs skip, lexbuf)
 
                 if tcConfig.tokenizeOnly then 
                     while true do 

src/fsharp/ErrorLogger.fs

@@ -702,4 +702,9 @@ let internal tryLanguageFeatureErrorRecover langVersion langFeature m =
     | None -> ()
 
 let internal tryLanguageFeatureErrorOption langVersion langFeature m =
-    tryLanguageFeatureErrorAux langVersion langFeature m 
+    tryLanguageFeatureErrorAux langVersion langFeature m
+
+let internal languageFeatureNotSupportedInLibraryError (langVersion: LanguageVersion) (langFeature: LanguageFeature) (m: range) =
+    let featureStr = langVersion.GetFeatureString langFeature
+    let suggestedVersionStr = langVersion.GetFeatureVersionString langFeature
+    error (Error(FSComp.SR.chkFeatureNotSupportedInLibrary(featureStr, suggestedVersionStr), m))

src/fsharp/FSComp.txt

@@ -1490,6 +1490,7 @@ notAFunctionButMaybeDeclaration,"This value is not a function and cannot be appl
 3350,chkFeatureNotLanguageSupported,"Feature '%s' is not available in F# %s. Please use language version %s or greater."
 3351,chkFeatureNotRuntimeSupported,"Feature '%s' is not supported by target runtime."
 3352,typrelInterfaceMemberNoMostSpecificImplementation,"Interface member '%s' does not have a most specific implementation."
+3353,chkFeatureNotSupportedInLibrary,"Feature '%s' requires the F# library for language version %s or greater."
 useSdkRefs,"Use reference assemblies for .NET framework references when available (Enabled by default)."
 optsLangVersion,"Display the allowed values for language version, specify language version such as 'latest' or 'preview'"
 optsSupportedLangVersions,"Supported language versions:"
@@ -1509,11 +1510,27 @@ featureFixedIndexSlice3d4d,"fixed-index slice 3d/4d"
 featureAndBang,"applicative computation expressions"
 featureNullableOptionalInterop,"nullable optional interop"
 featureDefaultInterfaceMemberConsumption,"default interface member consumption"
+featureStringInterpolation,"string interpolation"
 featureWitnessPassing,"witness passing for trait constraints in F# quotations"
 3353,fsiInvalidDirective,"Invalid directive '#%s %s'"
 3360,typrelInterfaceWithConcreteAndVariable,"'%s' cannot implement the interface '%s' with the two instantiations '%s' and '%s' because they may unify."
 3361,typrelInterfaceWithConcreteAndVariableObjectExpression,"You cannot implement the interface '%s' with the two instantiations '%s' and '%s' because they may unify."
 featureInterfacesWithMultipleGenericInstantiation,"interfaces with multiple generic instantiation"
 3362,tcLiteralFieldAssignmentWithArg,"Cannot assign '%s' to a value marked literal"
 3363,tcLiteralFieldAssignmentNoArg,"Cannot assign a value to another value marked literal"
+forFormatInvalidForInterpolated,"Interpolated strings may not use '%%' format specifiers unless each is given an expression, e.g. '%%d{{1+1}}'."
+forFormatInvalidForInterpolated2,".NET-style format specifiers such as '{{x,3}}' or '{{x:N5}}' may not be mixed with '%%' format specifiers."
+forFormatInvalidForInterpolated3,"The '%%P' specifier may not be used explicitly."
+forFormatInvalidForInterpolated4,"Interpolated strings used as type IFormattable or type FormattableString may not use '%%' specifiers, only .NET-style interpolands such as '{{expr}}', '{{expr,3}}' or '{{expr:N5}}' may be used."
+3371,tcInterpolationMixedWithPercent,"Mismatch in interpolated string. Interpolated strings may not use '%%' format specifiers unless each is given an expression, e.g. '%%d{{1+1}}'"
+3372,tcInvalidAlignmentInInterpolatedString,"Invalid alignment in interpolated string"
+3373,lexSingleQuoteInSingleQuote,"Invalid interpolated string. Single quote or verbatim string literals may not be used in interpolated expressions in single quote or verbatim strings. Consider using an explicit 'let' binding for the interpolation expression or use a triple quote string as the outer string literal."
+3374,lexTripleQuoteInTripleQuote,"Invalid interpolated string. Triple quote string literals may not be used in interpolated expressions. Consider using an explicit 'let' binding for the interpolation expression."
+3376,tcUnableToParseInterpolatedString,"Invalid interpolated string. %s"
+3377,lexByteStringMayNotBeInterpolated,"a byte string may not be interpolated"
+3378,parsEofInInterpolatedStringFill,"Incomplete interpolated string expression fill begun at or before here"
+3379,parsEofInInterpolatedString,"Incomplete interpolated string begun at or before here"
+3380,parsEofInInterpolatedVerbatimString,"Incomplete interpolated verbatim string begun at or before here"
+3381,parsEofInInterpolatedTripleQuoteString,"Incomplete interpolated triple-quote string begun at or before here"
+3382,lexRBraceInInterpolatedString,"A '}}' character must be escaped (by doubling) in an interpolated string."
 #3501   "This construct is not supported by your version of the F# compiler" CompilerMessage(ExperimentalAttributeMessages.NotSupportedYet, 3501, IsError=true)

src/fsharp/FSStrings.resx

@@ -441,6 +441,18 @@
   <data name="Parser.TOKEN.FIXED" xml:space="preserve">
     <value>keyword 'fixed'</value>
   </data>
+  <data name="Parser.TOKEN.INTERP.STRING.BEGIN.END" xml:space="preserve">
+    <value>interpolated string</value>
+  </data>
+  <data name="Parser.TOKEN.INTERP.STRING.BEGIN.PART" xml:space="preserve">
+    <value>interpolated string (first part)</value>
+  </data>
+  <data name="Parser.TOKEN.INTERP.STRING.PART" xml:space="preserve">
+    <value>interpolated string (part)</value>
+  </data>
+  <data name="Parser.TOKEN.INTERP.STRING.END" xml:space="preserve">
+    <value>interpolated string (final part)</value>
+  </data>
   <data name="Parser.TOKEN.CONSTRAINT" xml:space="preserve">
     <value>keyword 'constraint'</value>
   </data>