Source changes history

Changeset 15610, Nov 5, 2008 - Syntax coloring

• Support for syntax coloring in customized editor control
• Grammar Explorer form is finally fixed and allows loading arbitrary grammars from compiled assemblies, instead of hard-coded list as it was before

Changeset 12333, Sept 1, 2008 - language runtime

• Implemented LanguageRuntime with dynamic operator dispatch - base class for custom language runtimes. One example is Scheme sample and SchemeRuntime class - a customization of Irony's LanguageRuntime class. Another example - CalcGrammar in Tutorial, uses default runtime implementation and provides evaluation of simple arithmetic expressions.
• Big refactoring: moved LALR parsing classes to a separate folder/namespace; making it possible to create other, non-LALR parsers with the same Grammar-construction methods and Scanner implementation. Planning to implement NLALR (Non-canonical LALR) parser algorithm. As part of refactoring, separated scanner-related data from parser data into ScannerControlData class.
• Implemented GrammarHint class, to allow adding hints for parser in grammar expressions - for example, to explicitly set an action in case of shift/reduce or reduce/reduce conflicts. Grammar.PreferShiftHere() and Grammar.ReduceThis() methods can be used to create hints inside grammar expressions. See GwBasic sample grammar, ELSE_CLAUSE_OPT and FUN_CALL elements for examples of hint use; also see SchemeGrammar.
• Cleaned up Scheme grammar, removed LALR conflicts by rearranging NonTerminals and adding grammar hints. Scheme now has real runtime implementation, supports various data types for arithmetic operations.
• Bug fix: double-escape char at the end of the string literal was not parsed correctly.
• Applied SecurityTransparent attribute to Irony assembly; more info: http://msdn.microsoft.com/en-us/library/bb397858.aspx
• RegexTerminal - enhanced following suggestions by user sakana280.
• CodePlex member "notmasteryet" submitted improved GWBasic grammar - thanks! I've tweaked it a bit to fix the conflicts.

Changeset 10877, May 16, 2008 - it's all about performance

• Changed GrammarData to use StringSets instead of StringLists for lookaheads computations - 2x+ performance improvement in parser initialization
• Dynamic operator dispatch engine - implementation of binary operators (+, -, etc.) and dynamic dispatch to implementation based on arg types. See Runtime/OperatorDispatch.cs file. Not actually used by any samples yet, only in unit test class - see OperatorDispatchTests.cs file for sample use and performance evaluation. And performance is not bad - less than 200 ns per operation on 2Hrz machine.
• Tests moved to VS Test framework - now run directly in Visual Studio, without NUnit.
• Added CreateSchemeNumber method to TerminalFactory (not quite R6RS yet though)

Changeset 10663, May 5, 2008 - a scheme for interpreter and interpeter for Scheme The first elements of interpeter engine. Implementation for trivial subset of Scheme - advanced enough to run 99 bottles. More details are coming. Note: Python and Ruby grammars are temporarily excluded from grammar samples. These grammar classes use the versions of Kleene operators (Star, Plus) that had been deprecated - so until they are updated, they are not available in samples.

Changeset 10309, April 17, 2008 - It is 2008, finally

• Solution migrated to VS 2008/.NET 3.5. Tests are still through NUnit - will be moved to MSTest later
• Added GwBasic grammar
• Improved MakeStarRule implementation with null delimiter - now introduces less conflicts
• Renamed Grammar.ExtraTerminals list to Grammar.NonGrammarTerminals. Added IsNonGrammar as TermOption; for terminals in NonGrammarTerminals this flag is set automatically; and in this case the parser ignores tokens of this type. This usually happens with Comment tokens.
• Fixed CommentTerminal to allow line comments to be the last tokens in the file, without final line break.

Changeset 10146, April 12, 2008 - Time to get real

• First REAL language grammar - c# 3.0 grammar. All language features except LINQ queries and preprocessor directives.
• Philipp Serr completed implementation of NumberTerminal with support for Complex numbers, Big Integers, and some advanced features required by c#, Python, VB
• Deprecated Kleene operators methods BnfTerm.Plus, Star(). Introduced Grammar.MakePlusRule, MakeStarRule methods that should be used for creating lists.
• Removed ReservedWord identifier, introduced Token.IsKeyword flag; renamed ReservedWords to Keywords, created internal hash table for quick keyword lookup.
• Refactored GrammarDataBuilder method for performance improvements
• Added Parser.PreviewSymbols method for resolving conflicts; method finds which of the tokens in a given list is coming first
• Add LineTerminators property to Grammar, to explicitly set characters used by Scanner for line counting
• CommentTerminal now allows multiple end symbols - used in c# LineComment terminal which can end with one of several line terminator symbols defined in c#.
• Added AstNode.Span property of type SourceSpan (StartLocation + Length) to identify the span of source code for the node
• Created AstNodeArgs struct - container for AstNode constructor arguments. The purpose is to simplify custom AstNode signatures
• Created static method CompilerContext.CreateDummy for creating dummy context for unit tests
• Create Token.CreateMultitoken method, to allow terminals to return multiple tokens packed in one.
• Improved error handling in GrammarDataBuilder & Explorer
• Added Grammar.FallbackTerminals - terminals without explicitly declared Firsts (start chars) or having start chars defined by Unicode category (like c# Identifier)
• Added skeleton classes and methods for language Runtime implementation.

Changeset 9122, Mar 7, 2008 - Terminals, part II

• Added CompoundTerminalBase class as an abstract base class for terminals having 3-part structure - prefix-body-suffix. Refactored Identifier, StringLiteral and NumberLiteral classes to inherit from this class. For more explanation of the concept see comments on top of the source file.
• Added a property Token.Details (of type ScanDetails) containing (optionally) the detailed information from scanning compound tokens: prefix, suffix, flags, etc.
• Full-featured implementation of IdentifierTerminal supporting all fancy features of c# 3.0 identifiers - prefixes (@), escaped chars, non-ASCII charsets, specifying allowed chars by unicode categories (including formatting chars)
• Removed NodeCreator delegate, added NodeCreating, NodeCreated events on NonTerminal - following standard template
• Renamed BnfElement to BnfTerm, BnfFlags to TermOptions
• Added DebuggerStepThrough attribute to properties and one-liners
• Added ISourceStream.MatchSymbol method - simplifies code in terminals
• Added Search box to GrammarExplorer for searching text in top/active textbox - contributed by Andrew Bradnan.

Changeset 8968, Mar 3, 2008 - It's all about Terminals

• Fully implemented StringLiteral, now supports escaped characters, multi-line strings; doubled quote inside as single.
• Fully implemented NumberTerminal - exponent support; custom exponent symbols; support for hex prefixes (0x...), type suffixes; support for BIG INTEGERS
• New TerminalFactory static class with methods for creating Number and String literals for c#, VB, Python
• Unit tests for terminals - many of them. See Readme file for instructions on running unit tests.
• Comment terminal - removed a special treatment of Line comments; comment now is StartSymbol...EndSymbol; line comments fit well into this, for example in c#: StartSymbol="//", EndSymbol = "\n"
• Removed CharLiteral class - for c# char literal use StringLiteral with IsChar option
• Added Script.NET grammar to the list of sample grammars
• Removed ErrorTerminal class, using base Terminal class for Grammar.Error singleton

Changeset 7937, Feb 15

• Repaired terminal sorting for scanning
• Fixed terminals sorting for display
• Fixed BnfElement.Star(delimiter) method - caused false grammar conflicts
• Fixed parsing for case-insensitive grammars - now should work correctly
• Fixed exception when NonTerminal's Rule property was not assigned - now reports error nicely and cancels the process
• Refactored Scanner to accept ISourceStream instead of SourceFile instance; a move towards using general Stream as source
• Refactored handling punctuation symbols - added IsPunctuation flag to BnfElement and removed Grammar.PunctuationSymbols symbols collection; use Grammar.RegisterPunctuation method for setting the flag in punctuation symbols and even NonTerminals
• Refactored operator symbols handling - added IsOperator flag, precedence and associativity properties directly to SymbolTerminal
• Streamlined AstNode creation process - moved most of the logic to Parser.CreateNode method, cleaned it up. (Python grammar is a bit broken as a result - AST tree shows some "extra" symbols)
• Deprecated GenericNode - AstNode is main base class for all Ast nodes
• Added facilities for VS integration: Scanner.GetNext method, returning tokens one-by-one; BraceMatchTokenFilter builds full list of brace pairs.
• Basic AST Visitor facility: AstNode.AcceptVisitor(IAstVisitor)

Changeset 6381

• Refactored internal representation of BnfExpression implementation, now without ExpressionType enumeration property
• Improved grammar data presentation by removing \b symbol from names
• Renamed NotTerminal.Expression property to Rule
• Added Values dictionary to CompilerContext
• Added Compiler property to CompilerContext
• Added Attributes property (dictionary) to AstNode class, to store any custom values associated with node/token
• Implemented basic error reporting and recovery
• Scanner error recovery: added "Delimiters" list to Grammar: on error scanner skips all until whitespace or delimiter.
• Added ErrorRule property to Non-terminal to define error recovery paths. Error expressions can also be mixed with normal expressions in Rule property
• Added ErrorTerminal and SyntaxError singleton in Grammar, to be used in error rules
• Added TryMatch method to grammar as backup for token recognition; you can use this method to create "fancy" tokens - those that use extended character sets for example and don't have limited set of prefixes.
• Added error rules to Ruby grammar and source sample with errors for Ruby
• Added Terminal parameter to MatchHandler delegate to provide reference to caller object