update

Signed-off-by: George Lemon <[email protected]>
openpeeps · Aug 7, 2023 · 562bd1a · 562bd1a
1 parent b7e265f
commit 562bd1a
Showing 1 changed file with 41 additions and 74 deletions.
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 <p align="center">
-    <img src="https://raw.githubusercontent.com/openpeep/toktok/main/.github/logo.png" width="140px"><br>
-    Generic tokenizer written in Nim language, powered by Nim's Macros 👑   
+  <img src="https://raw.githubusercontent.com/openpeep/toktok/main/.github/logo.png" width="140px"><br>
+  Generic tokenizer written in Nim language, powered by Nim's Macros 👑
 </p>
 
 <p align="center">
@@ -14,90 +14,57 @@
 
 
 ## 😍 Key Features
-- [x] Support `char`, `string`, `int` based tokens
-- [x] Powered by Nim's Macros ✨
-- [x] No RegEx 🍃
-- [x] Meta-programming `TokenKind*` `enum`
-- [x] Meta-programming `case` statement for all tokens
-- [x] `getToken()` procedure to retrieve token by token
-- [ ] Multi `TokenKind` handler in one line (`=`, `==`, `!==`)
-- [x] Open Source | `MIT`
+- ✨ Powered by Nim's Macros
+- 🪄 Based on `std/lexbase` / Zero Regular Expression
+- Static generation using **TokenKind** `enum`, `lexbase`
+- Dynamic generation using **TokenKind** `tables`, `lexbase`
+- Open Source | `MIT`
 
-Toktok is a generic Lexer, based on standard Nim libraries `streams`, `lexbase` and `macros`.
-It is meant to be used by higher level parsers for writing any kind of tools or programs.
-
-## Debug
-Compile with `-d:toktokdebug` to print your tokens
+> __Note__ Toktok is a generic Lexer, based on standard Nim libraries `streams`, `lexbase` and `macros`. It is meant to be used by higher level parsers for writing any kind of tools or programs.
+> __Note__ Compile with `-d:toktokdebug` to inspect the generated code.
 
 ## Quick Example
 
-<details>
-    <summary>Show sample.txt example</summary>
-
-```
-const hello = 1 + 1
-```
-
-</details>
-
-```nim
-import toktok
-
-static:
-  Program.settings(
-    uppercase = true,
-    prefix = "Tk_",
-    allowUnknown = false,
-    keepUnknownChars = false
-  )
-
-tokens:
-  Plus      > '+'
-  Minus     > '-'
-  Multi     > '*'
-  Div       > '/':
-    BlockComment ? '*' .. "*/"
-    InlineComment ? '/' .. EOL
-  Assign      > '='
-  Var         > "var"
-  Let         > "let"
-  Const       > "const"
-  SetTrue     > {"TRUE", "True", "true", "YES", "Yes", "yes", "y"}
-  SetFalse    > {"FALSE", "False", "false", "NO", "No", "no", "n"}
-
-when isMainModule:
-  var lex = Lexer.init(fileContents = readFile("sample.txt"))
-  if lex.hasError:
-    echo lex.getError
-  else:
-    while true:
-      var curr = lex.getToken()           # tuple[kind: TokenKind, value: string, wsno, col, line: int]
-      if curr.kind == TK_EOF: break
-      echo curr
-```
-
-<details>
-    <summary>See output</summary>
-
 ```nim
-(kind: TK_CONST, value: "const", wsno: 1, col: 0, line: 1)
-(kind: TK_IDENTIFIER, value: "hello", wsno: 1, col: 6, line: 1)
-(kind: TK_ASSIGN, value: "", wsno: 0, col: 12, line: 1)
-(kind: TK_INTEGER, value: "1", wsno: 1, col: 15, line: 1)
-(kind: TK_PLUS, value: "", wsno: 0, col: 16, line: 1)
-(kind: TK_INTEGER, value: "1", wsno: 1, col: 19, line: 1)
+# Register your tokens
+registerTokens defaultSettings:
+  `const` = "const"
+  `echo` = "hello"
+  asgn = '=':   # `=` is tkAsgn
+    eq = '='    # `==` is tkEQ
+  excl = '!':
+    ne = '='
+  at = '@':
+    import = tokenizer(handleImport, "import") 
+
+
+# Tokenizing...
+var
+  tok = lexer.init(sample)
+  prev: TokenTuple
+  curr: TokenTuple = tok.getToken
+  next: TokenTuple = tok.getToken 
+
+proc walk(tok: var Lexer) =
+  prev = curr
+  curr = next
+  next = tok.getToken
+
+while likely(curr.kind != tkEOF):
+  if tok.hasError: break
+  echo curr # use `getToken` consumer to get token by token
+  walk tok
 ```
 
-</details>
-
-<a href="https://hetzner.cloud/?ref=Hm0mYGM9NxZ4"><img src="https://openpeep.ro/banners/openpeep-footer.png" width="100%"></a>
+## TODO
+- [ ] Runtime Token generation using tables/critbits
 
 ### ❤ Contributions & Support
 - 🐛 Found a bug? [Create a new Issue](https://github.com/openpeeps/toktok/issues)
 - 👋 Wanna help? [Fork it!](https://github.com/openpeeps/toktok/fork)
 - 😎 [Get €20 in cloud credits from Hetzner](https://hetzner.cloud/?ref=Hm0mYGM9NxZ4)
-- 🥰 [Donate to The Enthusiast via PayPal address](https://www.paypal.com/donate/?hosted_button_id=RJK3ZTDWPL55C)
+- 🥰 [Donate to OpenPeeps via PayPal address](https://www.paypal.com/donate/?hosted_button_id=RJK3ZTDWPL55C)
 
 ### 🎩 License
-`toktok` | `MIT` license. [Made by Humans from OpenPeeps](https://github.com/openpeeps).<br>
+`MIT` license. [Made by Humans from OpenPeeps](https://github.com/openpeeps).<br>
 Copyright &copy; 2023 OpenPeeps & Contributors &mdash; All rights reserved.