Skip to content

Commit

Permalink
[doc] More updates to architectures-notes.md.
Browse files Browse the repository at this point in the history
  • Loading branch information
Andy Chu committed Mar 18, 2020
1 parent 454ca55 commit 99f3349
Show file tree
Hide file tree
Showing 2 changed files with 48 additions and 43 deletions.
83 changes: 42 additions & 41 deletions doc/architecture-notes.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
---
in_progress: yes
---

Notes on OSH Architecture
=========================

Expand All @@ -15,24 +11,59 @@ Notes on OSH Architecture
This doc is for contributors or users who want to understand the Oil codebase.
These internal details are subject to change.

This article is more polished and covers some of the same material: [How to
Parse Shell Like a Programming Language][parse-shell].

[parse-shell]: https://www.oilshell.org/blog/2019/02/07.html#three-nice-properties-of-a-parser

<div id="toc">
</div>

## Source Code

[README](README.html) describes how the code is organized.

### Build Dependencies

- Essential: [libc]($xref)
- Optional: GNU [readline]($xref) (TODO: other line editing libraries).
- Only in the OVM build (as of March 2020): [yajl]($xref)

### Borrowed Code

- [ASDL]($oil-src:asdl/) front end from [CPython]($xref:cpython) (heavily
refactored)
- [frontend/tdop.py]($oil-src): Adapted from tinypy, but almost no original code
remains
- [pgen2]($oil-src:pgen2/)
- All of OPy (will be obsolete)
- compiler2 from stdlib
- byterun
- Build Dependency: [MyPy]($xref:mypy)

### Metaprogramming / Generated Code

- Try `ls */*_def.py */*_gen.py`
- The `def.py` files are abstract definitions. They're not translated by
[mycpp]($xref).
- The `gen.py` files generate source code in Python and C++ from these
definitions.
- For example, we define the core `Id` type and the lexing rules abstractly.
- See [build/dev.sh]($oil-src) and [build/codegen.sh]($oil-src)


## Lexing

Note: This article is more polished and covers some of the material in the
next two sections: [How to Parse Shell Like a Programming
Language][parse-shell].

[parse-shell]: https://www.oilshell.org/blog/2019/02/07.html


### List of Regex-Based Lexers

Oil uses regex-based lexers, which are turned into efficient C code with
[re2c]($xref). We intentionally avoid hand-written code that manipulates
strings char-by-char, since that strategy is error prone; it's inevitable that
rare cases will be mishandled.

The list of lexers can be found by looking at [native/fastlex.c]($oil-src):
The list of lexers can be found by looking at [native/fastlex.c]($oil-src).

- The large, modal OSH/Oil lexer in [frontend/lexer_def.py]($oil-src).
- Lexers for OSH sublanguages
Expand Down Expand Up @@ -118,7 +149,7 @@ These are handled up front, but not in a single pass.

- See [frontend/parse_lib.py]($oil-src) and its callers.

## Runtime Issues
## Runtime Parsing

### Where OSH Dynamically Parses

Expand Down Expand Up @@ -222,36 +253,6 @@ Other Pairs:
- [printf]($help) can have a static variant like `${myfloat %.3f}`
- `find` and our own language (although this may be done with blocks)

## Build Time

### Dependencies

- Essential: [libc]($xref)
- Optional: readline
- Only in OVM now: [yajl]($xref)

### Borrowed Code

- [ASDL]($oil-src:asdl/) front end from [CPython]($xref:cpython) (heavily
refactored)
- [frontend/tdop.py]($oil-src): Adapted from tinypy, but almost no original code
remains
- [pgen2]($oil-src:pgen2/)
- All of OPy (will be obsolete)
- compiler2 from stdlib
- byterun
- Build Dependency: [MyPy]($xref:mypy)

### Metaprogramming / Generated Code

- Try `ls */*_def.py */*_gen.py`
- The `def.py` files are abstract definitions. They're not translated by
[mycpp]($xref).
- The `gen.py` files generate source code in Python and C++ from these
definitions.
- For example, we define the core `Id` type and the lexing rules abstractly.
- See [build/dev.sh]($oil-src) and [build/codegen.sh]($oil-src)

## State Machines

- `$IFS` splitting in `osh/split.py`
Expand Down
8 changes: 6 additions & 2 deletions doc/release-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ Note: `oil-native` is a **preview** release, not a working shell.

Oil has three major sources of docs:

- The [Doc Overview](doc/) links to all other docs, including the Manual and
Oil Language Reference. It's work in progress. Up-to-date docs include:
- The [Doc Overview](doc/) links to all other docs, including the Manual
and Oil Language Reference. It's work in progress. Up-to-date docs
include:
- [INSTALL][]
- [OSH User Manual](doc/osh-manual.html)
- [Known Differences Between OSH and Other
Expand All @@ -53,6 +54,9 @@ Oil has three major sources of docs:
- [JSON Support](doc/json.html)
- [Simple Word Evaluation](doc/simple-word-eval.html)
- [Quirks](doc/quirks.html)
- [Notes on OSH Architecture](doc/architecture-notes.html)
- [Online Help](help-index.html) (incomplete). This document underlies the
`help` builtin. It links to sections in the [Help](help.html) page.
- [Github Wiki for oilshell/oil](https://github.com/oilshell/oil/wiki)
- The [Oil Deployments](https://github.com/oilshell/oil/wiki/Oil-Deployments)
wiki page has other ways of getting Oil. These versions may not be
Expand Down

0 comments on commit 99f3349

Please sign in to comment.