Import docs into repo and use ReadTheDocs

CHANGELOG.md

@@ -2,7 +2,9 @@
 
 ## [Unreleased] \(0.6.0\) - Black Diamonds [v6]
 
-  -
+  - 
+
+  - Included documentation into the repository and enabled http://somns.readthedocs.io/ [\#212](https://github.com/smarr/SOMns/pull/212)
 
 ## [0.5.0] - More Newspeak [v5]
 

README.md

@@ -40,15 +40,15 @@ the specifications are as follows:
  - simultaneous slots clauses are not fully supported (spec. 6.3.2)
 
  - as in SOM, blocks can only have 3 arguments (counting `self`)
- 
+
  - object literals currently require a keyword prefix `objL`, to work around
    parser limitations
 
 Obtaining and Running SOMns
 ---------------------------
 
-This is a brief guide, a more comprehensive overview is available here:
-[Getting Started Guide](http://som-st.github.io/somns/getting-started/).
+This is a brief guide, a more comprehensive overview for users or developers is
+available in the `docs` folder and on [ReadTheDocs](http://somns.readthedocs.io/en/dev/).
 
 To checkout the code:
 
@@ -71,7 +71,7 @@ Setup Development Environment with Eclipse and VS Code
 1. Install JDK 1.8 and Eclipse Mars (or later)
 
 2. Download the project from Github
-   `git clone https://github.com/smarr/SOMns`
+   `git clone https://github.com/smarr/SOMns.git`
 
 3. Run `ant compile` on the command line, or via Eclipse, to make sure that all
    libraries are loaded and available.
@@ -132,6 +132,9 @@ Academic Work
 SOMns is designed as platform for research on concurrent programming models,
 and their interactions. Here, we collect related papers:
 
+ - [A Concurrency-Agnostic Protocol for Multi-Paradigm Concurrent Debugging Tools](http://stefan-marr.de/papers/dls-marr-et-al-concurrency-agnostic-protocol-for-debugging/),
+   S. Marr, C. Torres Lopez, D. Aumayr, E. Gonzalez Boix, H. Mössenböck; Dynamic Language Symposium'17.
+
  - [Kómpos: A Platform for Debugging Complex Concurrent Applications](http://stefan-marr.de/downloads/progdemo-marr-et-al-kompos-a-platform-for-debugging-complex-concurrent-applications.pdf),
    S. Marr, C. Torres Lopez, D. Aumayr, E. Gonzalez Boix, H. Mössenböck; Demonstration at the <Programming>'17 conference.
 

docs/AUTHORS.md

@@ -0,0 +1 @@
+../AUTHORS

docs/CHANGELOG.md

@@ -0,0 +1 @@
+../CHANGELOG.md

docs/LICENSE.md

@@ -0,0 +1 @@
+../LICENSE

docs/README.md

@@ -0,0 +1 @@
+../README.md

docs/basic-setup.md

@@ -0,0 +1,36 @@
+# Basic User Setup
+
+A brief overview for a basic development setup for SOMns.
+
+## Minimal Software Requirements
+
+SOMns is implemented in Java 8, uses Ant as a build system, git as
+source control system, and Python for a launcher script.
+
+On Ubuntu, the following instructions will install the necessary dependencies:
+
+```bash
+sudo add-apt-repository ppa:webupd8team/java
+sudo apt install oracle-java8-installer git ant
+```
+
+## Getting the Code and Running Hello World
+
+To checkout the code:
+
+```bash
+git clone https://github.com/smarr/SOMns.git
+```
+
+Then, SOMns can be build with Ant:
+
+```bash
+cd SOMns
+ant compile  ## will also download dependencies
+```
+
+Afterwards, the simple Hello World program is executed with:
+
+```bash
+./som -G core-lib/Hello.ns
+```

docs/conf.py

@@ -0,0 +1,8 @@
+from recommonmark.parser import CommonMarkParser
+
+source_parsers = {
+    '.md': CommonMarkParser,
+}
+
+source_suffix = ['.md']
+html_theme = 'sphinx_rtd_theme'

docs/csp-design.md

@@ -0,0 +1,51 @@
+# Communicating Sequential Processes for Newspeak/SOMns
+
+> These notes were originally written as blog post and published on
+> [stefan-marr.de](http://stefan-marr.de/2017/01/communicating-sequential-processes-for-newspeak-somns/).
+
+One possible way for modeling concurrent systems is Tony Hoare's classic approach of having isolated processes communicate via channels, which is called [Communicating Sequential Processes (CSP)](https://en.wikipedia.org/wiki/Communicating_sequential_processes). Today, we see the approach used for instance in [Go](https://blog.golang.org/share-memory-by-communicating) and [Clojure](http://clojure.com/blog/2013/06/28/clojure-core-async-channels.html).
+
+While [Newspeak's specification](http://bracha.org/newspeak-spec.pdf) and implementation come with support for Actors, I want to experiment also with other abstractions, and CSP happens to be an interesting one, since it models systems with blocking synchronization, also known as channels with rendezvous semantics. I am not saying CSP is better in any specific case than actors. Instead, I want to find out where CSP's abstractions provide a tangible benefit.
+
+But, the reason for this post is another one. One of my biggest quibbles with most CSP implementations is that they don't take isolation serious. Usually, they provide merely lightweight concurrency and channels, but they rarely ensure that different processes don't share any mutable memory. So, the door for low-level race conditions is wide open. The standard argument of language or library implementers is that guaranteeing isolation is not worth the performance overhead that comes with it. For me, concurrency is hard enough, so, I prefer to have the guarantee of proper isolation. Of course, another part of the argument is that you might need shared memory for some problems, but, I think we got [a more disciplined approach for those problems](http://stefan-marr.de/2016/02/domains-sharing-state-in-the-communicating-event-loop-actor-model/), too.
+
+## Isolated Processes in Newspeak
+
+Ok, so how can we realize isolated processes in Newspeak? As it turns out, it is pretty simple. Newspeak already got the notion of _values_. Values are deeply immutable objects. This means values can only contain values themselves, which as a consequence means, if you receive some value from a concurrent entity, you are guaranteed that the state never changes.
+
+In [SOMns](https://github.com/smarr/SOMns), you can use the [`Value`](https://github.com/smarr/SOMns/blob/master/core-lib/Kernel.som#L82) mixin to mark a class as having value semantics. This means that none of the fields of the object are allowed to be mutable, and that we need to check that fields are only initialized with values in the object's constructor. Since Newspeak uses nested classes pretty much everywhere, we also need to check that the outer scope of a value class does not have any mutable state. Once that is verified, an object can be a proper deeply immutable value, and can be shared without introducing any data races between concurrent entities.
+
+Using this as a foundation, we can require that all classes that represent CSP processes are values. This gives us the guarantee that a process does not have access to any shared mutable state by itself. Note, this is only about the class side. The object side can actually be a normal object an have mutable state, which means, within a process, we can have normal mutable state/objects.
+
+Using the _value_ notion of Newspeak feels like a very natural solution to me. Alternative approaches could use a magic operator that cuts off lexical scope. This is something that I have seen for instance in AmbientTalk with its [isolates](https://soft.vub.ac.be/amop/at/tutorial/actors#isolates). While this magic `isolate` keyword gives some extra flexibility, it is also a new concept. Having to ensure that a process' class is a value requires that its outer lexical scope is a value, and thus, restricts a bit how we structure our modules, but, it doesn't require any new concepts. One other drawback is here that it is often not clear that the lexical scope is a value, but I think that's something where an IDE should help and provide the necessary insights.
+
+In code, this looks then a bit like this:
+
+```java
+class ExampleModule = Value ()(
+  class DoneProcess new: channelOut = Process (
+  | private channelOut = channelOut. |
+  )(
+    public run = ( channelOut write: #done )
+  )
+
+  public start = (
+    processes spawn: DoneProcess
+               with: {Channel new out}
+  )
+)
+```
+
+So, we got a class `DoneProcess`, which has a `run` method that defines what the process does. Our `processes` module allows us to spawn the process with arguments, which is in this case the output end of a channel.
+
+## Channels
+
+The other aspect we need to think about is how can we design channels so that they preserve isolation. As a first step, I'll only allow to send values on the channel. This ensures isolation and is enabled by a simple and efficient check for whether the provided object is a value.
+
+However, this approach is also very restrictive. Because of the deeply immutable semantics of values, they are quite inflexible in my experience.
+
+When thinking of what it means to be a value, imagine a bunch of random objects: they all can point to values, but values can never point back to any mutable object. That's a very nice property from the concurrency perspective, but in practice this means that I often feel the need to represent data twice. Once as mutable, for instance for constructing complex data structures, and a second time as values so that I can send data to another process.
+
+A possible solution might be objects with copy-on-transfer semantics, or actual ownership transfer. This could be modeled either with a new type of transfer objects, or a copying channel. Perhaps there are other options out there. But for the moment, I am already happy with seeing that we can have proper CSP semantics by merely checking that a process is constructed from values only and that channels only pass on values.
+
+Since the [implementation is mostly a sketch](https://github.com/smarr/SOMns/pull/84), there are of course more things that need to be done. For instance, it doesn't yet support any nondeterminism, which requires an `alt` or `select` operation on channels.

docs/dev-setup.md

@@ -0,0 +1,91 @@
+# A Complete Development Setup
+
+To setup a complete development environment, we need also the Graal just-in-time
+compiler, and Node.js to work on the Kompos Web Debugger.
+
+On Ubuntu, the necessary software can be installed with:
+
+```bash
+## First, installing Node.js and the Node Package Manager (NPM)
+curl -sL https://deb.nodesource.com/setup_9.x | sudo -E bash -
+sudo apt install npm nodejs
+
+## Second, installing and compiling the Graal JIT Compiler
+cd .. ## leaving the SOMns folder
+git clone --recursive https://github.com/smarr/GraalBasic.git
+cd GraalBasic
+yes "n" | ./build.sh
+cd ../SOMns
+ant ## to ensure everything is compiled
+```
+
+At this point, it should be possible to use Graal to run SOMns by dropping the
+`-G` option from the command line:
+
+```bash
+./som core-lib/Hello.ns
+```
+
+<figure style="float: right; margin-right: -100px">
+<img style="width: 200px;" src="../eclipse-project-outline.png" alt="Eclipse Project outline" />
+<figcaption>
+Eclipse Project outline.
+</figcaption>
+</figure>
+
+## Eclipse to Develop the Interpreter
+
+SOMns is currently developed with Eclipse. While other Java IDEs can also be
+used, for brevity, we'll focus on Eclipse only.
+
+At the time of writing, I am using [Eclipse Neon 4.6](https://eclipse.org/downloads/).
+Please also install the **Eclipse Checkstyle Plugin** from the [Eclipse Marketplace](http://eclipse-cs.sourceforge.net/#!/install).
+
+For development, we also need to setup all Eclipse projects:
+
+```bash
+ant ideinit
+```
+
+After the Eclipse projects for the Truffle library were generated by this step,
+we can import the *existing* projects into Eclipse with *File* -> *Import...* ->
+*Existing Projects into Workspace* and pointing Eclipse to the SOMns folder.
+
+I like to import the Truffle projects into a separate Truffle working set. This
+makes working in Eclipse with many projects easier.
+
+The result should be looking roughly, but not exactly like in the screenshot on
+the right.
+
+For debugging the interpreter with Eclipse, create a Eclipse run configuration
+for the SOMns project. The main class is `som.VM`. To run for instance  the
+Mandelbrot benchmark, add the following as program arguments:
+
+    core-lib/Benchmarks/Harness.ns Mandelbrot 2 0 500
+
+In VM arguments, enable assertions with:
+
+    -ea -esa
+
+I personally start the various SOMns programs from the command line:
+
+```bash
+./som -d -G core-lib/Benchmarks/Harness.ns Mandelbrot 2 0 500
+```
+
+For this approach, we need a *Remote Java Application* debug configuration
+in Eclipse. After starting SOMns, it should tell you that it is waiting on port
+`8000`, which is used as the port in the Eclipse debug configuration.
+
+## Summary
+
+A brief list of steps:
+
+1. Install software dependencies: ant, git, Java 8, Eclipse 4.6 (or later),
+   VS Code 1.8 (or later), Node.js, NPM, Graal JIT compiler
+
+2. Create Truffle Eclipse projects: `ant ideinit`
+
+3. Import Eclipse projects
+
+4. Run `ant` from the command line or Eclipse

docs/index.md

@@ -0,0 +1,8 @@
+# SOMns: A Simple Newspeak Implementation focusing on Concurrency Features
+
+This documentation is split in multiple parts.
+Please see the navigation on the left to find links to:
+
+ - documentation for users
+ - documentation for developers
+ - an overview of SOMns' design

docs/infrastructure.md

@@ -0,0 +1,83 @@
+# Infrastructure
+
+This gives a brief overview of some of the infrastructure used in SOMns.
+
+## Build System
+
+SOMns uses Ant as build system. The setup tries to minimize
+the external software dependencies. Currently, instead of using some automatic
+dependency management system for SOMns, we use an *uberjar* that combines all
+rarely changing Java dependencies.
+
+The corresponding project is [SOMns-deps](https://github.com/smarr/SOMns-deps),
+which is essentially a shell script creating a *jar* file from a set of
+libraries an then uploading it onto [Bintray](https://bintray.com/smarr/SOM).
+
+The Truffle library is however directly used as a
+[git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) dependency,
+because it changes frequently, and we sometimes need changes in Truffle.
+Currently, SOMns also relies on a personal fork of Truffle to support changes
+in the instrumentation and debugging support.
+
+## GitHub
+
+SOMns relies on GitHub, its issue tracking, and pull request system for
+development.
+
+**Change Tracking with Pull Requests:**  The general approach is all changes are
+tracked with a pull request.
+
+When you are getting started with working on the SOMns interpreter internals,
+consider checking out the
+[**Good First Issue**](https://github.com/smarr/SOMns/labels/good%20first%20issue)
+label. These issues are more or less simple changes that with a bit of guidance
+should provide a good introduction to the SOMns code base, an basic
+understanding of how Truffle-based interpreters work, and a few SOMns specific
+insights.
+
+## Code Style
+
+When working on SOMns code, please look at the code around you and stick to the
+style. It might be *particular*, but it is consistent in this code base.
+
+To ensure basic compliance with the style, we **use
+[checkstyle](http://checkstyle.sourceforge.net/)**. It is integrated into the
+build system and continuous integration system. Please use something like
+[Eclipse Checkstyle](http://eclipse-cs.sourceforge.net/) to integrate it in
+your editor.
+
+We are also using Codacy to monitor additional style issues or potential bugs.
+See the [STM pull request](https://github.com/smarr/SOMns/pull/81#pullrequestreview-17422244) for examples.
+
+## Development Support
+
+**Continuous Integration:** To automatically run unit tests for the interpreter, SOMns, and the debugger,
+we use [Travis CI](https://travis-ci.org/smarr/SOMns/builds) (see `.travis.yml`)
+as well as a private GitLab instance to run benchmarks (see `.gitlab-ci.yml`).
+
+In case you forked SOMns on GitHub for your own work, please consider [enabling Travis CI](https://docs.travis-ci.com/user/getting-started/).
+
+The current build status is: [![Build Status](https://travis-ci.org/smarr/SOMns.png?branch=master)](https://travis-ci.org/smarr/SOMns)
+
+**Performance Tracking:**
+Since one goal of SOMns is to be a platform for research on concurrency with
+performance close to state-of-the-art JVMs, we continuously track benchmark
+performance, for startup as well as peak performance with
+[Codespeed](http://somns-speed.stefan-marr.de/timeline/#/?exe=14,23,19,21,22,20,16,17,15,18&base=none&ben=peak.Havlak&env=1&revs=10&equid=off).
+It is run on every change to the master branch, and can be used to track and
+compare performance of experimental changes as well.
+
+<figure style="text-align:center">
+<img style="width:400px" src="../codespeed.png" alt="SOMns Codespeed: Havlak performance" />
+<figcaption>
+SOMns Codespeed, tracking benchmark performance. Example shows *Havlak* peak
+performance.
+</figcaption>
+</figure>
+
+The benchmark execution is configured with `codespeed.conf` and are executed
+with the [ReBench](https://github.com/smarr/ReBench) tool.
+
+**SOMns Code Coverage:**
+To track SOMns code coverage, we use
+[Coveralls](https://coveralls.io/github/smarr/SOMns).

docs/kompos-connection.md

@@ -0,0 +1,39 @@
+# Interaction with Kompos Debugger
+
+This section gives a brief overview of the interaction between SOMns and Kompos.
+
+## Communication with Kompos
+
+Communication between Kompos and SOMns is implemented with WebSockets, the code can be found in [tools.debugger.FrontendConnector](https://github.com/MetaConc/SOMns/blob/16af2c3e3a41d13ab8c10ec289181db9284b49aa/src/tools/debugger/FrontendConnector.java) and [vm-connection.ts](https://github.com/MetaConc/SOMns/blob/16af2c3e3a41d13ab8c10ec289181db9284b49aa/tools/kompos/src/vm-connection.ts). Data is transferred on two separate sockets, one is for JSON and the other one is for binary data.
+
+JSON is used for two-way communication in a message-oriented style. Message objects are [serialized to JSON](https://github.com/MetaConc/SOMns/blob/16af2c3e3a41d13ab8c10ec289181db9284b49aa/src/tools/debugger/FrontendConnector.java#L181) when sending, and [received JSON data](https://github.com/smarr/SOMns/blob/16075443872e8de63e2bc71817552731f85eb1f0/src/tools/debugger/WebSocketHandler.java#L40) is used to create message objects.  Kompos also (de)serializes communication data, [messages.ts](https://github.com/MetaConc/SOMns/blob/16af2c3e3a41d13ab8c10ec289181db9284b49aa/tools/kompos/src/messages.ts) provides some interfaces that help accessing data of message objects.
+
+On the SOMns side, message classes need to be "registered" in [tools.debugger.WebDebugger.createJsonProcessor()](https://github.com/MetaConc/SOMns/blob/16af2c3e3a41d13ab8c10ec289181db9284b49aa/src/tools/debugger/WebDebugger.java#L211), they extend either IncomingMessage or OutgoingMessage. When an IncomingMessage is received, its [process method](https://github.com/MetaConc/SOMns/blob/16af2c3e3a41d13ab8c10ec289181db9284b49aa/src/tools/debugger/message/Message.java#L9) is called.
+
+The binary socket is used for directly sending tracing data, which is pushed to
+Kompos whenever available. Kompos parses the tracing data according to the
+trace format and uses the data to generate the actor graph.
+
+## Trace Format
+
+The trace data includes for instance the following events:
+
+- Actor creation
+- Promise creation
+- Promise resolution: when a Promise is resolved with a value.
+- Promise chained: when a Promise is resolved with another Promise.
+- and various others
+
+An academic write up of the tracing feature and its format is part of our paper
+titled [A Concurrency-Agnostic Protocol for Multi-Paradigm Concurrent Debugging Tools](http://stefan-marr.de/papers/dls-marr-et-al-concurrency-agnostic-protocol-for-debugging/).
+
+## Startup Protocol
+
+The following diagram gives an overview of the startup protocol. For simplicity the binary WebSocket for trace data and the view object are not included.
+
+<figure style="text-align:center">
+<img style="width:600px" src="../kompos-startup.svg" alt="kompos startup protocol" />
+<figcaption>
+Simplified overview of startup protocol between SOMns interpreter and Kompos debugger.
+</figcaption>
+</figure>