Skip to content
/ lab Public
forked from concord-consortium/lab

HTML5-based scientific models, visualizations, graphing, and probeware

lab.dev.concord.org/

License

View license
0 stars 39 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

jmclaus/lab

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2,885 Commits
attic
attic
 
 
config
config
 
 
imports
imports
 
 
node-bin
node-bin
 
 
script
script
 
 
server
server
 
 
src
src
 
 
test
test
 
 
validation/models/md2d/epsilon
validation/models/md2d/epsilon
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
.gitmodules
.gitmodules
 
 
.rvmrc
.rvmrc
 
 
.travis.yml
.travis.yml
 
 
Capfile
Capfile
 
 
Gemfile
Gemfile
 
 
Gemfile.lock
Gemfile.lock
 
 
Guardfile
Guardfile
 
 
Makefile
Makefile
 
 
cloud.thor
cloud.thor
 
 
docs-in-progress.txt
docs-in-progress.txt
 
 
jshint-config.json
jshint-config.json
 
 
lab.sublime-project
lab.sublime-project
 
 
license.md
license.md
 
 
package.json
package.json
 
 
readme.md
readme.md
 
 

Repository files navigation

lab.concord.org

HTML5-based open source scientific models, visualizations, graphing, and probeware from the Concord Consortium.

Production site: : lab.concord.org

Technical Readme: : lab.concord.org/readme.html

Development site: : lab.dev.concord.org

Source code repository: : github.com/concord-consortium/lab

Note: many of the links in this readme only work properly when this document is served from a web-server and do not resolve properly at github.com/concord-consortium/lab. A live version of the readme is always available here: lab.concord.org/readme.html.

Table of Contents

  • toc {:toc}

Licensing

Lab is Copyright 2012 (c) by the Concord Consortium and is distributed under any of the following licenses:

The complete licensing details can be read here.

If you have have received a distribution archive of the Concord Consortium Lab project our copyright applies to all resources except the files in the vendor/ directory. The files in the vendor/ directory are from third-parties and are distributed under either BSD, MIT, or Apache 2.0 licenses.

Examples

Molecular Modeling Examples

Energy2D Examples

Graphing examples

Probeware Examples

Distribution of Project and Examples

Compressed archives of the generated Lab distribution are available for download:

Download and expand one of these archives to create a folder named concord-consortium-lab-xxxxxxx. The seven characters at the end of the archive filename are the first seven characters of the git commit SHA.

Open the file index.html in this folder in your browser to get an offline version of the Lab project.

Features that require integration of the back-end web application such as saving changes in models and interactives and use of Java applets do not work directly from a downloaded distribution.

NOTE: the downloaded distribution of examples may not work properly in Chrome due to a long-standing bug in Chrome: Issue 49001: Regression: cssRules null when stylesheets loaded from local disk. The problem only occurs when loading the web pages directly from your filesystem. If instead you use a local web server on your computer to serve the downloaded distribution Chrome works properly.

Setup Development

Lab uses a number of RubyGems and node modules to manage development. Lab's test framework uses Vows, which depends on nodejs and npm (Node Package Manager). In addition JavaScript minification is done using UglifyJS. JavaScript dependency management is handled by RequireJS.

Prerequisites:

RVM, Ruby 1.9 and the RubyGem bundler

We use RVM to mange our development dependency on Ruby 1.9.3 and the specific Ruby Gems needed for building Lab and running the Lab server.

  1. Install RVM

After installation you should see something like the following:

$  ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin10.8.0]

Once you have a working version of Ruby 1.9.3 check to see if the RubyGem bundler is already installed:

$ gem list bundler

*** LOCAL GEMS ***

bundler (1.1.3)

If Bundler is not installed install it now:

$ gem install bundler
Fetching: bundler-1.1.3.gem (100%)
Successfully installed bundler-1.1.3
1 gem installed

nodejs and npm, the Node Package Manager

nodejs and npm, the Node Package Manager are additional development dependencies.

npm, the Node Package Manager has been included as part of nodejs since version 0.8.7.

Install the latest stable version of node (currently v0.6.18) with installers available here: http://nodejs.org/#download

Currently development is being done with these versions of node and npm:

$ node -v
v0.8.7

$ npm -v
1.1.49

CouchDB

Install the nosql document-oriented CouchDB database server to support persistence for the Lab server.

Installation options:

MacOS X

Most of the developers on Lab use Homebrew a package manager for Mac OS X.

  1. Install Homebrew

  2. Install CouchDB using homebrew

     brew doctor     # fix issues if needed
 brew update     # if you haven't run it in the last 24 hours
 brew install couchdb

Java

Java is needed for compiling the invisible sensor applet as well as the legacy Java applications we are porting to HTML5.

Test to see if Java is installed and available:

$ java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)

On Mac OS X 10.7 and later Java is not automatically installed. However running the command java -version when Java is not installed will bring up an operating system dialog enabling Java to be installed.

Customize Maven mirror

Note: Currently in order to compile OTrunk a very old version of the maven jarjar plugin is needed (v0.4). Only newer versions of the jarjar plugin are available in the public Maven artifact repositories. Until OTrunk is updated you will need to include the Concord nexus Maven mirror (which has v0.4 of the maven jarjar plugin) in ~/.m2/settings.xml.

A minimal working ~/.m2/settings.xml will look like this:

<?xml version="1.0"?>
<settings>
  <mirrors>
    <mirror>
      <id>nexus</id>
      <mirrorOf>*</mirrorOf>
      <url>http://source.concord.org/nexus/content/groups/public</url>
    </mirror>
  </mirrors>
</settings>

###Linux (Ubuntu)

Before any of this make sure that "run command as login shell" is checked. if it isn't go to edit-profile preferences-Title and Command, and check Run command as a login shell.

To Install RVM you need to have curl:

$ sudo apt-get install curl </code></pre>

Install RVM using curl with:

$ curl -L https://get.rvm.io | bash -s stable --ruby

After RVM has finnished installing it will ask you to run a command similar to

$ source /home/user_name/.rvm/scripts/rvm

After installation you should see something like the following:

$  ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

RVM has some additional dependancies, to view these type:

$ rvm requirements

Under additional dependancies and ruby, copy the list of dependancies and paste them in to your terminal and install them using sudo. The command will look something like this:

sudo /usr/bin/apt-get install build-essential openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion</code></pre>

Becouse of some unknown bugs RVM doesn't recognise readline without being explictly pointed to it. To do this I've had to reinstall ruby 1.9.3-p194.

$ rvm reinstall 1.9.3-p194 --with-zlib1g-dev

ruby gem bundler should be installed. To check if it has been try:

$ gem list bundler

*** LOCAL GEMS ***

bundler (1.1.4)

If it doesn't return anything install Bundler:

$ gem install bundler

nodejs and npm and couchdb

nodejs and npm, the Node Package Manager are additional development dependencies.

npm, the Node Package Manager has been included as part of nodejs since version 0.6.3.

The nosql document-oriented CouchDB database server is installed to support persistence for the Lab server.

To install the latest stable versions of node and couchdb you first need to add these PPA repositories:

  1. node PPA repo
  2. couchdb PPA repo

For these to work as intended python software properties must also be installed.

$ sudo apt-get install python-software-properties
$ sudo apt-add-repository ppa:chris-lea/node.js
$ sudo apt-add-repository ppa:longsleep/couchdb
$ sudo apt-get update

Now install node and couchdb:

$ sudo apt-get install couchdb nodejs npm

Currently development is being done with these versions of node and npm:

$ node -v
v0.8.2

$ npm -v
1.1.37

Neither the Java run time environment nor the Java development kit are installed by default, both of which are used for the java projects.

sudo apt-get install  default-jre openjdk-6-jdk

The Ruby Gem Nokogiri requires libxslt and libxml2, install them with:

sudo apt-get install libxslt-dev libxml2-dev

Final note, before you make everything setup the project configurations files by copying the configuration samples:

cp config/config_sample.yml config/config.yml
cp server/config/couchdb.sample.yml server/config/couchdb.yml

Use git to create a local clone of the Lab repository.

If you have commit access to the repository use this form:

git clone [email protected]:concord-consortium/lab.git

Alternatively if you don't have commit access use this form:

git clone git://github.com/concord-consortium/lab.git

Setup the local Lab repository for development

Make sure you have already installed the prerequistes: Ruby 1.9, the RubyGem bundler, and nodejs (which now includes npm the Node Package Manager.

Open a shell and change to the lab/ directory. The first time you cd into the lab/ directory RVM will ask you if this new .rvmrc file should be trusted.

The .rvmrc specifies that this project dependeds on Ruby 1.9.3-p194 and all the required Ruby Gems will be installed in the RVM gemset named ruby-1.9.3-p194@lab.

cd lab
make everything

When make everything is run on a freshly cloned repository it performs the following tasks:

  1. Install the runtime dependencies as git submodules into the vendor/ directory:

    git submodule update --init --recursive

  2. Install the development dependencies that use nodejs and are managed by npm:

    npm install

    You can see the list of dependencies to be installed in the file package.json. In addition vendor/d3 and vendor/science.js are manually installed into node_modules/.

  3. Install the additional RubyGems used for development: haml, sass, guard ...

    bundle install --binstubs

    This creates the bin/ directory and populates it with command-line executables for running the specific versions of the RubyGems installed for development.

  4. Generates the server/public directory:

  5. Generates the Java resources in the server/public/jnlp directory:

You should now be able to open the file: server/public/index.html in a browser and run some of the examples.

Automatic build processing using Guard

Start watching the src/ and test/ directories with Guard and when files are changed automatically generate the JavaScript Lab modules, the examples, and run the tests.

bin/guard

Now any change you make in src/examples/ will generate the corresponding content in server/public/examples/. In addition changes in src/lab/ generate the associated Lab modules in lab/ and copy these modules to server/public/lab/. In addition any change in either the src/lab/ or test/directories will run the tests and display the results in the console window where bin/guard is running.

Setup the Rails Lab server for development

The Lab server is a very simple Rails 3.2 application that uses CouchDB for persistence.

Open a shell and change to the lab/server directory. The first time you cd into the lab/server directory RVM will ask you if this new .rvmrc file should be trusted.

The .rvmrc specifies that this project dependeds on Ruby 1.9.3-p194-server and all the required Ruby Gems will be installed in the RVM gemset named ruby-1.9.3-p194@lab-server.

cd lab/server
bundle install

Create a couchdb configuration by copying the sample:

cp config/couchdb.sample.yml config/couchdb.yml

If you have setup your local CouchDB server to require admin login for creating new databases you will need to enter user and password for a valid admin user.

Starting the Rails Lab server
cd lab/server
thin start

You can now open your local Lab application at this url:

http://localhost:3000/

You can use a pre-configured route to open the local CouchDb admin web interface:

http://localhost:3000/_utils
Entering the Rails Lab server console
cd lab/server
rails console

Building the Classic Java applications and the Sensor Java Applet

The Lab repository can build the legacy Java applications Molecular Workbench and Energy2D we are converting to HTML5.

Building these Java applications allows developers to more easily compare the operation of the HTML5 versions of these applications to the Java versions running in he browser as applets.

In addition we can create the Java resources to run the invisible Java applet needed for communicating with Vernier GoIO probware in the browser however the Java resources for communicating with the Vernier GoIO Probeware need to be digitally signed.

Running the Classic Java Molecular Workbench and Energy2D as Applications

After building the Java resources the Java applications Molecular Workbench and Energy2D can be run as applications from the command line:

  1. Molecular Workbench

     bin/mw

  2. Energy2D

     bin/energy2d
Java Code-Signing Certificate and Keystore

A self-signed Java certificate is included with the Lab repository: config/lab-sample-keystore,jks with a password and private key password of abc123 however for production use you will want to use a keystore with a publically-recognized Java code-siging certificate from a company like Thawte.

To build the Jar resources for the probeware using either the self-signed certificate provided with the Lab repository or one of your own first create the file config/config.yml by copying config/config.sample.yml and editing appropriately.

cp config/config.sample.yml config/config.yml

The :java: section of the config.yml yaml file looks like this:

---
# password and alias for Java siging certificate.
:java:
  :password: abc123
  :alias: lab-sample-keystore
  :keystore_path: config/lab-sample-keystore.jks

If you have a keystore already accessible via an alias replace lab-sample-keystore with the alias for your existing keystore. If your keystore is stored in your home directory in the file .keystore then you do should leave the :keystore_path empty.

:keystore_path:

The self-signed lab-sample-keystore,jks keystore was generated with the Java keytool command as follows:

$ keytool -genkey -keyalg RSA -keystore config/lab-sample-keystore,jks -alias lab-sample-keystore -storepass abc123 -validity 360 -keysize 2048
What is your first and last name?
  [Unknown]:  Stephen Bannasch
What is the name of your organizational unit?
  [Unknown]:  Lab Project
What is the name of your organization?
  [Unknown]:  Concord Consortium
What is the name of your City or Locality?
  [Unknown]:  Concord
What is the name of your State or Province?
  [Unknown]:  Massachusetts
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=Stephen Bannasch, OU=Lab Project, O=Concord Consortium, L=Concord, ST=Massachusetts, C=US correct?
  [no]:  yes
Enter key password for <lab-sample-keystore>
	(RETURN if same as keystore password):

$ keytool -selfcert -alias lab-sample-keystore -keystore config/lab-sample-keystore.jks
Enter keystore password:
Building the Java Resources

Note: Currently in order to compile OTrunk you need to include the Concord nexus Maven mirror in ~/.m2/settings.xml. See Customize Maven mirror

Run make jnlp-all to erase, build, package, sign and deploy all the Java resurces.

The first time this task is run it:

  1. Creates a java/ top-level directory and checks out the required Java projects into this directory.
  2. Builds each of the projects
  3. Copies the jar resources into the server/public/jnlp/ directory packing and signing them as needed.

Later if you have made updates in the Java source code or need to re-build and deploy for any reason you can run:

script/build-and-deploy-jars.rb --maven-update

If one of the maven projects fails to build because a dependency could not be found try running the command again with the --maven-update argument:

script/build-and-deploy-jars.rb --maven-update

Details about each Java project, where the repository is located, what branch is compiled, the specific compilation details are all contained in this Ruby file: config/java-projects.rb

Java build/deploy integration

There is a configuration file expressed in Ruby code here which defines build specifications for each Java project: config/java-projects.rb

The specification indicates whether the Java Jar resource will be directly downloaded or whether the source code will be checked-out and compiled to create the Jar.

Java Projects Build Strategies

The :build_type option is used to specify the Java Projects Build Strategy. Five different kinds of build strategies are available. Each strategy includes additional build information in the :build option.

The :maven, :custom, and :ant build strategies all expect to be able to get the source code to build projects from git repositories.

Most of Concord Consportium's open source Java codebase is published at our public Subversion repository: svn.concord.org/svn/projects/trunk however all of the legacy Java code the Lab project references is also mirrored to separate git repositories at: github.com/concord-consortium/.

The trunk branches in the git mirrors we maintain such as: concord-consortium/otrunk always represent the latest code checked into the Subversion trunk branch.

We have made small changes to the Molecular Workbench codebase and we are maintaining these in the master branch of the git mirror here: concord-consortium/mw.

  1. :maven

    Concord's OTrunk framework uses the :maven build strategy:

     'otrunk' => { :repository => 'git://github.com/concord-consortium/otrunk.git',
               :branch => 'trunk',
               :path => 'org/concord/otrunk',
               :build_type => :maven,
               :build => MAVEN_STD_CLEAN_BUILD,
               :has_applet_class => true,
               :sign => true },

    The trunk branch of the otrunk repo will be checked out into ./java/otrunk and be built using Maven. Because the otrunk jar is used with the sensor-applet code (which uses a native library) it must also be signed.

    Deploying both signed and unsigned jars

    Concord's Molecular Workbench also uses the :maven build strategy however when running it as a Java Web Start jnlp it needs to be signed while normally when run as an applet it should be unsigned.

     'mw'             => { :build_type => :maven,
                       :build => MAVEN_STD_CLEAN_BUILD,
                       :repository => 'git://github.com/concord-consortium/mw.git',
                       :branch => 'master',
                       :path => 'org/concord/modeler',
                       :main_class => "org.concord.modeler.ModelerLauncher",
                       :has_applet_class => true,
                       :sign => true,
                       :also_unsigned => true }

    By setting both the :sign and :also_unsigned options to true two jars will be deployed:

    • jnlp/org/concord/modeler/mw.jar
    • jnlp/org/concord/modeler/unsigned/mw.jar

  2. :ant

  3. :custom

    For Energy2D a :custom build strategy is used and the command line invocation necessary is in the MANUAL_JAR_BUILD constant.

     'energy2d' => { :repository => 'git://github.com/concord-consortium/energy2d.git',
                 :branch => 'trunk',
                 :path => 'org/concord/energy2d',
                 :build_type => :custom,
                 :version => '0.1.0',
                 :build => MANUAL_JAR_BUILD,
                 :has_applet_class => true,
                 :sign => false }

    In this case MANUAL_JAR_BUILD has been defined as:

     MANUAL_JAR_BUILD = "rm -rf bin; mkdir bin; find src -name *.java | xargs javac -target 5 -sourcepath src -d bin"

  4. :copy_jars

  5. :download

    JDom uses the :download build strategy:

    'jdom' => { :build_type => :download, :url => 'http://repo1.maven.org/maven2/jdom/jdom/1.0/jdom-1.0.jar', :path => 'jdom/jdom', :version => '1.0', :sign => true }

The script that runs the checkout-build-pack-sign-deploy can either operate on ALL projects specified or on a smaller number.

Running script/build-and-deploy-jars.rb with no arguments operates on all projects listed in config/java-projects.rb.

Optionally you can specify one or more projects to operate on. This builds just sensor and sensor-applets:

script/build-and-deploy-jars.rb sensor sensor-applets

The Jar resources deployed to the server/public/jnlp directory include a timestamp in the deployed artifact so unless you specifically request an earlier version you will always get the latest version deployed.

JnlpApp Rack Application Service

The Rails server has a Rack application JnlpApp mounted at the route /jnlp for servicing requests for Java jar resources.

The JnlpApp Rack application uses the Rack::Jnlp middleware defined here server/lib/rack/jnlp.rb.

Normally versions for jars can only be specified by using a jnlp form. A jnlp form of specification can be used for webstart and also for applets.

The older form of applet invocation that uses the html element normally can't specify version numbers for jar dependencies, however the Jnlp::Rack application included with Lab does allow version specification.

Example: right now on my local system there are two different versions of the vernier-goio-macosx-x86_64-nar.jar:

$ ls -l server/public/jnlp/org/concord/sensor/vernier/vernier-goio/vernier-goio-macosx-x86_64-nar*
  98396 May 28 01:55 ../org/concord/sensor/vernier/vernier-goio/vernier-goio-macosx-x86_64-nar__V1.5.0-20101012.203835-2.jar
  99103 May 28 16:40 ../org/concord/sensor/vernier/vernier-goio/vernier-goio-macosx-x86_64-nar__V1.5.0-20120528.164030-3.jar

Note the different lengths for the two different versions.

If a request comes in from Java for vernier-goio-macosx-x86_64-nar.jar the most recent version is returned:

$ curl --user-agent java -I http://localhost:3000/jnlp/org/concord/sensor/vernier/vernier-goio/vernier-goio-macosx-x86_64-nar.jar
HTTP/1.1 200 OK
Last-Modified: Mon, 28 May 2012 20:40:34 GMT
Content-Type: application/java-archive
Content-Length: 99103

However the version number can be added as a http query parameter.

$ curl --user-agent java -I http://localhost:3000/jnlp/org/concord/sensor/vernier/vernier-goio/vernier-goio-macosx-x86_64-nar.jar?version-id=1.5.0-20101012.203835-2
HTTP/1.1 200 OK
Last-Modified: Mon, 28 May 2012 05:55:05 GMT
Content-Type: application/java-archive
Content-Length: 98396
x-java-jnlp-version-id: 1.5.0-20101012.203835-2

Note that in the response the x-java-jnlp-version-id HTTP header is added with teh actual version.

This feature of specifying versioned jar resources should NOT be used for production because Java won't properly cache the jar locally. Eveytime a request is made for a jar with a version-id query parameter the complete jar will be downloaded again.

When a version is specified in a jnlp form for an applet the jar WILL be cached properly.

Development Note: If the applets no longer operate properly it may be that the server is no longer operating properly and needs to be restarted. The Java console log for the applet may show requests made for jars that are not fulfilled.

If a request like the following produces an error:

$ curl --user-agent java -I http://localhost:3000/jnlp/org/concord/sensor-native/sensor-native.jar
HTTP/1.1 500 Internal Server Error

Restart the server and the request should now suceed:

$ curl --user-agent java -I http://localhost:3000/jnlp/org/concord/sensor-native/sensor-native.jar
HTTP/1.1 200 OK
Last-Modified: Thu, 07 Jun 2012 16:44:28 GMT
Content-Type: application/java-archive
Content-Length: 34632
content-encoding: pack200-gzip

Project Configuration

Configuration variables used by the runtime JavaScript code are available in the JavaScript global object Lab.config.

In a full build environment the JavaScript configuration is set in the :jsconfg section of config/config.yml:

:jsconfig:
  :sharing: true
  :home: http://lab.concord.org
  :homeInteractivePath: /examples/interactives/interactive.html
  :homeEmbeddablePath: /examples/interactives/embeddable.html
  :utmCampaign: <external-campaign-key>

sharing A boolean attribute used to determine if the Share link in the Interactives will be enabled. The default value for this is true.

home Url used to reference cannonical site when sharing is turned off.

homeInteractivePath Path to page to run non-embeddable version of Interactives.

homeEmbeddablePath Path to page to run embeddable version of Interactives.

utmCampaign If present a UTM suffix is added to links in the About box. Set to a string which identifies the external organization.

When the build environment is active these values are used to generate JavaScript code integrated into the project by the Ruby program: script/generate-js-config.rb

Interactive Share link

Normally the Share link in an Interactive is enabled. The Share dialog allows a user to more easily share the Interactive in an email or IM, and also provides generated HTML content that can be copied and pasted to embed the Interactive into a blog or web page.

If you are hosting this content on an external server where supporting sharing is impractical in some manner you can disable the display of the Interactive Share link by setting :sharing in config/config.yml to false:

:jsconfig:
  :sharing: false
  :home: http://lab.concord.org
  :homeInteractivePath: /examples/interactives/interactive.html
  :homeEmbeddablePath: /examples/interactives/embeddable.html
  :utmCampaign: <external-campaign-key>

The additional values for :home, homeInteractivePath, and homeEmbeddablePath are used to construct an additional paragraph in the Interactive About box providing a link to the Interactive on the production site for the project.

The value for utmCampaign is optional. If present and the home site has enabled Google Analytics setting a value for utmCampaign will allow better tracking of users who click through links in the Interactive About box.

Google Analytics

In addition there is a optional section in config/config.yml which if present enables embedding google analytics script into the head of the main index.html and all html pages in the examples/ and doc/ directories. This includes all the Interactives which are located in examples/interactives directory.

Include your Google Analytics account number here to enable insertion of the Google Analytics script by the build system into the generated HTML pages.

:google_analytics:
  :account_id: <account-id>

The content from which the embedded Google Analytics script is generated is contained in this Ruby file: script/setup.rb.

Limitations changing configuration in an archive distribution

If you have downloaded a distribution archive you can manually modify the code that initializes the JavaScript runtime configuration in the files: lab/lab.js and lab/lab.min.js. Editing the value for Lab.config.sharing in these files will affect the JavaScript runtime settings when these files are loaded.

Additionally you can turn on UTM suffixes by adding a string value to `Lab.config.utmCampaign``.

However generation and insertion of the Google Analytics script into HTML pages can only be done by setting a value for the :google_analytics :account_id and running the build process.

Contributing to Lab

If you think you'd like to contribute to Lab as an external developer:

  1. Create a local clone from the repository located here: http://github.com/concord-consortium/lab. This will by default have the git-remote name: origin.

  2. Make a fork of http://github.com/concord-consortium/lab to your account on github.

  3. Make a new git-remote referencing your fork. I recommend making the remote name your github user name. For example my username is stepheneb so I would add a remote to my fork like this:

     git remote add stepheneb [email protected]:stepheneb/lab.git

  4. Create your changes on a topic branch. Please include tests if you can. When your commits are ready push your topic branch to your fork and send a pull request.

Continuous Integration on travis-ci

travis-ci is a free open-source web-based distributed continuous integration system.

When code is checked in to the master branch the concord-consortium/lab project on travis-ci automatically runs all the unit tests.

If any test fails the author of the commit will get an email as well the developers listed in the .travis.yml configuration file.

Setting up travis-ci integration

I created an account on travis-ci linked to the stepheneb acocunt on github by having travis-ci authenticate me with github using oauth.

I was then able to manually setup a Travis service hook for the lab repository. I needed to do this manually because I was integrating a repository under the concord-consortium github organization instead of one directly under my own account.

Useful travis-ci resources

Measuring Performance

The Complex Atoms Model includes several features for estimating and measuring performance of the molecular modeler.

  1. End of Stats section displays average number of model steps/s.

  2. Run Benchmarks button stops model and measures time for running the model 100 steps and also time for running the model and rendering the graphics for 100 steps. When measuring the speed of the model and graphics together the test is run continuously and control is not returned to the browser for repainting the screen.

  3. A separate lab.performance repository with a Ruby script performance.rb available which uses Selenium Webdriver to automate running the Run Benchmarks test 10 times and collects , averages, and reports the results.

Installing and using performance.rb

We use a completely separate repository for the performance monitoring tools so the same performance mesuring scripts can be used to measure performance over a range of commits to the Lab project. This makes it easier to monitor performance and investigate performance regressions.

In your working copy of the Lab project:

git checkout git://github.com/concord-consortium/lab.performance.git

Edit paths in the file lab.performance/performance.rb to reference locations for FireFox and Chrome on your computer as well as the variable URL_TO_COMPLEX_MODEL for running the Complex Atoms Model locally.

Gem prequisites: selenium-webdriver

gem install selenium-webdriver

Measuring performance:

$ ./lab.performance/performance.rb

browser: Chrome: 19.0.1085.0, Intel Mac OS X 10_6_8
Date: 2012-03-30 08:44
Molecule number: 50
Temperature: 5

git commit
commit 2c3a9328a43964485ed5f661cfb6e6cc6850ce95
Author: Stephen Bannasch <[email protected]>
Date:   Fri Mar 30 08:44:05 2012 -0400

    whitespace fixups
true

average steps                  167.30
average steps w/graphics       101.63

Repository structure

Summary of Directories in src/

src/lab

The src/lab directory includes JavaScript source code for the Lab JavaScript modules. During the build process individual files are copied into modules which are placed in the server/public/lab directory.

src/examples/, src/doc/, and src/experiments/

The src/examples/, src/doc/, and src/experiments/ directories contain additional resources for generating the html, css, and image resources for the matching target folders in server/public.

Note: remember to make changes you want saved in the src/examples/, src/doc/, and src/experiments/ and not in the target directories of the same names in server/public. Change made in server/public will be overwritten during the next build process.

src/vendor

Third-party JavaScript runtime dependencies for Lab are located in the src/vendor directory and are installed as git submodules the first time make is run in a new checkout of the source code repository.

Only the necessary JavaScript library files and other resources needed for runtime operation along with the associated README and LICENSE files are copied to server/public/vendor during the build process.

All of the files copied to server/public/vendor are licensed and distributed under one or more of the following licenses: Simplified BSD, The BSD 3-Clause, MIT, or Apache 2.0.

src/resources

The src/resources directory contains image resources and are copied directly to server/public/resources.

src/sass

The src/sass directory contains Sass templates and the Bourbon Sass library are are used during the build process to generate CSS resources.

src/mw-helpers

The src/mw-helpers directory contains JavaScript modules only used as part of the testing and build process and are not copied to server/public/resources.

src/jnlp

The src/jnlp directory currently contains the Vernier GoIO Java sensor Jars. While these files are needed to use Vernier GoIO sensor in the browser normally there are no compiled artifacts included in either the Lab repository or referenced and used by the Lab either as external git submodules or as separately downloaded Java projects. The presence of these specific files is temporary until I can recreate or redevelop the specifc combination of Java, Vernier GoIO native libraries and JNI code generated by SWIG used 18 months ago to create these resources at Concord Consortium.

Lab Modules: src/lab

The source code for the Lab modules is all contained in src/lab:

The list above presents independent modules built as separate sub-libraries (md2d, energy2d, grapher) and one directory which contains common utilities and modules (common).

Lab Modeling Engines: md2d and energy2d

2D Molecular Dynamics: md2d

md2d module contains a basic Next Gen Molecular Workbench application. It built over MVC design pattern and consist of following units:

The source code of the core molecular dynamics engine is currently located in the [src/lab/md2d/models/engine](https://github.com/concord- consortium/lab/tree/master/src/lab/md2d/models/engine) directory, which is organized as a set of related RequireJS modules. These modules are compatible both with the Web browser environment and Node. When used in Node, amdefine package must be available (it is listed as an dependency in the Lab [packages.json](https://github.com/concord- consortium/lab/blob/master/package.json) file). The entry point for external Node.js applications is the file [src/lab/md2d/models/engine/md2d.js](https://github.com/concord- consortium/lab/blob/master/src/lab/md2d/models/engine/md2d.js).

In addition, Node-based executables can be written and placed in [src/lab/md2d/models/engine](https://github.com/concord- consortium/lab/tree/master/src/lab/md2d/models/engine) or a subdirectory. These are expected to be useful for verifying and tuning the model by running the model headless and saving summary results into a file for offline analysis; see, e.g., https://github.com/rklancer/script-md.

Hashbang scripts for starting these executables (i.e., files which start with the line #!/usr/bin/env node and which have the execute bit set) should be placed in the directory node-bin, and should execute by require()ing the appropriate module and calling its entry point method. Lab's packages.json file specifies node-bin as the location of the executable scripts which npm should make available whenever Lab is imported into another project as a Node module. (For developer convenience, bin/ is being reserved for Ruby executables made available via Bundler.)

2D Thermal Energy: energy2d

energy2d module contains a basic Energy2D application. It is a direct port of Java Energy2D. Energy2D is also built over MVC design pattern and consist of following units:

and additionally:

GPU Toolkit is a small set of utilities which wraps basic WebGL structures and objects, providing higher level API. It is useful, as Energy2D uses WebGL for General-Purpose Computing on Graphics Processing Unit. So, a lot of physics calculations are performed on the GPU if user's Web browser supports WebGL technology.

The source code of the core physics engine is located in the src/lab/energy2d/models directory. Especially important units are listed below:

Necessary GLSL (GL Shading Language) sources are stored in separate files. They are loaded using RequireJS text plug-in which just allows to load plain text files. Finally, they are inlined in the resulting library due to the RequireJS optimization process.

JavaScript Dependency Management and Build Process - RequreJS

Lab's modules use RequireJS for dependency management. It is a JavaScript file and module loader optimized for in-browser use, but it can be used in other JavaScript environments, like Rhino and Node. So, you don't have to worry about manual JavaScript files concatenation and the library build process - everything is done automatically.

RequireJS might be used for fully asynchronous in-browser module loading, but it can also be used for combining all source files into one output JavaScript file. The Lab project mostly uses the second approach. The tool which resolves all dependencies and combines them into single output file is called RequireJS Optimizer.

Useful RequireJS resources:

Adding new source file intended to work in the Web browser

Adding a new source to existing module is straightforward.

  1. Put the source file in an appropriate directory (e.g. src/lab/module-name/).

  2. Define it as a module using RequireJS syntax, e.g. following this pattern:

     define(function (require) {
   // Dependencies.
   var utils = require('other-module-name/file-name');
   // Preparation code.
   // (...)
   // Finally, return module API.
   return {
     // (...)
   };
   // Or just constructor function:
   // return function ClassName() {
   //   (...)
   // };
 });

    You can read more about RequireJS modules syntax here.

  3. In case of need, reference this file in other sources using RequireJS syntax:

       var dependency = require('module-name/file-name');

That's all! Your file will be automatically included during module build process.

Adding new source file intended to work in the Web browser and in Node

If you are working on file which should also work in the Node environment as a typical package (without using RequireJS as a dependency and its syntax), follow instructions above and additionally:

  1. Add following snippet at the beginning of the source file:

     if (typeof define !== 'function') { var define = require('amdefine')(module) }

  2. Make sure that amdefine package is listed as a dependency in your package.json file (it can be Lab's packages.json file or separate one if you work on the independent module, like those stored in src/modules directory).

The Lab's array module uses this technique, so you may look for a reference in src/modules/arrays.

Adding new module which should be built as a separate library

This involves a few additional steps comparing with adding a single source file.

  1. Create a new directory in src/lab.

  2. Put all related sources there or in src/lab/common (if you think that they are generic enough and may be reused by other modules).

  3. Define module-name.build.js and public-api.js files in your new directory (described below).

  4. Add build routines to the Makefile:

    4.1. Define a new target, for example:

     server/public/lab/lab.module-name.js: \
     $(NEW_MODULE_SRC_FILES) \
     $(COMMON_SRC_FILES)
     $(R_OPTIMIZER) -o src/lab/module-name/module-name.build.js

    4.2. List this target in LAB_JS_FILES Makefile variable containing the list of all Lab JavaScript modules to be generated.

Your module will be built during Lab's build process. You may use one of the existing modules for reference in case of any troubles (md2d, energy2d or grapher).

Module Build Configuration - *.build.js file

Each Lab's module contains file name.build.js. It is a RequireJS Optimizer build profile. Useful, related resources:

If you create new build file, make sure that you use one of the existing build profiles as a reference! It will enforce consistent style and options across all Lab's modules.

Lab's build profiles use almond module - a replacement AMD loader for RequireJS. It is a smaller "shim" loader, providing the minimal AMD API footprint that includes loader plugin support.

Why? almond allows us to create the resulting library which is totally independent from RequireJS. It is a reasonable approach as RequireJS is used only for module definition, dependency resolving and building a single file library using Optimizer. The asynchronous module loading is not utilized by the final Lab library, so there is no need to force users to load whole RequireJS library. Instead, use and include minimalistic RequireJS API replacement.

Module Public API - public-api.js file

If module exposes API using global variables, it should define it in public-api.js file. It is a typical RequireJS module, which just adds properties to window object. You can look at src/md2d/public-api.js, src/energy2d/public-api.js or src/grapher/public-api.js file for a reference.

This files are not necessary, but highly recommended if module has to define some global variables. It is a convention used internally by Lab repository. Such files are enforcing clean definition of public API exposed by modules. Developers will have certainty that all global variables are defined there and only there.

Execution of this script should be enforced by build profile (*.build.js files described above). Most often is done by wrap option:

  // Protect global namespace and call Public API export.
  wrap: {
    start: "(function() {",
    // Almond by default simulates async call of require (sets timeout).
    // Last argument (true) forces sync call instead.
    end: "require(['module-name/public-api'], undefined, undefined, true); }());"
  }

CoffeeScript Files Support

The Lab project is configured to easily support CoffeeScript sources. RequireJS plugin called require-cs is used for dynamic loading of CoffeeScript sources.

To enable CoffeeScript support, make sure that module's build profile (see section about *.build.js files) contains following options:

// Additional modules.
paths: {
'cs' :'../vendor/require-cs/cs',
'coffee-script': '../vendor/coffee-script/extras/coffee-script'
},
//Stub out the cs module after a build since
//it will not be needed.
stubModules: ['cs'],
// The optimization will load CoffeeScript to convert
// the CoffeeScript files to plain JS. Use the exclude
// directive so that the coffee-script module is not included
// in the built file.
exclude: ['coffee-script']

md2d module has CoffeeScript support enabled, so you can use its build profile as a reference.

  • To define a CoffeeScript module just use typical RequireJS syntax converted to CoffeeScript:

      define (require) ->
    # Dependencies.
    CoffeeScriptDependency = require 'cs!other-module-name/file-name'
    JavaScriptDependency   = require 'module-name/file-name'

    class SomeClass extends BaseClass
    // (...)

  • You can also load CoffeeScript in JavaScript files:

      define(function (require) {
    // Dependencies.
    var CoffeeScriptDependency = require('cs!module-name/file-name');
    // (...)

Just remember about cs! prefix in paths when loading CoffeeScript sources. RequireJS Optimizer will convert such files to plain JavaScript and include them in the final library.

Generated Examples: server/public/examples/

The server/public/examples/ directory is automatically generated running make and is not part of the repository.

When bin/guard is running any changes to files in the src/examples/ directory cause automatic rebuilding of the associated files in the server/public/examples/ directory.

HTML and CSS Generation

Haml is used to generate most of the HTML in the server/public/ directory.

kramdown is used to generate readme.html in server/public/ from Mardown markup.

Sass is used to generate the CSS assets. The Sass markup may be in the form of *.sass or *.scss files

The Bourbon library of Sass mixins is included.

Testing: test/

Lab's JavaScript tests use Vows, an asynchronous behavior driven framework based on Node.js. In addition Lab uses jsdom, a lightweight CommonJS implementation of the W3C DOM specifications. Lab's test setup was inspired by that used by d3.js. The development dependencies for running the tests are installed using npm.

Running the tests:

$ make test
................................. . . .. . . .
x OK > 40 honored (0.012s)

If you are running bin/guard the tests run automatically anytime a change is made in the JavaScript files in the src/ or test/ directory.

The results of the tests are displayed in the console that bin/guard is running in.

If the bottom of the console window is viewable you will see new test results whenever you save a changes.

Recent versions of nodejs/v8 support TypedArrays -- this make it possible to more extensively test lab.arrays which is designed to support using either typed or regular arrays for computation.

test/env.js uses the node module jsdom to setup resources for simple emulation of a browser.

Vows integrates the standard nodejs assertions with an additional collection of useful assertions summarized below:

  • numerical

      assert.greater (3, 2);
  assert.lesser (2, 3);
  assert.inDelta (Math.random(), 0, 1);

  • equality

      assert.equal          (4, 4);
  assert.strictEqual    (4 > 2, true);
  assert.notEqual       (4, 2);
  assert.strictNotEqual (1, true);
  assert.deepEqual      ([4, 2], [4, 2]);
  assert.notDeepEqual   ([4, 2], [2, 4]);

  • type

      assert.isFunction (function () {});
  assert.isObject   ({goo:true});
  assert.isString   ('goo');
  assert.isArray    ([4, 2]);
  assert.isNumber   (42);
  assert.isBoolean  (true);
  assert.typeOf     (42, 'number');
  assert.instanceOf ([], Array);

  • truth

      assert.isTrue  (true);
  assert.isFalse (false);

  • null, undefined, NaN

      assert.isNull      (null);
  assert.isNotNull   (undefined);
  assert.isUndefined ('goo'[9]);
  assert.isNaN       (0/0);

  • inclusion

      assert.include ([4, 2, 0], 2);
  assert.include ({goo:true}, 'goo');
  assert.include ('goo', 'o');

  • regexp matching

      assert.match ('hello', /^[a-z]+/);

  • length

      assert.length ([4, 2, 0], 3);
  assert.length ('goo', 3);  *** not working ***

  • emptiness

      assert.isEmpty ([]);
  assert.isEmpty ({});
  assert.isEmpty ("");

  • exceptions

      assert.throws(function () { x + x }, ReferenceError);
  assert.doesNotThrow(function () { 1 + 1 }, Error);

Additionally test/env-assert.js has a number of useful additional assertions copied from d3.js.

Note: Using a more specific assertion usually results in more useful error reports.

There are also many interesting test examples and patterns in the d3.js test directory that can be adapted for use in Lab.

A Simple Example of Test Driven Development

Here's a simple example that is part of the tests for lab.arrays.js to test the arrays.max() function:

"find max in array with negative and positive numbers": function(max) {
  assert.equal(max([3, -1, 0, 1, 2, 3]), 3);
},

The 'model stepping' tests are a good example where the tests help helped drive new features. The basic features I was testing in this section relate to the existing functionality exposed by the Stop, Start, Go, and Reset buttons as wells as the extended keyboard controls that allow stepping forward and backwards a step at a time.

First I created this test that passed:

"after running running one tick the model is at step 1": function(model) {
  model.tick();
  assert.equal(model.stepCounter(), 1);
  assert.isTrue(model.isNewStep());
},

In thinking about driving out changes to KE, PE and Temperature of the molecular model itself I realized I'd like the capability to run a specific number of steps forward and then check the results.

I then wrote this test that failed -- because the model.tick() function didn't yet take an optional argument to run multiple steps forward:

"after running 9 more ticks the model is at step 10": function(model) {
  model.tick(9);
  assert.equal(model.stepCounter(), 10);
  assert.isTrue(model.isNewStep());
},

After saving the change I saw the new test failure reported in my console. I then implemented the new feature in the actual src/lab/molecules.js. Less than a second after saving the file the tests completed and the report showed it passing.

This is a very simple example -- but part of the value of this kind of test driven development is in first thinking of how something should behave rather than in how to get it to actually do the work.

Since I already had this function for running one model step:

model.tick()

Adding an optional numeric argument for running more steps is a fine way to express the intent of the new feature:

model.tick(9)

In more complicated coding thinking about how to express the intent clearly and then what the result should be if that intent is successful FIRST ... and then 'driving out' the actual implementation to achieve that result can result in a better architecture -- and of course you also end up with tests.

Because the tests run SO quickly I can interactively change the code in the module or the test and immediately see results.

Debugging Tests using the node debugger

Sometimes it can be helpful to break into a debugger when there is a problem in either the code or the test setup itself. Node comes with a debugger which can be used in combination with vows and the tests.

First set a breakpoint by inserting the statement: debugger;

suite.addBatch({
  "Thermometer": {
    topic: function() {
      debugger;
      return new components.Thermometer("#thermometer");
    },
    "creates thermometer": function(t) {
      assert.equal(t.max, 0.7)
    }
  }
});

Start the node debugger and pass in the full command line to run the tests:

node debug ./node_modules/vows/bin/vows --no-color

The debugger will break at the beginning of vows:

< debugger listening on port 5858
connecting... ok
break in node_modules/vows/bin/vows:3
  1
  2
  3 var path   = require('path'),
  4     fs     = require('fs'),
  5     util   = require('util'),

Enter cont to continue execution until your breakpoint.

debug> cont
< ·
< ········
< ·
<
break in test/lab/components/components-test.js:13
 11   "Thermometer": {
 12     topic: function() {
 13       debugger;
 14       return new components.Thermometer("#thermometer");
 15     },

To evaluate expressions type repl -- use ctrl-C to exit the repl:

repl
Press Ctrl + C to leave debug repl
> initialization_options
{ model_listener: false,
  lennard_jones_forces: true,
  coulomb_forces: true }
> atoms[0].charge
-1

Enter ctrl-C to exit the repl and return to the debugger.

Enter ctrl-D to exit the debugger.

node-inspector npm package for node-inspector

Physical constants and units

The core of the molecular dynamics engine performs computations using dimensioned quantities; we do not nondimensionalize to reduced units. The units used internally are:

  • femtoseconds
  • nanometers
  • Dalton (atomic mass units)
  • elementary charges
  • Kelvin

(Note that we will shortly switch to representing time in picoseconds rather than femtoseconds.)

The above implies that the 'natural' unit of energy within the application is the "Dalton nm^2 / fs^2", and the natural unit of force is the "Dalton nm / fs^2". We call these "MW Energy Units" and "MW Force Units" respectively; however, externally-facing methods accept and report energies in electron volts, rather than "MW Units".

The molecular dynamics engine in src/md-engine contains a submodule, defined in src/md- engine/constants/ which defines physical useful constants and allows one to perform some unit conversions in a mnemonic way.

Once you have require()d the constants module appropriately, you can access the constants, 2 converter methods, and an object that defines the various units. For the following, assume the constants module has been require()d into the variable constants.

Units

The various units are available as properties of the object constants.unit, and are named appropriately. The units themselves are objects, but their properties are not external API; rather, the unit objects are expected to be passed as arguments to conversion methods which return numeric values. Units are named in the singular and are written as all-uppercase (they are constants).

Some example units are:

  • constants.unit.JOULE constants.unit.MW_ENERGY_UNIT` (Dalton nm^2 / fs^2, see above)
  • constants.unit.METERS_PER_FARAD

Physical Constants

The various constants are defined as properties of the constants object. However, these do not have numerical values; instead, they each contain a single method, as, which accepts a unit (see above) and returns the numerical value of that constant in terms of that unit. This is intended to be a convenience for the programmer and to reduce the likelihood that he or she will forget to keep track of the units in which a value is stored.

For example,

  • constants.BOLTZMANN_CONSTANT.as(constants.unit.JOULES_PER_KELVIN) (== 1.380658e-23)
  • constants.PERMITTIVITY_OF_FREE_SPACE.as(constants.unit.FARADS_PER_METER) (== 8.854187e-12)

Unit conversions

The constants module does not attempt to do dimensional analysis (for example, converting kg m/s^2 into Newtons). However, it can convert a value between two different units that measure the same type of quantity, and it can supply conversion ratios that make it easier to do dimensional analysis carefully in your own code.

Converting a value between two unit types:

To convert the value 1kg into Daltons (aka atomic mass units), use the convert method:

constants.convert(1, { from: constants.unit.KILOGRAM, to: constants.unit.DALTON }) (== 6.022137376997844e+26)

Finding the ratio between two unit types and rolling your own unit conversions:

To find the number of atomic masses in 1 kg, use the ratio method with the per argument:

constants.ratio(constants.unit.DALTON, { per: constants.unit.KILOGRAM }) (== 6.022137376997844e+26)

This form of ratio is especially useful for unit conversion, and the "per" is intended as a mnemonic. The number reported above, for example, is easily understood to be "6.022 x 10^26 Dalton per kilogram" and can therefore be used as a factor that "cancels" kilograms and replaces them with Daltons in a compound unit such as "kg m/s".

However, sometimes you want the value of a Dalton expressed as kilograms. Although you could reverse the units in the above function call, or divide 1 by the result above, it is better to use the mnemonic as form of ratio:

constants.ratio(constants.unit.DALTON, { as: constants.unit.KILOGRAM }) (== 1.66054e-27)

Deployment

Deploying static content to a Github gh-pages branch

Github's github:pages feature supports sharing any content in a gh-pages repository branch as static web content.

The gh-pages branch of the Lab repository is used to store the static pages and client-side code built by the Makefile at the directory server/public.

concord-consortium.github.com/lab

In addition the content of the gh-pages branch is used to create the downloadable archive distributions of Lab

The contents of the gh-pages branch are automatically made available in a standard web-page form (as opposed to the standard Github page for showing a repository) at this url:

concord-consortium.github.com/lab

when you push to the gh-pages branch.

If you maintain a fork of this project on Github, you get a Github Page for free, and the instructions below apply to you as well!

Making the server/public/ folder track the gh-pages branch

If you haven't done this yet, make the server/public folder track the contents of the gh-pages branch.

If you have a Guard process running make sure and stop it before continuing!

# server/public/ needs to be empty for git clone to be happy:
rm -rf server/public

# substitute the URL for whatever fork of the Lab repository you have write access to:
git clone [email protected]:concord-consortium/lab.git -b gh-pages server/public

Note that make clean now empties the server/public folder in-place, leaving the Git server/public/.git and server/public/jnlp directories intact.

Pushing changes to gh-pages branch

First, make sure your server/public folder tracks the gh-pages branch, as per the above.

Then run the following shell command in the script/ folder:

script/gh-pages

This script will first make sure there is nothing that isn't committed. If there are unstaged or staged and uncommitted files the gh-pages script will halt.

Test and commit (or save the changes to a topic branch) and if your testing show the bugs are fixed or the new features or examples are stable then push these changes to the master branch and try running the gh-pages script again:

git push origin master
script/gh-pages

Deploying to a remote server

The Lab project has a suite of scripts for creating and deploying to EC2 servers running on Amazon Web services (AWS).

The scripts for creating, stopping and re-creating a server require an AWS account. These scrpts use the Ruby proge

The Capistrano scripts for updating a deployment to an existing server only require that you have a copy of the pem file associated

To deploy to AWS you will need an AWS account and the ability to create and modify EC2 instances.

If you are create a new server from scratch you will also need access to the AWS DNS service Route53.

Note: currently there are a few steps/resources that only work properly if you have a concord.org AWS account.

AWS Setup/Prerequisites

  1. AWS account. For deployment to the lab servers managed by CC you will need an AWS account managed by concord.org.

  2. Create your AWS Access Key ID and AWS Secret Access Key. To do this, go to the IAM Dashboard in AWS (Services -> IAM), click 'Users', click the checkbox by your username, and select User Actions -> Manage Access Keys

  3. Copy your AWS Access Key ID and AWS Secret Access Key to the following yaml configuration file .fog in your home directory:

    :default:
  :aws_access_key_id: YOUR_AWS_ACCESS_KEY_ID
  :aws_secret_access_key: YOUR_AWS_SECRET_ACCESS_KEY

  4. Place a copy of a the appropriate AWS PEM file on your local files system. For deployment to the lab servers managed by CC use the lab-dev pem. This can be found in CC's Google Docs/Drive. This file should have its permissions set to 600, and the folder it is in (e.g. ~/.ec2) should be 700.

  5. Create or identify an appropriate AWS security group. For deployment to the lab servers managed by CC the lab.dev security group is used.

  6. Edit the :deploy section of config/config.yml using values in config/config_sample.yml as a guide.

    Here's an example from the :deploy section of a working config.yml:

    :deploy:
  :targets:
  - :name: lab-dev
    :url: lab.dev.concord.org
    :branch: master
  :group_name: lab.dev
  :zone_domain: concord.org.
  :zone_records_name: dev.concord.org
  :pem:
    :name: lab-dev
    :path: ~/.ec2

    There is one deploy target named lab-dev associated with a server running at lab.dev.concord.org. Deployments to lab-dev use the master branch of the repository. The lab.dev security group is used when new servers are created or existing sever are re-created.

    When a whole new server is created the DNS entry is created in the concord.org. zone domain and when searching for an existing DNS entry for a deploy-target the zone record name dev.concord.org is used.

    Besides the AWS Access Key ID and AWS Secret Access Key security credentials copyied locally to to the file ~/.fog the lab-dev.pem file saved in the directory: ~/.ec2 is also used when communicating with AWS.

  7. List the deploy targets described in config/config.yml with the task: thor cloud:list_targets to confirm the configuration is valid:

    $ thor cloud:list_targets

  Deploy Targets
  name                    url                           branch
------------------------------------------------------------------------------------------
  lab-dev                 lab.dev.concord.org           master

  8. Generate specific Capistrano deploy tasks and littlechef nodes using deploy-targets specified in config/config.yml. Run this thor task whenever you change the :deploy section in config/config.yml to generate the Ruby Capastrano configuration files in config/deployment/<deploy-target>.rb and the littlechef JSON configurations in config/littlechef/nodes/<deploy-target>.json

    $ thor cloud:setup

  9. List the running AWS server instances to confirm that your local AWS security credentials are setup correctly:

    $ thor cloud:list

  target              hostname                      state         ipaddress           ec2-id          ec2-dns
------------------------------------------------------------------------------------------------------------------------------------------------------
  lab-dev             lab.dev.concord.org.          running       107.22.184.173      i-f844ec81      ec2-107-22-184-173.compute-1.amazonaws.com

  10. If you are working with an existing host that has already been setup such as lab.dev.concord.org generate the proper ssh configuration and add the remote host key to ~/.ssh/known_hosts. This adds a local ubuntu user in ~/ssh/config and connects to the remote host to add the key.

    Example of setting up SSH configuration with the existing remote AWS host: lab..dev.concord.org:

    $ thor cloud:setup_ssh lab.dev.concord.org

Using Capstrano to deploy new code to an existing server

After testing, committing, and pushing code to a public repository use the Capistrano tasks to update a remote server.

The capistrano commands take the form:

cap <deploy-target> task

The basic command to update a server:

cap <deploy-target> deploy:update

Here are the list of current Capistrano deploy commands:

$ cap -T deploy:
cap deploy:restart          # restart rails app
cap deploy:clean_and_update # clean and update server
cap deploy:setup            # setup server
cap deploy:status           # display last commit on deployed server
cap deploy:update           # update server
cap deploy:update_jnlps     # update public/jnlp dir on server

Update the lab.dev.concord.org server with the latest code committed on the master branch on concord-consortium/lab:

cap lab-dev deploy:update

When you have made changes in the repository like adding or updating a git submodule in src/vendor then you will need instead run the deploy:clean_and_update task:

cap lab-dev deploy:clean_and_update

Updating the Java jar resources on a remote rerver

The Java resources require much less frequent updates since the main body of work for Lab is occuriring in the HTML5 development.

The capistrano task: deploy:update_jnlps erases the server/public/jnlp/ directory on the remote server and re-generates and deploy the packed signed jars from source or from downloads:

$ cap <deploy-target> deploy:update_jnlps

The resulting directory on the server will look something like this:

$ tree /var/www/app/server/public/jnlp/
server/public/jnlp/
├── jdom
│   └── jdom
│       ├── jdom__V1.0.jar
│       └── jdom__V1.0.jar.pack.gz
├── jug
│   └── jug
│       ├── jug__V1.1.2.jar
│       └── jug__V1.1.2.jar.pack.gz
└── org
    └── concord
        ├── data
        │   ├── data__V0.2.0-20120531.005123-1.jar
        │   └── data__V0.2.0-20120531.005123-1.jar.pack.gz
        ├── energy2d
        │   ├── energy2d__V0.1.0-20120531.005123-1.jar
        │   └── energy2d__V0.1.0-20120531.005123-1.jar.pack.gz
        ├── framework
        │   ├── framework__V0.1.0-20120531.005123-1.jar
        │   └── framework__V0.1.0-20120531.005123-1.jar.pack.gz
        ├── frameworkview
        │   ├── frameworkview__V0.1.0-20120531.005123-1.jar
        │   └── frameworkview__V0.1.0-20120531.005123-1.jar.pack.gz
        ├── modeler
        │   ├── mw__V2.1.0-20120531.005123-1.jar
        │   └── mw__V2.1.0-20120531.005123-1.jar.pack.gz
        ├── otrunk
        │   ├── otrunk__V0.3.0-20120531.005123-1.jar
        │   └── otrunk__V0.3.0-20120531.005123-1.jar.pack.gz
        ├── sensor
        │   ├── sensor-applets
        │   │   ├── sensor-applets__V0.1.0-20120531.005123-1.jar
        │   │   └── sensor-applets__V0.1.0-20120531.005123-1.jar.pack.gz
        │   ├── sensor__V0.2.0-20120531.005123-1.jar
        │   ├── sensor__V0.2.0-20120531.005123-1.jar.pack.gz
        │   └── vernier
        │       └── vernier-goio
        │           ├── vernier-goio-macosx-i386-nar__V1.5.0-20101012.203834-2.jar
        │           ├── vernier-goio-macosx-ppc-nar__V1.5.0-20101012.203834-2.jar
        │           ├── vernier-goio-macosx-x86_64-nar__V1.5.0-20101012.203835-2.jar
        │           └── vernier-goio-win32-nar__V1.4.0.jar
        └── sensor-native
            ├── sensor-native__V0.1.0-20120531.005123-1.jar
            └── sensor-native__V0.1.0-20120531.005123-1.jar.pack.gz

Managing AWS servers with thor tasks

There are a set of thor tasks for managing, creating, and re-creating AWS servers for Lab:

$ thor -T
cloud
-----
thor cloud:create hostname                       # create a new server instance using this hostname
thor cloud:delete hostname                       # delete an existing server instance running at this hostname
thor cloud:find_dns_record hostname              # find dns record for hostname
thor cloud:list                                  # list existing servers
thor cloud:list_targets                          # list existing deploy targets
thor cloud:recreate hostname                     # recreate a new server instance for this hostname by destroying and rebuilding an existing server
thor cloud:setup                                 # setup capistrano deploy tasks and littlechef nodes using targets in config/config.yml
thor cloud:setup_ssh hostname                    # setup ssh configuration for communication to hostname
thor cloud:start ec2_id                          # start a stopped existing server instance using the ec2-id
thor cloud:stop reference                        # stop a running existing server instance at this hostname or ec2-id
thor cloud:update reference                      # update server <ec2_id|hostname> provisioning with littlechef 'lab-server' role
thor cloud:update_dns_record hostname ipaddress  # updating IP address for DNS record hostname to ipaddress

Creating a new AWS Lab Server

Creating a new Lab server on AWS consists of three steps:

  1. Creating a new hostname, server, and provisioning the server with thor:

     $ thor cloud:create <hostname>

    This task will create a new hostname as a DNS A record if the hostname does not already exists.

    If the hostname already exists as a CNAME first login to the AWS:Route53 service and delete the existing host name.

    If the new DNS entry for hostname is not properly propogated when the hostname is created or changed you will get an error that looks something like this:

     *** running local command: echo '
 Host <hostname>
   User ubuntu
   IdentityFile ~/.ec2/lab-dev.pem
 ' >> ~/.ssh/config

 *** running local command: ssh-keygen -R <hostname>
 /Users/stephen/.ssh/known_hosts updated.
 Original contents retained as /Users/stephen/.ssh/known_hosts.old

 *** running local command: ssh ubuntu@<hostname> -o StrictHostKeyChecking=no exit
 ssh: Could not resolve hostname <hostname>: nodename nor servname provided, or not known

 *** updating littlechef node: <hostname>.json

 *** provisioning <hostname> with littlechef role: lab-server
     <ec2-id>, <ec2-hostname>, <new-ip-address>
     command: cd /Users/stephen/dev/concord/lab/config/littlechef && fix node:<hostname> role:lab-server


 == Applying role 'lab-server' to <hostname> ==

 Fatal error: Name lookup failed for <hostname>

 Underlying exception:
     nodename nor servname provided, or not known

    You will need to resolve the issue with getting the correct DNS record for hostname before continuing. After this is resolved you can follow these steps to continue the initial setup and provisioning:

     $ ssh-keygen -R <hostname>
 $ ssh ubuntu@<hostname> -o StrictHostKeyChecking=no exit
 $ thor cloud:update  <hostname>

  2. Optionally configure the server to use a valid Java code-siging certificate.

    If you wish to support the integration of the optional Java resources that are required to be signed to work:

    • legacy Molecular Worbench and Energy2D Java Web Start applications
    • Java-based Vernier GoIO browser-sensor applet integration

    You should put copy of a valid Java siging certificate keystore on hostname and edit config/config.yml to reference this keystore before running cap <deploy-target> deploy:setup

    The one supplied with the repository is a sample self-signed certificate and end user will be warned that it is not valid.

    Here is one way to acomplish this:

     $ scp <path-to-keystore> deploy@<hostname>:/var/www/app/config/<new-keystore-name>.jks

    Now ssh to the new host and edit the java section of /var/www/app/config/config.yml to update the values for :password, :alias, and :keystore_path.

  3. Finishing the setup of the server with a capistrano task

     $ cap <deploy-target> deploy:setup

    This completes the initial deploy and builds of all the project resources to the server.

References

Molecular Simulation

Courses

Reduced Units

JavaScript Library Runtime Dependencies

D3

D3: manipulating and visualizing documents based on data.

JQuery

JQuery: simplifies HTML document traversing, event handling, animating, and Ajax interactions.

JQuery-UI

JQuery-UI: abstractions for low-level interaction and animation, advanced effects and high-level, themeable widgets, built on top of the jQuery

Sizzle

sizzle: CSS selector engine designed to be easily dropped in to a host library.

Codemirror

Codemirror2: in-browser code editor.

science.js

science.js: scientific and statistical computing methods.

dsp.js

dsp.js: digital signal processing methods including functions for signal analysis and generation, Oscillators(sine, saw, square, triangle), Window functions (Hann, Hamming, etc), Envelopes(ADSR), IIR Filters(lowpass, highpass, bandpass, notch), FFT and DFT transforms, Delays, Reverb.

Modernizr

modernizr: detect HTML5 and CSS3 features in browsers.

Lab Example: index.html.haml uses Modernizer to check if the browser implents SVG and re-direct the user to an upgrade page if the feature is not presnet.

MathJax

MathJax is a display engine for mathematics that works in all modern browsers.

Lab Example: lennard-jones-potential.html.haml uses MathJax to display LaTeX formatted math equations.

OpenSans Font

OpenSans Font: used for most text display

Development Dependencies

node

node-inspector

npm

npm, the Node Package Manager isnow bundled with Node and is used to specify and manage external node pacage dependencies for a project.

More about using npm for development:

Lab Example: package.json specifies node pakage dependencies for the Lab project.

RequireJS

RequireJS is a JavaScript file and module loader. It is optimized for in-browser use, but it can be used in other JavaScript environments, like Rhino and Node.

Lab Example:

All Lab's modules use RequireJS for dependency management and build process. Its application is widely described in this section.

CoffeeScript

CoffeeScript is a language the compiles to JavaScript. Many programmers find it more expressive and productive. js2cofee can be used to convert JavaScript to CoffeeScript. RequireJS Optimizer also can convert CoffeeScript to JavaScrit. So you don't have to manually do it when referencing CoffeeScript files using RequireJS.

Lab Examples:

  1. get-md2d-data.coffee is a coffeescript program used to run the MD2D engine from the command line and generate data used for physics validation tests.
  2. PlaybackComponentSVG is an object written in CoffeeScript that creates and manages the SVG-based Playback control widget for the Molecule Container.
  3. InteractivesController is an object written in plain JavaScript which references directly CoffeeScript file using RequireJS (Thermometer = require('cs!common/components/thermometer')).

This section covers RequireJS support of CoffeeScript files.

RubyGems

Bundler

Bundler is a Ruby Gem used to express and manage Ruby Gem dependencies.

Lab Example: Gemfile is used to specify all the Ruby Gem dependencies to build and test the Lab project.

Haml

Haml is a Ruby Gem that processes HTML expressed in HAML markup into HTML.

Lab Example: index.html.haml is used to generate the main index.html page.

Sass

Sass is a Ruby Gem that provides many powerful extensions to CSS3 and works by processing files in either SASS-indented-syntax or SCSS format (a su[erset of standard CSS3) and generating CSS stylesheets.

Lab Examples:

  1. index.sass is used to generate: index.css
  2. readme.scss is used to generate: readme.css

Guard

Guard is a Ruby Gem that can efficiently watch for changes on the file system and automatically start the build process when needed.

Lab Example: Starting Guard with bin/guard loads and runs the configuration in Guardfile.

Thor

thor is a Ruby Gem for building self-documenting command line utilities.

Lab Example: cloud.thor are the Ruby command-line interface scripts for providing access to the AwsLabServer library for creating and managing AWS cloud servers.

Fog

fog is a Ruby Gem for working with many different cloud service providers.

Lab Example: AwsLabServer is a library built on top of fog for creating and managing Lab server instances on AWS.

Additional Testing Dependencies

Vows

Vows is an asynchronous behaviour driven testing framework for Node.

Lab Examples:

  1. axis-test.js is used to test the drag UI logic for interactively re-scaling Graph axes.
  2. arrays-test.js is used to test the utility class for working with regular or Typed Arrays.

jsdom

Miscellaneous

livereload

livereload is project that has created extensions for Chrome FireFox, and Safari to provide automatic browser reloading when the HTML, CSS and JavaScript files are changed on the server. The older version 1 extensions work with the guard-livereload gem.

Full Screen API

About

HTML5-based scientific models, visualizations, graphing, and probeware

lab.dev.concord.org/

Resources

Readme

License

View license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

7 tags

Packages

No packages published