OSF

• master Build Status:
• develop Build Status:
• Public Repo: https://github.com/CenterForOpenScience/osf.io/
• Issues: https://github.com/CenterForOpenScience/osf.io/issues?state=open
• COS Development Docs: http://cosdev.readthedocs.org/

Table of contents

• Help
• Running the OSF
• [Running the API Server] (#running-the-api-server)
• Installation
• Common Development Tasks

Help

The COS Development Docs provide detailed information about all aspects of OSF development. This includes detailed installation instructions, a list of common setup errors, and other troubleshooting.

The OSF invoke script provides several useful commands. For more information, run:

invoke --list

Running the OSF

If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local test server with the following sequence:

invoke mongo -d  # Runs mongod as a daemon
invoke mailserver
invoke rabbitmq
invoke celery_worker
invoke elasticsearch
invoke assets -dw
invoke server

Note that some or all of these commands will run attached to a console, and therefore the commands may need to be run in separate terminals. Be sure to activate your virtual environment each time a new terminal is opened. It is normal for the command to keep running!

Once started, you will be able to view the OSF in a web browser- by default, at http://127.0.0.1:5000/

In order to log in on your local server, you will also need to run the authentication server.

• For daily use, run fakeCAS. See CenterForOpenScience/fakeCAS for information on how to set up this service.
• For developing authentication-related features, run CAS. See CenterForOpenScience/docker-library/cas for information on how to set up this service.

Running the API Server

If you have already installed all of the required services and Python packages, and activated your virtual environment, then you can start a working local API server with the sequence delineated under [running the OSF] (#running-the-osf) and:

invoke apiserver

Browse to localhost:8000/v2/ in your browser to go to the root of the browsable API. If the page looks strange, run python manage.py collectstatic to ensure that CSS files are deposited in the correct location.

Livereload support

You can run the OSF server in livereload mode with:

$ invoke server --live

This will make your browser automatically refresh whenever a code change is made.

Optional extras

Some functionality depends on additional services that will not be started using the sequence above. For many development tasks, it is sufficient to run the OSF without these services, except as noted below. Some additional installation will be needed to use these features (where noted), in which case updates will also need to be installed separately.

Authentication

An authentication server (either CAS or FakeCAS) must be available in order to log in to the OSF while running locally. This must be installed separately from the OSF. See running the OSF for details.

Waterbutler

Waterbutler is used for file storage features. Upload and download features will be disabled if Waterbutler is not installed. Consult the Waterbutler repository and documentation for information on how to set up and run this service.

Modular File Renderer

The Modular File Renderer (MFR) is used to render uploaded files to HTML via an iFrame so that they can be viewed directly on the OSF. Files will not be rendered if the MFR is not running. Consult the MFR [repository] (https://github.com/CenterForOpenScience/modular-file-renderer) for information on how to install and run the MFR.

Celery Beat

Normally you don't need to run celery_beat. If you work on tasks that are dispatched by celery_beat:

invoke celery_beat

Some beat-dispatched tasks require metrics and release requirements. If needed:

invoke requirements --metrics

Sharejs

ShareJS is used for collaborative editing features, such as the OSF wiki. It will be installed by the OSF installer script, but must be run separately. To run a local ShareJS server:

$ invoke sharejs

Downloading citation styles

To download citation styles, run:

$ invoke update_citation_styles

Installation

These instructions assume a working knowledge of package managers and the command line. For a detailed step-by-step walkthrough suitable for new programmers, consult the COS Development Docs. See optional extras for information about services not included in the automated install process below.

Pre-requisites

Before attempting to run OSF setup commands, be sure that your system meets the following minimum requirements.

Mac OS

The following packages must be installed before running the automatic setup script:

• XCode command line tools (xcode-select --install)
• Homebrew package manager (run brew update and brew upgrade --all before OSF install)
• Java (if not installed yet, run brew install Caskroom/cask/java)
• Python 2.7
  • pip
  • virtualenv (pip install virtualenv)

El Capitan and newer

If you are using Mac OS X >= 10.11 (El Capitan), you will also need to install OpenSSL headers and set some configuration:

brew install openssl
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography

Quickstart

Mac OS X

These instructions should work on Mac OSX >= 10.7

• Clone the OSF repository to your computer. Change to that folder before running the commands below.
• Create and activate your virtualenv.

virtualenv env
source env/bin/activate

• Copy cp website/settings/local-dist.py to website/settings/local.py. NOTE: This is your local settings file, which overrides the settings in website/settings/defaults.py. It will not be added to source control, so change it as you wish.
• Copy cp api/base/settings/local-dist.py to api/base/settings/local.py. NOTE: This is your local settings file, which overrides the settings in website/settings/defaults.py. It will not be added to source control, so change it as you wish.

$ cp website/settings/local-dist.py website/settings/local.py
$ cp api/base/settings/local-dist.py api/base/settings/local.py

• On MacOSX with homebrew, there is a script that should automate much of the install process:

$ pip install invoke==0.13.0
$ invoke setup

To verify that your installation works, follow the instructions to start the OSF and run unit tests.

Additional configuration for Mac OS X

After running the automatic installer, you may find that some actions- such as running unit tests- fail due to an error with Mongo/ TokuMX. This can be resolved by increasing the system limits on number of open files and processes.

Add the following lines to /etc/launchctl.conf and/or /etc/launchd.conf (creating the files if necessary):

limit maxfiles 16384 16384
limit maxproc 2048 2048

Then create or edit either ~/.bash_profile or /etc/profile to include the following:

ulimit -n 2048

Then reboot.

Additional things to install

The automated installer does not install CAS, Waterbutler, or MFR, which may be needed to run some OSF features locally. Consult the optional extras section for more details.

Manual installation

At present, there is no complete automated install process for other platforms. Although the process above should perform most setup steps on Mac OS, users of other platforms will need to perform the steps below manually in a manner appropriate to their system. Some steps of the installer script can be re-used, in which case the appropriate commands are noted below.

On Mac OS, we recommend using Homebrew to install external dependencies.

• Create local.py files for addons that need them (invoke copy_settings --addons)

• Install TokuMX

• Install libxml2 and libxslt (required for installing python lxml)

• Install Java (if not already installed)

• Install elasticsearch

• Install python requirements (invoke requirements --dev --addons)

• Install npm

• Install node and bower packages

• Build assets (invoke assets --dev)

• If invoke setup hangs when 'Generating GnuPG key' (especially under linux), you may need to install some additional software to make this work. For apt-getters this looks like:

sudo apt-get install rng-tools

Followed by editing /etc/default/rng-tools to add the line:

HRNGDEVICE=/dev/urandom

And finally starting the rng-tools daemon with:

sudo /etc/init.d/rng-tools start

source: http://www.howtoforge.com/helping-the-random-number-generator-to-gain-enough-entropy-with-rng-tools-debian-lenny

Detailed installation and setup guides

Although some effort is made to provide automatic installation scripts, the following more detailed guides may be helpful if you are setting up the OSF on a machine already used for other development work, or if you wish to perform other advanced tasks. If the OSF is already working based on the instructions above, you can skip this section.

Using TokUMX

TokuMX is an open-source fork of MongoDB that provides support for transactions in single-sharded environments. TokuMX supports all MongoDB features as of version 2.4 and adds beginTransaction, rollbackTransaction, and commitTransaction commands.

Installing with Mac OS

$ brew tap tokutek/tokumx
$ brew install tokumx-bin

Installing on Ubuntu

$ apt-key adv --keyserver keyserver.ubuntu.com --recv-key 505A7412
$ echo "deb [arch=amd64] http://s3.amazonaws.com/tokumx-debs $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/tokumx.list
$ apt-get update
$ apt-get install tokumx

Installing on REHL/CentOS

$ sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm

Migrating from MongoDB

TokuMX and MongoDB use different binary formats. To migrate data from MongoDB to TokuMX:

• Back up the MongoDB data
  • invoke mongodump --path dump
• Shut down the MongoDB server
• Uninstall MongoDB
• Install TokuMX (see instructions above)
• Restore the data to TokuMX
  • invoke mongorestore --path dump/osf20130903 --drop
• Verify that the migrated data are available in TokuMX

Using Celery

Installing Celery + RabbitMQ

• Install RabbitMQ. On MacOSX with homebrew,

$ brew update
$ brew install rabbitmq

The scripts are installed to /usr/local/sbin, so you may need to add PATH=$PATH:/usr/local/sbin to your .bash_profile.

For instructions for other OS's, see the official docs.

Then start the RabbitMQ server with

$ invoke rabbitmq

If you want the rabbitmq server to start every time you start your computer (MacOSX), run

$ ln -sfv /usr/local/opt/rabbitmq/*.plist ~/Library/LaunchAgents
$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rabbitmq.plist

Starting A Celery Worker

invoke celery_worker

Search

Install Elasticsearch to use search features.

Mac OSX

$ brew install elasticsearch

note: Oracle JDK 7 must be installed for elasticsearch to run

Ubuntu

$ wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.deb
$ sudo dpkg -i elasticsearch-1.2.1.deb

REHL/CentOS

$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.2.1.noarch.rpm
$ sudo rpm -ivh elasticsearch-1.2.1.noarch.rpm

Using Elasticsearch

• In your website/settings/local.py file, set SEARCH_ENGINE to 'elastic'.

SEARCH_ENGINE = 'elastic'

• Start the Elasticsearch server and migrate the models.

$ invoke elasticsearch
$ invoke migrate_search

Starting a local Elasticsearch server

$ invoke elasticsearch

NPM

The Node Package Manager (NPM) is required for installing a number of node-based packages.

# For MacOSX
$ brew update && brew install node

Installing Node on Ubuntu is slightly more complicated. Node is installed as nodejs, but Bower expects the binary to be called node. Symlink nodejs to node to fix, then verify that node is properly aliased:

# For Ubuntu
$ sudo apt-get install nodejs
$ sudo ln -s /usr/bin/nodejs /usr/bin/node
$ node --version      # v0.10.25

Common Development Tasks

Running the shell

To open the interactive Python shell, run:

$ invoke shell

Running Tests

To run all tests:

$ invoke test --all

To run a certain test method

$ nosetests tests/test_module.py:TestClass.test_method

Run OSF Python tests only:

$ inv test_osf

Run addons Python tests only:

$ inv test_addons

Run Javascript tests:

$ inv karma

By default, inv karma will start a Karma process which will re-run your tests every time a JS file is changed. To do a single run of the JS tests:

$ inv karma --single

By default, Karma will run tests using a PhantomJS headless browser. You can run tests in other browsers like so:

$ inv karma -b Firefox

If you want to run cross browser tests with SauceLabs, use "sauce" parameter:

$ inv karma --sauce

Testing Addons

Addons tests are not run by default. To execute addons tests, run

$ invoke test_addons

Testing Email

First, set MAIL_SERVER to localhost:1025 in you local.py file.

website/settings/local.py

...
MAIL_SERVER = "localhost:1025"
...

Sent emails will show up in your server logs.

Optional: fire up a pseudo-mailserver with:

$ invoke mailserver -p 1025

Building front-end assets with webpack

Use the following command to update your requirements and build the asset bundles:

$ inv assets -dw

The -w option puts you in "watch" mode: the script will continue running so that assets will be built when a file changes.

Getting application credentials

Many addons require application credentials (typically an app key and secret) to be able to authenticate through the OSF. These credentials go in each addon's local.py settings file (e.g. website/addons/dropbox/settings/local.py).

COS is Hiring!

Want to help save science? Want to get paid to develop free, open source software? Check out our openings!