Trebuchet

Trebuchet is a two-phase deployment system with reporting implemented using SaltStack. Its primary transportation method uses git.

Design

See the design documentation at Wikimedia's Wikitech wiki.

Prerequisites

SaltStack is a hard requirement.

A mininum of two nodes is required: salt master/deployment server and a deployment target. It is recommended to use three nodes, splitting the salt master apart from the deployment server.

Installation

Using SaltStack

Include the salt states, based on the node type:

• salt master: deployment.salt_master
• deployment server: deployment.deployment_server
• deployment target: deployment.deployment_target

Note: It's best to include deployment.deployment_target from states that configure your targets. For instance, if you are running MediaWiki, you should include this state from the state that installs MediaWiki's prerequisites.

TODO: Add these salt states to the repo

Using puppet

Using the trebuchet puppet module (https://github.com/trebuchet-deploy/puppet).

TODO: Add puppet module documentation

Configuration

All of Trebuchet is configured using Salt Pillars. Two pillars are used, deployment_config and repo_config.

deployment_config

• parent_dir (default: none; required)

  A string that defines the parent directory for all repositories on both the deployment server and the targets. A repo of name test/testrepo would reside at <parent_dir>/test/testrepo.

• servers (default: none; required)

  A hash that defines datacenter to deployment server mappings.

• redis (default: none; required)

  A hash that defines redis information; required fields: host, port, db

repo_config

• grain (default: none; required)

  A string that defines the grain this repo targets. This should be the same value used in the deployment::target definition for the hosts in puppet.

• upstream (default: none)

  A string that defines the url of the upstream repo associated with this deployment repo. This is used to initialize the repo on the deployment server. If no upstream is defined then an empty repository will be created.

• shadow_reference (default: false)

  A boolean that defines whether or not this repo will create a reference clone of this repo during the fetch stage. Example: test/testrepo would also have a test/.testrepo clone on the targets that is fully checked out to the deployment tag during the fetch stage.

• fetch_module_calls (default: {})

  A hash of salt modules with a list of arguments that will get called on the minion at the end of the fetch stage of deployment.

• checkout_module_calls (default: {})

  A hash of salt modules with a list of arguments that will get called on the minion at the end of the checkout stage of deployment. The following argument expansions exist: __REPO__ expands to the name of the repo.

• checkout_submodules (default: false)

  A boolean that defines whether or not this repo should also do submodule actions. The following argument expansions exist: __REPO__ expands to the name of the repo.

• location (default: /srv/deployment/)

  The string location of the repository on the deployment system and on the minion. You should avoid setting this unless you really know what you are doing.

• sync_script (default: shared.py)

  A string that defines the sync script used for this repository. Options are depends.py and shared.py. This is used by the git-deploy hooks and is otherwise unused.

• dependencies (default: {})

  A hash of repositories with dependency scripts that this repository depends on. These repositories will be deployed automatically before this repository. Example to add the l10n-slot0 dependency for a repo, with that dependency using the l10n dependency script: l10n-slot0: l10n. Dependencies are awkward and should be avoided if possible.

• automated (default: false)

  A boolean that defines whether or not this repository is automatically or manually deployed. This is used by the git-deploy hooks and is otherwise unused.

Configuration Example

The following configuration will configure two repositories: mediawiki/slot0 and test/testrepo. Each repository will exist under /srv/deployment on both the deployment server (deployment-west.example.org) and all deployment targets with a matching grain.

Note: It's possible to break this pillar configuration into multiple files and include the configuration on sets of deployment targets, to limit the ability for a target to access a repo. To do so, you'll need to use the extend support for pillars available in the current development release.

deployment_config:
  parent_dir: /srv/deployment
  servers:
    uswest: deployment-west.example.org
  redis:
    host: deployment-west.example.org
    db: 0
    port: 6379
repo_config:
  test/testrepo:
    grain: test
  mediawiki/slot0:
    grain: mediawiki
    upstream: https://gerrit.wikimedia.org/r/p/mediawiki/core
    shadow_reference: true
    checkout_submodules: true
    fetch_module_calls:
      mediawiki.generate_localization_cache:
        - __REPO__
    checkout_module_calls:
      mediawiki.update_localization_cache:
        - __REPO__

Usage

It it currently easiest to interact with Trebuchet using Trebuchet Trigger, a git interface. Future plans will allow the use of Trebuchet through the Trebuchet Ricochet web frontend.

Assuming you've installed trigger support as described in trigger's Installation section, the basic use of trebuchet is controlled through trigger:

git deploy start
<git changes you would like to make>
git deploy sync

External Documentation

Wikimedia Foundation is currently using Trebuchet for deployment, and has more detailed documentation that is specific to their use case.

Getting Help

For bugs, please use the github issues tracker for this project. To discuss usage or development of trigger, please join me on #trebuchet-deploy on Freenode.