Builder Development

Overview

This document outlines the steps to configure and run a Builder environment for development. The Builder dev environment includes the Builder API service, as well as the Builder Web UI.

There are potentially multiple ways of creating a Builder dev environment - but supporting various systems and environments has proven to be untenable. This document includes the recommended and supported way of creating a Builder dev environment.

Prerequisites

This document currently only contains information for running on MacOS with Intel based Ubuntu Linux VMs. Information for running on Windows or other platforms is not provided in this document.

In order to run Builder on MacOS, it required that the builder services run in a Linux-based VM that can receive web content through builders ports. Also it is important that the Linux VM has access to the internet and is DNS enabled. The VM we describe here is for the open source application multipass. While the complete information source for multipass setup is contained here, we provide a section highlighting the use of multipass in our environment.

To configure multipass to work for our standalone Builder, we need to ensure we can forward content from our MacOS workstation to the builder instance running in the multipass VM.
In order to do that, use these steps to enable NAT forwarding:

Add the following line to the file /etc/pf.conf
nat on utun1 from bridge100:network to any -> (utun1)
Save the file and run the following command
sudo pfctl -f /etc/pf.conf

In addition, if running Builder UI, we will need to ensure port 9636 is forwarded to your VM. Add these entries to /etc/pf.anchors/com.apple to enable port forwarding for builder:

rdr pass on lo0 inet proto tcp from any to self port 3000 -> 127.0.0.1 port 9636

rdr pass on en0 inet proto tcp from any to self port 9636 -> <YOUR VM IP ADDRESS> 9636

If using wireless connection, the network value in the second line is likely en0. Check by running ifconfig at the terminal and update this value if necessary.

Multipass assigns a static IP address for you when you create a virtual machine. It can be found by running multipass list from a MacOS terminal under the IPv4 column, noting that the instance must be running (multipass start <VM instance>). For convenience, adding a line to /etc/hosts will enable referring to this VM by its name rather than its IP address:

builder-vm 192.168.1.10

NOTE: If you are doing active Web UI development, then you will likely want to run the UI on your Host (Mac OS). If so, there are some extra steps and configuration changes that will be needed, and those are called out below. See the Web UI README for more info. As port forwarding is required to run in this setup, you will need to ensure that the Builder API port (9636) is forwarded from your VM to your host.

Builder Status Check

You can test the API port access from your Host OS after starting Builder services (steps below) by issuing the following from the command line:

curl -v http://localhost:9636/v1/status

This should return a 200 OK.

Guest OS Provisioning

Before you can successfully build, you need to provision the OS with some basic tools and configuration.

Use visudo to grant your account the ability to do passwordless sudo. Add a line similar to the following to the end of your sudoers file: <username> ALL=(ALL) NOPASSWD: ALL
Run the following provisioning script: ./support/linux/provision.sh (Complete the repository setup step below to run this script)
Ensure you have your github SSH keys in your ~/.ssh directory (will need for cloning in the next step)

Repository Setup

The sections below will walk through the steps for getting the source and configuration ready.

Builder repo clone

Select a location to clone the Builder repo on your Linux VM, eg, ~/Workspace (this directory will be referred to as ${BUILDER_SRC_ROOT} in the sections below)

cd ${BUILDER_SRC_ROOT}
git clone https://github.com/habitat-sh/builder.git

This will clone the Builder repo into your Workspace directory.

OAuth application setup

You will need to create an OAuth application in GitHub, and use the private key, client id and client secret from the app to configure Builder's environment (below).

APP_HOSTNAME mentioned below, will typically be localhost.

However, if you are going to be doing Web UI development, and running the Web UI on your Host OS, then you will need to use localhost:3000 instead of localhost for APP_HOSTNAME.

Create a new GitHub application in your GitHub account
Give it a meaningful GitHub App name, e.g., "Builder Local Dev"
Set the value of Homepage URL to http://${APP_HOSTNAME}. A host alias that you define on your workstation pointed to a local IP such as the loopback (127.0.0.1) will suffice for APP_HOSTNAME when testing locally.
Set the value of User authorization callback URL to http://${APP_HOSTNAME}/ (The trailing / is important)
Set the value of Webhook URL to http://${APP_HOSTNAME}/ (Optional - only needed for testing builds triggered from github. APP_HOSTNAME will need to be routable on the Internet, localhost will not work.)
Set Repository metadata, Repository administration, Repository content and Organization members to read only (this is only used for your org so it's safe)
Download and save the private key. It will result in a file like app-name.date.private-key.pem
Record the the client-id, client-secret, app_id and public page link (in the left sidebar). These will be used for the GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, GITHUB_APP_ID and GITHUB_APP_URL config variables in the section below.

Builder configuration

Copy the GitHub application private key (from section above) to the following location (Important: name it exactly as shown) ${BUILDER_SRC_ROOT}/.secrets/builder-github-app.pem
Make a copy of the sample env file: cp ${BUILDER_SRC_ROOT}/.secrets/habitat-env.sample ${BUILDER_SRC_ROOT}/.secrets/habitat-env
Edit the env file with your favorite editor ${BUILDER_SRC_ROOT}/.secrets/habitat-env and populate the variables appropriately
Save and close the env file

Builder Services Setup

Starting Builder services

Once the Builder Repo is configured, Builder services can be started inside the Habitat Studio.

cd ${BUILDER_SRC_ROOT}
direnv allow
export HAB_AUTH_TOKEN=your_live_builder_token
ls ~/.hab/cache/keys/habitat-* || hab origin key generate habitat
export HAB_ORIGIN=habitat
hab studio enter

Once inside the Habitat Studio, you should see a welcome message along with a list of useful commands (Use the dev_docs command if you need to print out the commands list again).

You may now start the builder services by issuing the following command: start-builder

This will download and run the latest stable Builder packages (you will re-build everything locally in a later step).

Run status to ensure all the services are up.

You can also run sl to output the running Supervisor log as needed.

Starting the Web UI (Optional)

If you are NOT doing UI development and standing up the Web UI on your Host OS, then you don't need to do anything extra. You can just navigate to ${APP_HOSTNAME}/#/sign-in

If there are recent UI changes not yet promoted to stable that you wish to try out, then follow these additional steps to build and deploy the node/angular assets locally off of your branch:

cd components/builder-api-proxy && build
source results/last_build.env && hab pkg install results/"${pkg_artifact}"
stop-builder api-proxy
start-builder api-proxy

In the event that you ARE developing the UI then you will need to follow the instructions in the Web UI README to get the Web UI running on your Host OS.

Personal Access Token generation

Once the Builder services are up, you should generate a Personal Access Token. Currently, this can only be done via the Web UI.

Log into the Web UI - eg, navigate to http://${APP_HOSTNAME}/#/sign-in
Go the Profile page (click on the user icon in the upper right corner to get to it)
Click on the 'Generate Token' button
Save the token somewhere for later use (eg, your .bashrc or Hab cli.toml, etc.)

Note: If you need to perform commands where you auth with both the prod site, as well as the local site, remember to switch the auth tokens appropriately.

Origin creation

You should now be able to create a core origin, as well as an origin for yourself.

From within the Habitat Studio, issue the following commands:

export HAB_AUTH_TOKEN=<your token>
origin
origin <username>

This should create the origins appropriately. Note that the auth token is the Personal Access Token that you generated in the last step.

Seeding base packages

In order to do package builds locally, at a minimum you will need to seed the your dev repo with the latest version of core/hab-backline.

From within your Studio, do the following (for example, using the 0.64.1 version of hab-backline):

load_package /hab/cache/artifacts/core-hab-backline-0.64.1-20180928012546-x86_64-linux.hart

Alternatively, you can use the on-prem-archive.sh script from the on-prem repo to do the initial hydration (and sync) of base packages - see the Synchronizing Packages section below.

Plan file connection

Currently, connecting a plan file is only available from within the Web UI.

Go the Builder Web UI
Click on My Origins, and then select your origin
Click on the Connect a plan file button
Click on the Install Github App button to install the Builder Dev app on your github account
Go back to the Packages page (from Step 3), and follow the instructions to link the plan you want to build

Note: your GitHub app must have access to the repo containing the plan file you are testing. Forking habitat-sh/core-plans is an easy way to test, or feel free to create your own repo with a test plan.

Package build

You can test that the plan file you just connected actually builds by issuing a build command. You can do that either via the Builder Web UI, or via the hab cli.

Option A: From the Web UI

Navigate to http://${APP_HOSTNAME}/#/pkgs
If you are not already logged in, log in.
Click on "My origins"
Click on your origin
Click on the package you wish to build
Click on "Latest"
Click on "Build latest version"
Click on "Build Jobs" and "View the output" to see the job in progress
The job should complete successfully! Congrats, you have a working build!

Option B: From the Command Line

Issue the following command (replace origin/package with your origin and package names):

hab bldr job start origin/package

This should create a build job, and then dispatch it to the build worker.

You can view the build progress in the web UI or by viewing /hab/svc/builder-worker/data/876066265100378112/log_pipe-876066265100378112.log. Replace 876066265100378112 with the group ID output by the start command.

Once the build kicks off, you should be able to see the streaming logs for the build job in the Web UI.

Developing Builder services

Before building Builder you must ensure that your Personal Access Token is set to the production instance of Builder. This can be done by clearing the HAB_AUTH_TOKEN environment variable or explicitly setting it to your production token.

export HAB_AUTH_TOKEN=<your production token>

If the HAB_AUTH_TOKEN is not set correctly, you will likely see an error similar to the following when trying to build.

Unloading builder-api
Unloading habitat/builder-api
   : Loading /src/components/builder-api/habitat-dev/plan.sh
   builder-api: Plan loaded
   builder-api: Validating plan metadata
   builder-api: Using HAB_BIN=/hab/pkgs/core/hab/0.79.1/20190410220617/bin/hab for installs, signing, and hashing
   builder-api: hab-plan-build setup
   builder-api: Writing pre_build file
   builder-api: Resolving build dependencies
» Installing core/protobuf-cpp
☁ Determining latest version of core/protobuf-cpp in the 'stable' channel
✗✗✗
✗✗✗ [401 Unauthorized] Please check that you have specified a valid Personal Access Token.
✗✗✗
   builder-api: WARN: Could not find a suitable installed package for 'core/protobuf-cpp'
   builder-api: ERROR: Resolving 'core/protobuf-cpp' failed, should this be built first?
   builder-api: Build time: 0m0s
   builder-api: Exiting on error
ERROR: _build-builder aborted due to error

If you are developing the Builder services and changing the back end code, you will want to update the Builder services with the latest code. When first doing this, you will need to issue a full build by doing the following from within your Studio:

build-builder

This will build and restart all the services with the changes from your local branch.

Once this is done, you can incrementally change code and re-build only the services that are impacted by specifying the service name, e.g.:

build-builder api

Testing

In order to verify the API functionality, run the automated tests:

test-builder

If you'd like to preserve the resultant test data in Postgres, run as follows:

test-builder preserve

To view the DEBUG level logs from the API tests:

test-builder suplogs

Testing against pre-release core packages

In some scenarios, it's valuable to test against core packages that haven't been promoted to stable yet. Testing these requires some extra effort in the set up, as you will also need to build components from habitat

Build Habitat components

First, you will need to clone https://github.com/habitat-sh/habitat and build a subset of the components. It is important they are built in the correct order so that dependencies are correct at install time. You can use the below snippet to build them, replacing the channel as necessary.

git clone https://github.com/habitat-sh/habitat
cd habitat
env HAB_BLDR_CHANNEL=stable HAB_ORIGIN=core hab studio run "for component in hab plan-build backline studio pkg-export-docker; do build components/\$component; done"

Next, copy the hart files produced to the results directory in your copy of the Builder repository. Assuming your habitat and builder checkout share the same parent directory:

cp habitat/results/core-hab*.hart builder/results/

Next, you will need to enter the studio inside the builder directory, install the Habitat harts, and rebuild Builder against them. Once this is complete, you can follow the testing instructions detailed in the testing readme. It is safe to skip the build-builder step in that document. You can also use the test-builder helper function, shown below.

hab studio enter
hab pkg install results/core-hab*.hart
for component in builder-api builder-api-proxy builder-datastore builder-graph builder-jobsrv builder-minio builder-worker; do
  build components/$component
done

test-builder preserve

Advanced Usage

Receiving metrics

Some services like builder-api and builder-jobsrv send statsd metrics. These are easy to monitor if needed for dev purposes.

The below assumes node and npm is already installed and available.

npm install -g statsd-logger
statsd-logger

Once statsd-logger is running, it should receive and display any metrics sent by the services.

Synchronizing Packages

Follow the instructions for bootstrapping an on-prem Builder instance.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEVELOPING.md

DEVELOPING.md

Builder Development

Overview

Prerequisites

Builder Status Check

Guest OS Provisioning

Repository Setup

Builder repo clone

OAuth application setup

Builder configuration

Builder Services Setup

Starting Builder services

Starting the Web UI (Optional)

Personal Access Token generation

Origin creation

Seeding base packages

Plan file connection

Package build

Option A: From the Web UI

Option B: From the Command Line

Developing Builder services

Testing

Testing against pre-release core packages

Build Habitat components

Advanced Usage

Receiving metrics

Synchronizing Packages

Files

DEVELOPING.md

Latest commit

History

DEVELOPING.md

File metadata and controls

Builder Development

Overview

Prerequisites

Builder Status Check

Guest OS Provisioning

Repository Setup

Builder repo clone

OAuth application setup

Builder configuration

Builder Services Setup

Starting Builder services

Starting the Web UI (Optional)

Personal Access Token generation

Origin creation

Seeding base packages

Plan file connection

Package build

Option A: From the Web UI

Option B: From the Command Line

Developing Builder services

Testing

Testing against pre-release core packages

Build Habitat components

Advanced Usage

Receiving metrics

Synchronizing Packages