Fissile requires some configuration to work with BOSH releases. It is necessary to describe what docker images should be created; in general, each docker image (termed "instance group") will have one or more BOSH jobs.
There are three main files for configuration: the role manifest, opinions, and dark opinions. A set of environment files will also be required.
For examples of established configuration, please see uaa-fissile-release.git or scf.git. They are, respectively, configuration for UAA and a full (single-node) Cloud Foundry deployment.
The rest of this document will also, as an example, describe how to create
fissile configuration to place nats in a docker image.
The role manifest is the main configuration file for fissile. It must contain a list of instance groups (analogous to BOSH VMs); each group will result in one docker image. It can also have a configurable variables section to describe the tunable inputs, and a configuration templates section to map those variables to BOSH properties and related BOSHisms. We will be working with the example role manifest for NATS:
instance_groups:
- name: nats # The name of the instance group
jobs: # BOSH jobs this group will have
- name: nats
release: nats # The name of the BOSH release this is from
properties:
bosh_containerization:
run: # Runtime configuration
scaling: # Auto-scaling limits
min: 1
max: 3
memory: 256 # Memory request for each instance (MB)
virtual-cpus: 4 # CPU request for each instance
ports:
- name: nats
protocol: TCP # TCP or UDP
external: 4222 # Port visible outside the container
internal: 4222 # Port inside the container
public: false # Whether to expose to outside the cluster
- name: nats-routes
protocol: TCP
external: 4223
internal: 4223
public: false
tags:
- indexed # Mark this group as indexed (load-balanced) => StatefulSet
configuration:
templates:
networks.default.dns_record_name: '"((DNS_RECORD_NAME))"'
networks.default.ip: '"((IP_ADDRESS))"'
properties.nats.user: '"((NATS_USER))"' # In BOSH templates, `p('nats.user')`
properties.nats.password: '"((NATS_PASSWORD))"'
variables:
- name: NATS_PASSWORD
options:
description: Password for NATS
secret: true
required: true
type: password
- name: NATS_USER
options:
description: User name for NATS
required: true
previous_names: [NATS_USR]Note that there are a few special variables that are automatically supplied to the container (via run.sh). They are:
| Name | Description |
|---|---|
DNS_RECORD_NAME |
Hostname of the container |
IP_ADDRESS |
Primary IP address of the container |
KUBE_COMPONENT_INDEX |
Numeric index for instance groups with multiple replicas |
KUBERNETES_CLUSTER_DOMAIN |
Kubernetes cluster domain, cluster.local by default |
There are also some fields not shown above (as the are not needed for NATS):
For the instance group:
| Name | Description |
|---|---|
scripts |
scripts relative to the role manifest that are executed before expanding BOSH templates and starting jobs |
environment_scripts |
scripts that are sourced in bash (and could modify environment variables); executed before scripts above. |
post_config_scripts |
scripts executed after BOSH templates have been expanded, before starting jobs |
type |
bosh or bosh-task; the latter will result in a Kubernetes Job |
For the run section:
| Name | Description |
|---|---|
capabilities |
additional capabilities to grant the container (see man 7 capabilities); drop the CAP_ prefix (e.g. use NET_ADMIN) |
persistent-volumes |
volumes to attach to the instance group |
shared-volumes |
volumes shared across all containers of the instance group |
healthcheck |
optional healthchecking parameters, see below |
env |
list of environment variables, as FOO=bar |
flight-stage |
one of pre-flight, post-flight, manual, or flight (default). The first three are for jobs. |
A run section can optionally have health checking via Kubernetes container
probes. The healthcheck field may have liveness and readiness subfields,
each with one the following:
| Name | Description |
|---|---|
url |
URL to HTTP GET; expects a 2xx or 3xx reply. Use container-ip as the hostname. |
command |
Command to run; should be a list of strings. |
port |
TCP port to connect to; success is declared when the port is open |
Additionally, the following options are available:
| Name | Description |
|---|---|
initial_delay |
wait this many seconds before performing the first check |
period |
wait this many seconds between check check |
timeout |
timeout for unresponsive checks |
success_threshold |
minimal consecutive successful checks required to be considered up |
failure_threshold |
minimal consecutive failed checks required to be considered down, after previously have been successful |
For url type checks, a headers map is also available for additional HTTP
headers (for example, to set the Accept: header to request JSON responses).
The NATS instance group above was tagged as indexed, causing fissile to emit
it as a StatefulSet.
The second way of causing fissile to do that is to tag a instance group as
clustered. The main difference between clustered and indexed
groups is that fissile creates a public service (of type
LoadBalancer) for the latter, providing a single point of access to
the pods for the group.
Note that both clustered and indexed index groups can take advantage of
volume claim templates for local storage.
Therefore the user should index index groups which require load balancing and need a 0-based, incremented index, and mark them as clustered otherwise.
An example of a clustered index group is the MYSQL database of CF. See the
example below. While mysql actually needs a load balancer for access
this index group is made explicit in CF through index group mysql-proxy.
instance_groups:
- name: mysql
jobs:
- name: mysql
release: cf-mysql
provides:
mysql: {}
properties:
bosh_containerization:
run:
scaling:
min: 1
max: 3
ha: 2
capabilities: []
volumes:
- path: /var/vcap/store
tag: mysql-data
size: 20
type: persistent
memory: 2841
virtual-cpus: 2
healthcheck:
readiness:
url: http://container-ip:9200/
ports:
- name: mysql
protocol: TCP
internal: 3306
[...]
tags:
- clustered
# No implicit LB, handled by mysql-proxy, and use of volumes.
configuration:
templates:
[...]For BOSH properties that are constant across deployments, but that do not match the upstream defaults, they can be stored in an opinions file which will be embedded within the docker image. An additional dark opinions file is used to ensure that we block out anything that must be different per-cluster (for example, passwords). A third file is used for the variables found in last section of the role manifest. For the NATS instance group, we can use the following files:
-
opinions.yml
properties: nats: port: 4222
-
dark-opinions.yml
properties: nats: password: ""
-
defaults.txt
NATS_PASSWORD=nats_password
All fissile options are also available as environment variables. For that NATS release, we are just setting the paths to the things we created above. For additional options, please refer to the command documentation:
# This is a comma separated list of paths to the local repositories
# of all the releases
export FISSILE_RELEASE="nats-release"
# Path to a role manifest
export FISSILE_ROLE_MANIFEST="role-manifest.yml"
# Path to a BOSH deployment manifest that contains light opinions
export FISSILE_LIGHT_OPINIONS="opinions.yml"
# Path to a BOSH deployment manifest that contains dark opinions
export FISSILE_DARK_OPINIONS="dark-opinions.yml"
# Path to a location where all fissile output is stored
export FISSILE_WORK_DIR="output/fissile"
# The image to use as the stemcell; note that the specific image here may be outdated
export FISSILE_STEMCELL="splatform/fissile-stemcell-opensuse:42.2-6.ga651b2d-28.33"We can now assemble all the files necessary from the information above:
$ vim role-manifest.yml # See above for contents
$ vim opinions.yml
$ vim dark-opinions.yml
$ vim defaults.txt
$ git clone https://github.com/cloudfoundry/nats-release.git --recurse-submodules # from $FISSILE_RELEASEWe will also need to create a BOSH dev release:
$ rm -rf nats-release/dev_releases
$ docker run --rm \
--volume "${HOME}/.bosh/cache:/bosh-cache" \
--volume "${PWD}/nats-release:${PWD}/nats-release" \
--env "RUBY_VERSION=2.2.3" \
splatform/bosh-cli \
/usr/local/bin/create-release.sh \
"$(id -u)" "$(id -g)" /bosh-cache --dir "${PWD}/nats-release" --force --name "nats"Finally, use fissile to build the image and Kubernetes configs
# Compile packages from the nats release
fissile build packages
# Build the nats docker image
fissile build images
# Build kubernetes deployment yaml
fissile build kube