From d401ccaf426f6b4d2e5a6f233e1bae3c45f6d005 Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Mon, 6 Nov 2023 18:19:22 +0100 Subject: [PATCH 01/11] Complete docs about evaluation context in ERB templates --- content/jobs.md | 167 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 153 insertions(+), 14 deletions(-) diff --git a/content/jobs.md b/content/jobs.md index eeb833212..5ce10761d 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -93,20 +93,159 @@ Advanced usage: #### Using `spec` {: #properties-spec } -Each template can also access the special `spec` object for instance-specific configuration: - -- `spec.address`: Default network address (IPv4, IPv6 or DNS record) for the instance. Available in bosh-release v255.4+. -- `spec.az`: Availability zone of the instance. -- `spec.bootstrap`: True if this instance is the first instance of its group. -- `spec.deployment`: Name of the BOSH deployment containing this instance. -- `spec.id`: ID of the instance. -- `spec.index`: Instance index. Use `spec.bootstrap` to determine the first instead of checking whether the index is 0. Additionally, there is no guarantee that instances will be numbered consecutively, so that there are no gaps between different indices. -- `spec.name`: Name of the instance. -- `spec.networks`: Entire set of network information for the instance. -- `spec.release`: BOSH release for the instance. -- `spec.ip`: IP address of the instance. In case multiple IP addresses are available, the IP of the [addressable or default network](networks.md#multi-homed) is used. Available in bosh-release v258+. - -Use the `spec` object directly in your templates: `<%= spec.ip %>` +Each template can also access the special `spec` object for instance-specific +configuration. Remember that job properties are initially defined at the +_instance group_ level in the deployment manifest. + +Release authors can the `spec` object directly in the ERB templates: +`<%= spec.ip %>`. + +The accessible properties fall into three categories: Bosh structure +information, networking setup, and instance configuration. + +##### Structural info + +- `spec.deployment`: Name of the BOSH deployment defining the instance group. +- `spec.name`: Name of the instance group that the instance belongs to. +- `spec.az`: The availability zone that the instance is placed into. +- `spec.id`: Unique and immutable UUID of the instance. +- `spec.index`: Ordinal and numeric “human friendly” instance index. Indexes + usually start a `0`, but with no guarantee. Gaps may appear anywhere in the + numbering, and the first instance in the group may have a non-zero index. +- `spec.bootstrap`: Boolean that is `true` if the instance is the first + instance of its group. + +!!! Note + With `spec.index`, Bosh doesn't guarantee that instances will be numbered + consecutively. Determining which instance is the first its group is a very + common requirement, so that certain things get bootstrapped by one single + node of a cluster, like database schema migrations, or admin password + enforcement. When facing such requirement, release authors should not + assume there is necessarily an instance with index `0`, and use + `spec.bootstrap` instead. + +##### Networking setup + +- `spec.address`: Default network address for the instance. This can be an + IPv4, an IPv6 address or a DNS record, depending on the Director's + configuration. Available in bosh-release v255.4+. +- `spec.ip`: IP address of the instance. In case multiple IP addresses are + available, the IP of the + [addressable or default network](networks.md#multi-homed) is used. Available + in bosh-release v258+. +- `spec.dns_domain_name`: the configured root domain name for the Director, + which defaults to `bosh`, meaning that the configured Top-Level Domain (TLD) + for Bosh DNS is `.bosh`. +- `spec.networks`: Entire set of network information for the instance. Example: + + ```yaml + default: + type: manual + ip: 10.224.0.129 + netmask: 255.255.240.0 + cloud_properties: + name: random + default: + - dns + - gateway + gateway: 10.224.0.1 + dns_record_name: 0....bosh + ``` + +!!! Note + Release authors are encouraged to favor `spec.address` over `spec.ip`. The + `spec.ip` property is provided only for use-cases where a numeric IP + address (either IPv4 or IPv6) is absolutely required. !!! warning When **dynamic** networks are being used, `spec.ip` might not be available. + +##### Instance configuration + +- `spec.persistent_disk`: is `0` if no persistent disk is mounted to the + instance. In case the deployment manifest does declare a persistent disk + attached to the instances of the group, this `persistent_disk` is given a + `0` value when the deployment manifests instructs to remove the instance + from the group and delete it (typical for _scaled-in_ operations, as opposed + to _scale-out_ where new instances are “horizontally” added to a group). +- `spec.release.name`: The name of the BOSH Release where the instance job is + originally defined. +- `spec.release.version`: Version of the BOSH release that the instance job + relies on. + +##### Link properties + +Remember that the job targeted through alink can live in a different instance +group of a different deployment. + +- Structural info + - `link(name).deployment_name`: Deployment name of the linked job. + - `link(name).instance_group`: Instance group name of the linked job. + - `link(name).group_name`: A concatenation of the link name and link type, + separated by a dash `-`, i.e. `-`. + - `link(name).instances`: An array of details for each instance of the group. + - `link(name).instances[].az`: the availability zone hat the instance is + placed into + - `link(name).instances[].name`: instance group name. Alias for + `link().instance_group`. + - `link(name).instances[].id`: instance immutable UUID + - `link(name).instances[].index`: human-friendly instance ordinal + - `link(name).instances[].bootstrap`: whether the instance is the first of + its group +- Networking setup + - `link(name).default_network`: default network for the instance group. + - `link(name).networks`: list of all networks for the instance group. **TO BE TESTED** + - `link(name).address`: an address for the instance group, using the `q-s0` + prefix, indicating the `smart` health filter. + See [Native DNS Support](dns.md) for more details. + - `link(name).domain`: the root top-level domain name suffix. Defaults to `bosh`. + - `link(name).use_link_dns_names`: applicable config for the link. **TO BE TESTED** + - `link(name).use_short_dns_addresses`: applicable config for the link. **TO BE TESTED** + - `link(name).instances[].address`: the instance address, that can be an + IPv4, an IPv6 address or a DNS record, depending on the Director's + configuration, but is usually a DNS name, ending with the suffix + indicated in the `link().domain` property. + - `link(name).instances[].addresses`: several addresses including aliases? **TO BE TESTED** + - `link(name).instances[].dns_addresses`: same as above, but preferring DNS entry +- Configuration + - `link(name).properties`: The job properties that are exposed by the link. + +##### Deprecated properties accessors + +- `name`: the instance group name. Alias for `spec.name` (recommended). +- `index`: the instance index in its group. Alias for `spec.index` (recommended). +- `properties`: the job properties, as defined in the instance group. Alias + for `spec.properties`. Doesn't provide elementary error reporting. +- `spec.properties`: The properties defined in the deployment manifest for the + instance job that the templates belongs to. Accessing properties through + this object leads to poor error reporting and is highly discouraged. Bosh + Release authors should use the `p()` accessor instead, which implements + proper error reporting, and properly prevents misconfiguration. + +With Bosh v1, the term “job” was designating an “instance group”. The use of +`spec.job` in ERB templates could possibly be used by legacy Bosh releases but +its usage is highly discouraged. It is documented here only to help release +authors to migrate to the standardized `p()` property accessor. + +- `spec.job`: instance group spec. This is an old Bosh v1 naming, when _job_ + did actually mean _instance group_. `nil` when no job is defined for the + instance group. +- `spec.job.name`: name of the instance group that the template belongs to. +- `spec.job.template`: name of the first job in the instance group, which is + only relevant if it is the “default errand”, a legacy “Bosh v1” concept + before it was decided that any job that defines a `bin/run` script can be + run as an errand. +- `spec.job.version`: version of the first job in the instance group, only + relevant if it is the “default errand” (legacy concept). +- `spec.job.templates`: an array of jobs for the instance group +- `spec.job.templates.*.name`: name of the job +- `spec.job.templates.*.version`: version as defined in the release jobs manifests +- `spec.job.templates.*.sha1`: digital fingerprint of the job (nowadays with a + `sha256:` prefix for SHA256) +- `spec.job.templates.*.blobstore_id`: where to find the job tarball in the + Director's blobstore. +- `spec.job.templates.*.logs`: an empty array of logs files, related to the + legacy `logs` hash in a release job spec, which is undocumented. +- `spec.properties_need_filtering`: Whether properties from other instance + groups should not be exposed to this job. This is legacy, and should not be + here. From 11c4637102654b56934ce89343269f0b0e0fb344 Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Mon, 6 Nov 2023 18:20:27 +0100 Subject: [PATCH 02/11] Provide more examples on possible ERB idioms --- content/jobs.md | 45 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/content/jobs.md b/content/jobs.md index 5ce10761d..fa47bfad8 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -82,14 +82,55 @@ Basic ERB syntax includes: - `<%= "value" %>`: Inserts string "value". - `<% expression %>`: Evaluates `expression` but does not insert content into the template. Useful for `if/else/end` statements. +- `<% expression -%>`: Evaluates `expression`, does not insert any content, + and remove the newline `\n` character that might be after the `-%>`. Templates have access to merged job property values, built by merging default property values and operator specified property values in the deployment manifest. To access properties `p` and `if_p` ERB helpers are available: - `<%= p("some.property") %>`: Insert the property `some.property` value, else a default value from the job spec file. If `some.property` does not have a default in the spec file, error will be raised to the user specifying that property value is missing. Advanced usage: - Operator `p` can take optional parameter as a default value, e.g. `<%= p("some.property", some_value) %>`. This value is used as a last resort. - - The first parameter can be an array, e.g. `<%= p(["some.property1", "some.property2"], some_value) %>`. Value of the first property which is set will be returned. -- `<% if_p("some.property") do |prop| %>...<% end %>` - Evaluates the block only if `some.property` property has been provided. The property value is available in the variable `prop`. Multiple properties can be specified: `<% if_p("prop1", "prop2") do |prop1, prop2| %>`. + - The first parameter can be an array, e.g. + `<%= p(["some.property1", "some.property2"], some_value) %>`. Value of the + first property which is set (i.e. non-`null`) will be returned. +- A part of the template can be evaluated only when some property is provided. + `<% if_p("some.property") do |prop| %>...<% end %>` evaluates the block only + if `some.property` property has been provided. The property value is + available in the variable `prop`. + - Multiple properties can be specified: + `<% if_p("some.prop1", "other.prop2") do |prop1, prop2| %>...<% end %>`, + in which case the block is evaluated only if _all_ the properties are + defined. + +After the `end` of an `if_p` block, the `.else ... end` and +`.else_if_p("other.property") ... end` syntaxes are supported. + +- `<% if_p("some.property") do |prop| %>...<% end.else %>...<% end %>` - + Evaluates first block if `some.property` has been provoded (or has a default + in job spec), otherwise evaluates the second block. +- `<% if_p("some.property") do |prop| %>...<% end.else_if_p("other.property") |prop2| %>...<% end.else %>...<% end %>` - + Evaluates first block if `some.property` has been provided (or has a default + in job spec), otherwise evaluates the second block if `other.property` has + been provided (or has a default in job spec), otherwise evaluates the third + block. + +The link navigation syntax `link()` also provides similar `.p()` and `.if_p()` +methods, and `.else_if_p()` or `.else` blocks. + +- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end %>` - + If `remote.prop` is defined in the job that is resolved through navigating + the `relation-name` link, then the block is evaluated with the value in the + local variable `prop`. +- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end.else %>...<% end %>` - + Same as above with an `.else ... end` block. +- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end.else_if_p("other.prop2") |prop2| %>...<% end.else %>...<% end %>` - + Same as above with an `.else_if_p` block that evaluates only when + `other.prop2` is defined through navigating the `relation-name` link. + +See [Links](links.md) and [Links Properties](links-properties.md) for more +details on navigating links to fetch configuration properties from other jobs, +possibly declared in different instance groups, and even possibly living in +different deployments. #### Using `spec` {: #properties-spec } From c29315a3442821fab522c62b03d50725e6803059 Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Mon, 6 Nov 2023 18:26:37 +0100 Subject: [PATCH 03/11] Improve documentation for release job definition (in 'spec' file) + Document consumer/provider schema for links + Clarify templates declatation --- content/jobs.md | 69 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 61 insertions(+), 8 deletions(-) diff --git a/content/jobs.md b/content/jobs.md index fa47bfad8..c7e96e100 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -38,13 +38,63 @@ Schema: * **name** [String, required]: Name of the job. * **description** [String, optional]: Describes purpose of the job. -* **templates** [Hash, optional]: [Template files](#templates) found in the `templates` directory of the job and their final destinations relative to the job directory on the deployed VMs. By convention executable files should be placed into `bin/` directory so that the Agent can mark them as executable, and configuration files should be placed into `config/` directory. +* **templates** [Hash, optional]: [Template files](#templates) found in the + `templates` directory of the job (keys of the Hash) and their final + destinations (values of the Hash), relative to the job directory on the + deployed VMs. + * **<key>** [String, required]: the relative path and filename of the + ERB template provided by the job in the release, relative to the + `templates` sub-directory. No need for any `.erb` suffix, all templates + are treated as ERB templates whatever their name is. + * **<value>** [String, required]: the relative path and filename of the + rendered file, relative to the job directory (i.e. + `/var/vcap/jobs//`) on the managed Bosh instances (a.k.a. the + “deployed VMs”). By convention, executable files should be placed into + `bin/` directory so that the Agent can mark them as executables, and + configuration files should be placed into `config/` directory. * **packages** [Array, optional]: Package dependencies required by the job at runtime. +* **consumes** [Array, optional]: Links that are consumed by the job for + rendering ERB templates. + * **name** [String, required]: Name of the link to find. + * **type** [String, required]: Type of the link to be found. This is an + arbitrary naming. Usual and conventional types are `address` when the + link goal is to expose a Bosh DNS name atht allows accessing the + instances of the group. Usually typed by technology, like `mysql`, + `postgres`, `cassandra`, etc. Anything that makes sense is relevant and + matters. + * **optional** [Boolean, optional]: Whether finding an matching link is + optional (when `true`) or mandatory (when `false`. Default is `false`, + so optinal links must be explicitly declared as such. +* **provides** [Array, optional]: Links that are exposed to other jobs for + rendering their ERB templates. + * **name** [String, required]: Name of the exposed link. + * **type** [String, required]: Type of the exposed link. + * **properties** [Array, optional]: List of property keys in dot notation + (same as **properties.<name>** below) * **properties** [Hash, optional]: Configuration options supported by the job. - * **\** [String, required]: Property key in dot notation. Typical properties include account names, passwords, shared secrets, hostnames, IP addresses, port numbers, and descriptions. - * **description** [String, required]: Describes purpose of the property. - * **example** [Any, optional]: Example value. Default is `nil`. - * **default** [Any, optional]: Default value. Default is `nil`. + * **<name>** [String, required]: Property key in dot notation. Typical + properties include account names, passwords, shared secrets, hostnames, + IP addresses, port numbers, and descriptions. + * **description** [String, required]: Describes purpose of the + property. This is not used by the Director, but is displayed in job + configuration details provided by the [release index](/releases). + * **type** [String, optional]: The type of the property. This is only + a convention for release authors to provide a type when they + estimate it useful. Example: `type: certificate`. + * **example** [Any, optional]: Example value, to be displayed in the + [release index](/releases). Default is `nil`. + * **default** [Any, optional]: The default value for the property. + Default is `nil`. + +!!! Note + Within a peoperty definition, `default` is used by the Director, and + `description`, `default` and `example` are displayed by the + [release index](/releases). In turns, other keys like `type` are used only + for convenience, like Concourse does `env` keys in the + [“web” job definition][concourse_web_spec]. Indeed, the schema is not + formally validated by the Director when registering a release job. + +[concourse_web_spec]: https://github.com/concourse/concourse-bosh-release/blob/8d2cfa0/jobs/web/spec#L68-L71 --- ## Templates (ERB configuration files) {: #templates } @@ -180,7 +230,7 @@ information, networking setup, and instance configuration. - `spec.networks`: Entire set of network information for the instance. Example: ```yaml - default: + : type: manual ip: 10.224.0.129 netmask: 255.255.240.0 @@ -198,8 +248,11 @@ information, networking setup, and instance configuration. `spec.ip` property is provided only for use-cases where a numeric IP address (either IPv4 or IPv6) is absolutely required. -!!! warning - When **dynamic** networks are being used, `spec.ip` might not be available. +!!! Warning + When **dynamic** networks are being used, `spec.ip` might not be + available, then the value `127.0.0.1` is provided instead. This applies + to `spec..ip`, `spec..netmask` and + `spec..gateway`. ##### Instance configuration From baac18a79422fe21a0199ab6185f97bac6cbeeff Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Mon, 6 Nov 2023 18:26:37 +0100 Subject: [PATCH 04/11] Rewrite recommendations and bring BPM upfront + Document consumer/provider schema for links + Clarify templates declatation --- content/jobs.md | 123 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 111 insertions(+), 12 deletions(-) diff --git a/content/jobs.md b/content/jobs.md index c7e96e100..145f0caff 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -22,7 +22,7 @@ name: http-server description: This job runs a simple HTTP server. templates: - ctl.sh: bin/ctl + bpm.yml: config/bpm.yml config.json: config/config.json packages: @@ -99,30 +99,129 @@ Schema: --- ## Templates (ERB configuration files) {: #templates } -Release author can define zero or more templates for each job, but typically you need at least a template for a control script. +Release authors can define zero or more templates for each job, but typically +you need at least a template for the BPM config file, that is expected to +renderd to the `config/bpm.yml` location. -### Monit {: #monit } +### Favor Bosh Process Manager (BPM) over Monit {: #monit } -Example `monit` file for configuring single process that can start, monitor and stop a Postgres process: +Monit is a component of the Bosh architecture that was introduced in the early +days, and has always been deemed to get rid of soon. But history has shown +over the years that transitionning away from Monit is complex-engough for +being hold back since then. +As a consequence, Release authors should avoid at all costs relying on fancy +Monit features. Instead, they should use a very simple and standard `monit` +file, and leverage the battle-tested and well-designed +[Bosh Process Manager (BPM)](bpm/bpm). + +Whenever BPM would miss some required features, then contributions should be +submitted as Pull Requests to [its Git repository][bpm_repo] so that more +use-case are covered. + +[bpm_repo]: https://github.com/cloudfoundry/bpm-release/tree/master/src/bpm + +#### BPM Configuration + +The release needs to render a `config/bpm.yml` file following the +configuration schema defined in the BPM documentation. The schema is clean and +configuring BPM is straightforward. See [BPM Configuration Format](bpm/config) +for more details. + +Here is a simple example `bpm.yml` config, showcased in the +[Exemplar Bosh Release][exemplar_bpm]. + +[exemplar_bpm]: https://github.com/cloudfoundry/exemplar-release/blob/latest/jobs/sample-app/templates/bpm.yml + +```yaml +<% require "json" -%> +processes: + - name: sample-app + executable: /var/vcap/packages/sample-app/bin/sample-app + args: [] + env: + PORT: <%= p('port').to_json %> + CF_INSTANCE_INDEX: <%= spec.index.to_json %> ``` + +Release author need a basic understanding of the isolation mechanisms enforced +by BPM, especially read-only root disk remouting, and declarating read-write +access to portions only of the mount space. + +See the [BPM Runtime Environment](bpm/runtime) for more details on these topics. + +#### Standard `monit` shim with BPM + +Here is the standard `monit` file that Release authors should use. Only +replace `` by the actual job name. + +```monit +check process + with pidfile /var/vcap/sys/run/bpm//.pid + start program "/var/vcap/jobs/bpm/bin/bpm start " + stop program "/var/vcap/jobs/bpm/bin/bpm stop " + group vcap +``` + +#### Legacy pattern with `*_ctl` script (highly discouraged) + +Legacy `monit` files are using a `*_ctl` scripts that conventionally accept +`start` or `stop` as first argument. We document this here only for release +author to spot this old pattern and properly +[transition to the BPM pattern](bpm/transitioning). + +```monit check process postgres - with pidfile /var/vcap/sys/run/postgres/pid - start program "/var/vcap/jobs/postgres/bin/ctl start" - stop program "/var/vcap/jobs/postgres/bin/ctl stop" + with pidfile /var/vcap/sys/run/postgres/postgres.pid + start program "/var/vcap/jobs/postgres/bin/postgres_ctl start" + stop program "/var/vcap/jobs/postgres/bin/postgres_ctl stop" ``` -### Control script (`*_ctl` script) {: #ctl } +This is highly discouraged, because experience has show that the `*_ctl` +scripts have so many small details to care about, that this pattern leads to +tremendous boiler-plate, untested and fragile script code in Bosh releases. +Would release authors not be able to use BPM for some good reason, then +leveraging the standard `start-stop-daemon` utility is a cleaner and more +robust pattern. + +For completeness, see the [Exemplar Release][start_stop_daemon_example] with +detailed examples on the `start-stop-daemon` pattern, though release authors +are encouraged to use BPM instead. -In a typical setup, control script is called by the Monit when it tries to start, and stop processes. +[start_stop_daemon_example]: https://github.com/cloudfoundry/exemplar-release/blob/latest/jobs/paragon/templates/start -Monit expects that executing "start program" directive will get a process running and output its PID into "pidfile" location. Once process is started, Monit will monitor that process is running and if it exits, it will try to restart it. +#### Monit expectations -Monit also expects that executing "stop program" directive will stop running process when the Director is restarting or shutting down the VM. +In a typical setup, BPM is called by Monit when OS processes, whether daemons +or one-off errand jobs, need to be started or stopped. + +Monit expects that executing "bpm start" directive will get a process running +and output its PID into the file given by the `with pidfile` declaration. Once +the process is started by BPM, Monit will monitor the daemon process, based on +the PID that can be found in the `pidfile`, and if the process cease to exist, +Monit will try to start it again. + +Monit also expects that executing `bpm stop` will stop the running process. +BPM offers the best guarantees for that, and properly adapts to some +documented defects of Monit in that regards. +See the [Monit Workarounds](bpm/runtime.md#monit-workarounds) for more details. ### Hook scripts {: #hooks } -There are several job lifecycle events that a job can react to: pre-start, post-start, post-deploy, and drain. See [Job lifecycle](job-lifecycle.md) for the execution order. +There are several job lifecycle events that a job can react to: `pre-start`, +`post-start`, `post-deploy`, `pre-stop`, `post-stop`, and `drain`. + +See [Job lifecycle](job-lifecycle.md) for more details on the exact execution +order of these hook scripts. + +The Exemplar Release demonstrate state-of-the-art implementations for +[`post-start`][post_start_example] or [`drain`][drain_example], including +helpful boilerplate that provide proper timestamping of logs for hook scripts, +which has proven very useful when troubleshooting issues while developing Bosh +Releases. + +[post_start_example]: https://github.com/cloudfoundry/exemplar-release/blob/latest/jobs/paragon/templates/post-start +[drain_example]: https://github.com/cloudfoundry/exemplar-release/blob/latest/jobs/paragon/templates/drain ### Use of Properties {: #properties } From 286a0f0838717987ecd190e63b49052cbf9120ff Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Mon, 6 Nov 2023 18:35:58 +0100 Subject: [PATCH 05/11] Cleaner display through markdown and indenting improvements --- content/dns.md | 278 ++++++++++++++++++++++++++----------------------- 1 file changed, 145 insertions(+), 133 deletions(-) diff --git a/content/dns.md b/content/dns.md index a8fb14170..ac8852649 100644 --- a/content/dns.md +++ b/content/dns.md @@ -143,7 +143,7 @@ By default, a VM is considered healthy if the process manager reports all proces Jobs which provide services can be configured to be addressable via a static alias. A job that currently provides a link can be aliased directly by updating the manifest to add the alias in the `provides` configuration of the job. Here is an example: -``` +```yaml instance_groups: - name: instance-group0 jobs: @@ -159,7 +159,7 @@ instance_groups: If the job as defined in the release does not currently provide a link, you can still define an alias to that job but first you must define a custom link provider in order to do so like this: -``` +```yaml instance_groups: - name: instance-group0 jobs: @@ -183,7 +183,8 @@ instance_groups: A basic alias is an _unparameterized_ alias on a _constant domain_ with a _constant_ query. It returns all IPs matching the filter that provide that link. Example: -``` + +```yaml aliases: - domain: my-service.my-domain health_filter: smart/healthy/unhealthy/all @@ -195,7 +196,8 @@ A wildcard alias is an _unparameterized_ alias on a _wildcard domain_ with a _co It returns all IPs matching the filter that provide that link. Example: -``` + +```yaml aliases: - domain: "*.cloud-controller-ng.service.cf.internal" health_filter: smart/healthy/unhealthy/all @@ -203,13 +205,17 @@ aliases: ``` ###### Placeholder alias -A placeholder alias is a _parameterizable_ alias on a _wildcard domain_ with a _variable_ query. -It returns IPs matching both the filter that provides that link and the placeholder replacement. +A placeholder alias is a _parameterizable_ alias on a _wildcard domain_ with a +_variable_ query. It returns IPs matching both the filter that provides that +link and the placeholder replacement. -It allows referencing a placeholder (_) specified in the alias. The type of the placeholder can be configured, to allow referencing by instance uuid, index, availability_zone, or network. +It allows referencing a placeholder (_) specified in the alias. The type of +the placeholder can be configured, to allow referencing by instance `uuid`, +`index`, `availability_zone`, or `network`. Example: -``` + +```yaml aliases: - domain: "_.cloud-controller-ng.service.cf.internal" placeholder_type: uuid/index/az/network @@ -269,8 +275,8 @@ Setting this to `synchronous` will force BOSH to wait for the first health statu It is possible for more than one link provider to define domains with the exact same values. If this happens, the queries will each be run independently and the results will be merged. For example, with the following deployment manifest: -deployment.yml: -``` + +```yaml instance_groups: # ... - name: proxied @@ -278,7 +284,7 @@ instance_groups: - name: nginx provides: conn: - aliases: + aliases: - domain: "api.bosh.internal" # ... - name: direct @@ -504,7 +510,7 @@ The recommended way to hook the links providers with the consumers variables is In the example below, the variable of type certificate `app_server_cert` is explicitly consuming `alternative_name` from the `my-custom-app-server-address` provider. This will lead to the `app_server_cert` certificate being generated with an additional SAN: the BOSH DNS address of the instance group `server_ig` where the link provider (the job `app_server`) exists. For example: `q-s0.server_ig.default.app-service.bosh`. -``` +```yaml name: app-service ... @@ -517,7 +523,7 @@ instance_groups: app-server-address: as: my-custom-app-server-address custom_provider_definitions: - - name: app-server-address + - name: app-server-address type: address ... @@ -542,7 +548,7 @@ It is also possible to set the common name to the appropriate BOSH DNS record. In the example below, the variable of type certificate `app_server_cert` is explicitly consuming `common_name` from the `my-custom-app-server-address` provider. This will set the Common Name of `app_server_cert` generated certificate to be the BOSH DNS address of the instance group `server_ig` where the link provider (the job `app_server`) exists. For example, the common name will be set to `q-s0.server_ig.default.app-service.bosh`. -``` +```yaml variables: - name: app_server_cert type: certificate @@ -556,7 +562,7 @@ variables: If the application talks to specific instances or uses different healthiness filtering, it may be useful to request a wildcard DNS name when consuming a link for either SANs or common name: -``` +```yaml variables: - name: app_server_cert type: certificate @@ -585,7 +591,7 @@ For example, the `app_server_cert` cert below will have "**Application Server**" * DNS: `*.serverig.default.app-service.bosh` * IP: 172.158.20.255 -``` +```yaml variables: - name: app_server_cert type: certificate @@ -609,128 +615,134 @@ variables: BOSH DNS Health Monitor Certificates should be performed in three steps in order to achieve zero downtime. Given you used bosh-deployment to update your runtime config as in: -``` +```shell bosh update-runtime-config bosh-deployment/runtime-configs/dns.yml --vars-store bosh-dns-certs.yml ``` 1. Step 1 (Add new CA Certificates to runtime config): - This will make sure the new certificates (step 2) will be properly validated against new CA Certificate, - and old certificates will be validated against the previous one. - - ``` - cat > rotate-dns-certs-1.yml < rotate-dns-certs-2.yml < rotate-dns-certs-1.yml < rotate-dns-certs-2.yml < Date: Mon, 6 Nov 2023 18:44:31 +0100 Subject: [PATCH 06/11] Add note about placeholder types that target single instances --- content/dns.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/content/dns.md b/content/dns.md index ac8852649..f281dfb8d 100644 --- a/content/dns.md +++ b/content/dns.md @@ -223,6 +223,17 @@ aliases: initial_health_check: asynchronous/synchronous ``` +!!! Note + When using `placeholder_type: uuid` or `placeholder_type: index`, the + value for `health_filter` has some importance. Indeed, when the instance + behind the alias is unhealthy, the default `health_filter: smart` will + resolve the alias to no address at all, or if you prefer, an empty list of + IP addresses. Depending on your use-case, you might expect an `index`- or + `uuid`-based alias to always return the IP address of the designated + instance, and let clients load-balance the traffic to healthy instances + with their own mechanisms. With such use-case, that expects one instance + to always be resolved, then opt for `health_filter: all`. + ###### Parameters in Detail **domain** [String] (*required*) From d685372dfc2fe9efe926c595763df2f277361220 Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Fri, 10 Nov 2023 21:18:53 +0100 Subject: [PATCH 07/11] Add caveat about job name, missing in ERB evaluation context --- content/jobs.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/content/jobs.md b/content/jobs.md index 145f0caff..6fe5a0611 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -314,6 +314,13 @@ information, networking setup, and instance configuration. assume there is necessarily an instance with index `0`, and use `spec.bootstrap` instead. +!!! Caveat + From within an ERB template, there is no programatic way to know the name + of the job that the template is defined in. Thus release authors are + forced to hardcode the job name in ERB templates that need it. Due to this + limitation, there is unfortunately no way to write ERB templates that are + agnostic of the job name they belong to. + ##### Networking setup - `spec.address`: Default network address for the instance. This can be an From d2630518b50982d55e61e3e14bc48afd21da642d Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Tue, 28 Nov 2023 13:50:01 +0100 Subject: [PATCH 08/11] Also mention the 'do' that have to follow '.else' clauses --- content/jobs.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/jobs.md b/content/jobs.md index 6fe5a0611..9fdc7bccb 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -251,13 +251,13 @@ Advanced usage: in which case the block is evaluated only if _all_ the properties are defined. -After the `end` of an `if_p` block, the `.else ... end` and -`.else_if_p("other.property") ... end` syntaxes are supported. +After the `end` of an `if_p` block, the `.else do ... end` and +`.else_if_p("other.property") do ... end` syntaxes are supported. -- `<% if_p("some.property") do |prop| %>...<% end.else %>...<% end %>` - +- `<% if_p("some.property") do |prop| %>...<% end.else do %>...<% end %>` - Evaluates first block if `some.property` has been provoded (or has a default in job spec), otherwise evaluates the second block. -- `<% if_p("some.property") do |prop| %>...<% end.else_if_p("other.property") |prop2| %>...<% end.else %>...<% end %>` - +- `<% if_p("some.property") do |prop| %>...<% end.else_if_p("other.property") do |prop2| %>...<% end.else do %>...<% end %>` - Evaluates first block if `some.property` has been provided (or has a default in job spec), otherwise evaluates the second block if `other.property` has been provided (or has a default in job spec), otherwise evaluates the third @@ -270,9 +270,9 @@ methods, and `.else_if_p()` or `.else` blocks. If `remote.prop` is defined in the job that is resolved through navigating the `relation-name` link, then the block is evaluated with the value in the local variable `prop`. -- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end.else %>...<% end %>` - - Same as above with an `.else ... end` block. -- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end.else_if_p("other.prop2") |prop2| %>...<% end.else %>...<% end %>` - +- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end.else do %>...<% end %>` - + Same as above with an `.else do ... end` block. +- `<%= link("relation-name").if_p("remote.prop") do |prop| %>...<% end.else_if_p("other.prop2") do |prop2| %>...<% end.else do %>...<% end %>` - Same as above with an `.else_if_p` block that evaluates only when `other.prop2` is defined through navigating the `relation-name` link. From 11616e7f20e70413964f1e6207d51d78c03f830a Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Tue, 28 Nov 2023 14:36:02 +0100 Subject: [PATCH 09/11] Fix typos --- content/jobs.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/jobs.md b/content/jobs.md index 9fdc7bccb..829be046f 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -58,13 +58,13 @@ Schema: * **name** [String, required]: Name of the link to find. * **type** [String, required]: Type of the link to be found. This is an arbitrary naming. Usual and conventional types are `address` when the - link goal is to expose a Bosh DNS name atht allows accessing the + link goal is to expose a Bosh DNS name that allows accessing the instances of the group. Usually typed by technology, like `mysql`, `postgres`, `cassandra`, etc. Anything that makes sense is relevant and matters. * **optional** [Boolean, optional]: Whether finding an matching link is optional (when `true`) or mandatory (when `false`. Default is `false`, - so optinal links must be explicitly declared as such. + so optional links must be explicitly declared as such. * **provides** [Array, optional]: Links that are exposed to other jobs for rendering their ERB templates. * **name** [String, required]: Name of the exposed link. @@ -87,7 +87,7 @@ Schema: Default is `nil`. !!! Note - Within a peoperty definition, `default` is used by the Director, and + Within a property definition, `default` is used by the Director, and `description`, `default` and `example` are displayed by the [release index](/releases). In turns, other keys like `type` are used only for convenience, like Concourse does `env` keys in the @@ -101,14 +101,14 @@ Schema: Release authors can define zero or more templates for each job, but typically you need at least a template for the BPM config file, that is expected to -renderd to the `config/bpm.yml` location. +rendered to the `config/bpm.yml` location. ### Favor Bosh Process Manager (BPM) over Monit {: #monit } Monit is a component of the Bosh architecture that was introduced in the early days, and has always been deemed to get rid of soon. But history has shown -over the years that transitionning away from Monit is complex-engough for -being hold back since then. +over the years that transitioning away from Monit is complex-enough for being +hold back since then. As a consequence, Release authors should avoid at all costs relying on fancy Monit features. Instead, they should use a very simple and standard `monit` From 9c9df0ddaf480cb702d14dd2228e8cc349b510d9 Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Tue, 28 Nov 2023 14:51:02 +0100 Subject: [PATCH 10/11] Mention that 'use_dns_addresses' is actually enabled on most Bosh installations --- content/dns.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/content/dns.md b/content/dns.md index f281dfb8d..15a4c2439 100644 --- a/content/dns.md +++ b/content/dns.md @@ -353,7 +353,13 @@ As of bosh-release v263 opting into DNS addresses in links must be done explicit You can control type of addresses returned at three different levels: -- for the entire Director via Director job configuration [`director.local_dns.use_dns_addresses` property](https://bosh.io/jobs/director?source=github.com/cloudfoundry/bosh#p=director.local_dns.use_dns_addresses) that if enabled affects all deployments by default. We are planning to eventually change this configuration to true by default. +- for the entire Director, via the `director` job configuration [`director.local_dns.use_dns_addresses` property](https://bosh.io/jobs/director?source=github.com/cloudfoundry/bosh#p=director.local_dns.use_dns_addresses) that affects all deployments when enabled. We are planning to eventually change this configuration to `true` by default. + + Please note that although the default for this setting is `false` in the + Bosh Release for the Bosh Director, the “bosh-deployment” in turn, + [enables it by default](https://github.com/cloudfoundry/bosh-deployment/blob/1128dd7/bosh.yml#L82). + So, in fact most Bosh installations already have the `use_dns_addresses` + enabled. - for a specific deployment via [`features.use_dns_addresses` deployment manifest property](manifest-v2.md#features) that if enabled affects links within this deployment From a68fec3f6d45cd5b1ca3f945053b4dbefd48221b Mon Sep 17 00:00:00 2001 From: Benjamin Gandon Date: Tue, 28 Nov 2023 14:55:21 +0100 Subject: [PATCH 11/11] Document how to properly obtain the name of the 'default' network --- content/jobs.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/content/jobs.md b/content/jobs.md index 829be046f..99f7a6258 100644 --- a/content/jobs.md +++ b/content/jobs.md @@ -360,6 +360,29 @@ information, networking setup, and instance configuration. to `spec..ip`, `spec..netmask` and `spec..gateway`. +Fetching the name of the network that has the default gateway is particularily +complex, as the `spec` object is not a Ruby Hash, but an OpenStruct. + +As a consequence, one cannot use the `.keys` method for listing the network +names. But Bosh provides a special helper method `.methods(false)` on the +OpenStruct that does the trick. + +So, one can use the expression `spec.networks.methods(false)` in order to list +the network names. Based on this, use the following code snippet in your ERB +templates, whenever you need the name of the “default” network (i.e. the one +use for the default gateway, at least). + +```ruby +network = spec.networks.methods(false).find { |net_name| + # Pick the network that is used for the default gateway + default_for = spec.networks[net_name].default + !default_for.nil? && default_for.include?("gateway") + } +``` + +Obtaining the default network is absolutely necessary when building default +Bosh DNS FQDNs, which include that network name. + ##### Instance configuration - `spec.persistent_disk`: is `0` if no persistent disk is mounted to the