diff --git a/doc/man5/flux-config-bootstrap.rst b/doc/man5/flux-config-bootstrap.rst index 994bbd827750..899c65d0854a 100644 --- a/doc/man5/flux-config-bootstrap.rst +++ b/doc/man5/flux-config-bootstrap.rst @@ -10,11 +10,33 @@ The system instance requires that the overlay network configuration be statically configured. The ``bootstrap`` TOML table defines these details for a given cluster. -WARNING: Although ``flux config reload`` works on a live system, these -settings do not take effect until the next broker restart. As such, they -must only be changed in conjunction with a full system instance restart in -order to avoid brokers becoming desynchronized if they are independently -restarted before the next instance restart. +.. warning:: + Although ``flux config reload`` works on a live system, bootstrap + settings do not take effect until the next broker restart. As such, they + must only be changed in conjunction with a full system instance restart in + order to avoid brokers becoming desynchronized if they are independently + restarted before the next instance restart. + +Flux brokers are interconnected in a tree topology. A broker has zero or one +upstream peer (towards the root, rank 0) and zero or more downstream peers +(towards the leaves). A broker passively accepts connections from its +downstream peers on its ZeroMQ `bind` endpoint , and actively connects to its +upstream peer on that broker's ZeroMQ `connect` endpoint, which is the +remote version of the peer's `bind` endpoint. A broker calculates the ranks +of its peers based on the broker's rank, the tree fanout, and the size of the +instance. + +A broker determines its own rank by looking for its hostname in the ``hosts`` +array. The index of the first matching entry is the broker's rank. The +information in the ``hosts`` array also provides + - the `bind` endpoint that the broker configures to accept connections + from downstream peers + - the `connect` endpoint it uses to connect to its parent + +Each point to point connection between brokers is authenticated and encrypted +using ZeroMQ native CURVE cryptography. This requires a shared certificate +to bootstrap. This certificate is stored on disk and must be protected from +access by users other than the Flux instance owner. The ``bootstrap`` table contains the following keys: @@ -51,25 +73,39 @@ default_connect for the current host entry. hosts - (optional) A rank-ordered array of host entries. Each host entry is - a TOML table containing at minimum the ``host`` key. The broker determines - its rank by matching its hostname in the hosts array and taking the array - index. An empty or missing hosts array implies a standalone (single - broker) instance. The entry for a broker with downstream peers must - either assign the ``bind`` key to a ZeroMQ endpoint URI, or the ``default_bind`` - URI described above is used. The entry for a broker with downstream peers - must also either assign the ``connect`` key to a ZeroMQ endpoint URI, or - the ``default_connect`` URI described above is used. The same ``%h`` and ``%p`` - substitutions work here as well. + (optional) A rank-ordered array of host entries. Each entry is a TOML + table containing, at minimum, a ``host`` key and optionally, ``connect`` + and ``bind`` keys. These keys are described in detail below. The array + is required to be defined for instance sizes > 1. +HOSTS ENTRY +=========== + +Each ``hosts`` array entry contains the following keys: + +host + (required) The name of the host that corresponds to this entry. The value + must exactly match the output of the :linux:man1:`hostname` command. + +connect + The ZeroMQ endpoint that downstream peers use to connect to this broker. + The value may contain ``%h`` and ``%p`` tokens as described under + ``default_connect`` above. If the key is omitted, ``default_connect`` + is used. + +bind + The ZeroMQ endpoint that this broker uses to accept connections from + downstream peers. The value may contain ``%h`` and ``%p`` tokens as + described under ``default_bind`` above. If the key is omitted, + ``default_bind`` is used. ZEROMQ ENDPOINTS ================ -In this context, ZeroMQ endpoint URIs normally use the :linux:man7:`zmq_tcp` -transport, consisting of the transport name ``tcp://`` followed by an address. +Brokers use the :linux:man7:`zmq_tcp` transport, consisting of the transport +name ``tcp://`` followed by an address. -Bind addresses specify interface followed by a colon and the TCP port. +`Bind` addresses specify interface followed by a colon and the TCP port. The interface may be one of: - the wild-card ``*`` meaning all available interfaces @@ -80,7 +116,7 @@ The interface may be one of: The port should be an explicit numerical port number. -Connect addresses specify a peer address followed by a colon and the TCP port. +`Connect` addresses specify a peer address followed by a colon and the TCP port. The peer address may be one of: - the DNS name of the peer