This is a back end function for Hiera 5 that allows lookup to be sourced from HTTP queries. The intent is to make this backend adaptable to allow you to query any data stored in systems with a RESTful API such as CouchDB or even a custom store with a web front-end
- The 3.x series of hiera-http is only compatible with Hiera 5, that ships with Puppet 4.9+
- Users looking for older implementations for Hiera 1,2 and 3 should use hiera-http 2.x
- hiera-http 3 ships within the crayfishx/hiera_http
- hiera-http 2 (legacy) ships as a Rubygem "hiera-http"
The lookup_http gem must be installed and loadable from Puppet
# /opt/puppetlabs/puppet/bin/gem install lookup_http
# puppetserver gem install lookup_http
The data provider is available by installing the crayfishx/hiera_http module into your environment.
# puppet module install crayfishx/hiera_http
See The official Puppet documentation for more details on configuring Hiera 5.
The following is an example Hiera 5 hiera.yaml configuration for use with hiera-http
---
version: 5
hierarchy:
- name: "Hiera-HTTP lookup"
lookup_key: hiera_http
uris:
- http://localhost:5984/host/%{trusted.certname}
- http://localhost:5984/dc/%{facts.location}
- http://localhost:5984/role/%{facts.role}
options:
output: json
ignore_404: trueThe following mandatory Hiera 5 options must be set for each level of the hierarchy.
name: A human readable name for the lookup
lookup_key: This option must be set to hiera_http
uris or uri: An array of URI's passed to uris or a single URI passed to uri. This option supports interpolating special tags, see below.
The following are optional configuration parameters supported in the options hash of the Hiera 5 config
output: : Specify what handler to use for the output of the request. Currently supported outputs are plain, which will just return the whole document, or YAML and JSON which parse the data and try to look up the key
http_connect_timeout: : Timeout in seconds for the HTTP connect (default 10)
http_read_timeout: : Timeout in seconds for waiting for a HTTP response (default 10)
confine_to_keys: : Only use this backend if the key matches one of the regexes in the array
confine_to_keys:
- "application.*"
- "apache::.*"
failure: : When set to graceful will stop hiera-http from throwing an exception in the event of a connection error, timeout or invalid HTTP response and move on. Without this option set hiera-http will throw an exception in such circumstances
ignore_404: : If failure is not set to graceful then any error code received from the HTTP response will throw an exception. This option makes 404 responses exempt from exceptions. This is useful if you expect to get 404's for data items not in a certain part of the hierarchy and need to fall back to the next level in the hierarchy, but you still want to bomb out on other errors.
dig: : (true or false) When the output is parsed YAML or JSON, whether or not to dig into the hash and return the value defined by the dig_key option below. This option defaults to true
dig_key : When the dig option is true (default), this option specifies what key is looked up from the results hash returned by the HTTP endpoint. See Digging values below for more information
use_ssl:: When set to true, enable SSL (default: false)
ssl_ca_cert: Specify a CA cert for use with SSL
ssl_cert: Specify location of SSL certificate
ssl_key: Specify location of SSL key
ssl_verify: Specify whether to verify SSL certificates (default: true)
use_auth:: When set to true, enable basic auth (default: false)
auth_user:: The user for basic auth
auth_pass:: The password for basic auth
headers:: Hash of headers to send in the request
Previous versions of this backed allowed the use of variables such as %{key} and %{calling_module} to be used in the URL, this has changed with Hiera 5. To allow for similar behaviour you can use a number of tags surrounded by __ to interpolate special variables derived from the key into the uri or uris option in hiera.yaml. Currently you can interpolate __KEY__, __MODULE__, __CLASS__ and __PARAMETER__, these tags are derived from parsing the original lookup key.
In the case of a lookup key matching foo::bar::tango the following tags are available;
__KEY__: The original lookup key unchanched;foo::bar::tango__MODULE__: The first part of the lookup key;foo__CLASS__: All but the last parts of the lookup key;foo::bar__PARAMETER__: The last part of they key representing the class parameter;tango
Example using this backend to interact with the Puppet Enterprise Jenkins Pipeline plugin
---
version: 5
defaults:
datadir: hieradata
data_hash: yaml_data
hierarchy:
- name: 'Jenkins data source'
lookup_key: hiera_http
uris:
- "http://jenkins.example.com:8080/hiera/lookup?scope=%{trusted.certname}&key=__KEY__"
- "http://jenkins.example.com:8080/hiera/lookup?scope=%{environment}&key=__KEY__"
options:
output: json
failure: gracefulHiera-HTTP supports options to automatically dig into the returned data structure to find a corresponding key. Puppet lookup itself supports similar dig functionality but being able to specify it at the backend means that where an API wraps the required data up in a different way, we can always lookup the desired value before passing it to Puppet to ensure that class parameter lookups work without having to hard code the lookup function and dig down into the data for each request. The dig functionality in Puppet is intended to enable you to parse your data more effectivly, the dig functionality in hiera-http is intended to make the API of the endpoint you are talking to compatible.
By default, when a hash is returned by the HTTP endpoint (eg: JSON) then hiera-http will attempt to lookup the key corresponding with the lookup key. For example, when looking up a key apache::port we would expect the HTTP endpoint to return something like;
{
"apache::port": 80
}Returned value would be 80
Depending on what HTTP endpoint we are hitting, the returned output may contain other data with the key that we want to look up nested below it. This behaviour can be overriden by using the options dig and dig_key.
The dig_key option can be used to change the key that is looked up, it also supports a dot-notation for digging values in nested hashes. Special tags can also be used in the dig_key option. Consider the following example output from our HTTP endpoint;
{
"document": {
"settings": {
"apache::port": 80
}
}
}In this scenario we wouldn't be able to use class parameter lookups out-of-the-box, even if we just returned the whole structure, because we always need to drill down into document.settings to get the correct value, so In order to map the lookup to find the correct value, we can interpolate the KEY tag into lookup_key and tell hiera-http to always dig into the hash with the following option;
options:
dig_key: document.settings.__KEY__A more complicated example;
{
"document": {
"settings": {
"apache": {
"port": 80
}
}
}
}Can be looked up with;
options:
dig_key: document.settings.__MODULE__.__PARAMETER__
In both examples, the returned value to Puppet will be 80
The dig option can be used to disable digging altogether and the entire data hash will be returned with no attempt to resolve a key
- Craig Dunn [email protected]
- @crayfishX
- IRC (freenode) crayfishx
- http://www.craigdunn.org