Skip to content

Commit

Permalink
Propose Service Bootstrap config design
Browse files Browse the repository at this point in the history 
The patch represents a proposal to find a common pattern for
services bootstrap.

Signed-off-by: Francesco Pantano <[email protected]>
  • Loading branch information
@fmount
fmount committed May 23, 2023
1 parent 1bd5986 commit cdd75e2
Showing 1 changed file with 194 additions and 0 deletions.
194 changes: 194 additions & 0 deletions service_config.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Services configuration / k8s operators

## Goal

Describe an approach that can be adopted across the operators to improve the
deployment aspect and the way secrets and config files are generated. The
solution proposed in this document, in the first place, is to give an operator
the ability to render and inject sensitive snippets within the service config.

### Note:

The document is not supposed to cover the implementation details that might
converge or diverge according to a given operator, the number of deployed
services, and other potential requirements that are not the same across the
board.


## Proposed approach

The use cases are concentrated in two main aspects:

* Provide a common interface that can be used to build additional config
snippets containing sensitive information using k8s `Secrets` instead of
`ConfigMaps`

* Provide a common pattern to build regular service config:
* they will be processed via golang templates and mounted in a Pod from a
`ConfigMap` (which is what is currently happening)
* part of them will be rendered in a `Secret` and avoid exposing sensitive
information related to the deployment aspect


## Basic deployment template

Usually, the basic deployment config information is rendered in a `ConfigMap`.
However, the idea is to store sensitive information in a `Secret` that, during
the bootstrap of the service, will be mounted in the service Pod. For this
reason, a given operator should be responsible to process “on board” the
information coming from the main `osp_secret`, and generate a new `Secret` that
matches to the deployment golang template that will be used by other service
components.

```
+------------------------------------------------------------------------------------+
1. The template is processed by the operator | [database] |
2. A new Secret is created and mounted as a Volume | connection = mysql+pymysql://{{.DBUser}}:{{.DBPassword}}@{{.DBHost}}/{{.Database}} |
by the Pod | |
3. Kolla sync src / dest and applies the expected | [keystone_authtoken] |
ownership/permissions | password = {{.Password}} |
| |
+----------------------------- | (1) |
| | [service_user] |
| | password = {{.Password}} |
| +------------------------------------------------------------------------------------+
|
+----------------------------------------+ +----------------------------------------------------------------------------------------+
| apiVersion: v1 (2) | | “config_files”: [ (3) |
| metadata: | | ... |
| name: 01-<service>-deployment-secret | | { |
| namespace: openstack | ===> | "source": "/var/lib/config-data/deployment/01-deployment-<service>-secret.conf",|
| stringData: | | "dest": "/etc/<service>/<service>.conf.d/01-deployment-<service>-secret.conf", |
| 01-deployment-<service>.conf: | | | "owner": "<service>", |
| <data> | | "perm": "0600" |
+----------------------------------------+ | }, |
+----------------------------------------------------------------------------------------+
```

As the diagram above depicts, a given operator is supposed to implement the
logic to build and reconcile the `Secret` when the template has been processed.
The `Secret`, mounted to the resulting service Pod, in the very last step of
this process is synced by kolla to the destination directory: instead of having
a bash script (e.g. `init.sh`) doing the `chown` on the destination folder,
we'll rely (where possible) on kolla that allows setting ownership and
permissions, as well as copying optional files in case additional configuration
is provided.


## Render multiple secrets: “CustomServiceConfigSecret” interface

Currently the `OpenStack` storage operators expose the
`customConfigServiceSecret` parameter, which provides a mechanism for
specifying service configs via Secrets. Instead of specifying sensitive config
data directly in the `customServiceConfig`, a cloud admin can place sensitive
data in a `Secret`, and reference the secret by name in the service's
`customServiceConfigSecrets` as a list of `Secret` names. Such parameter is
used to select and collect existing k8s Secrets (without decoding any value in
plaintext) and provide them as Volumes/VolumeMounts, that are processed by the
operator and mounted in the target Pod.

```
+----------------------------+
| customServiceConfigSecret: |
+---| - service-secret1 |--------------------
| | - service-secret2 | |
| +----------------------------+ |
| | +-------------+
| | | - snippet 1 |
| +------------------------------------+ | - snippet 2 |
| | apiVersion: v1 | | - snippet 3 |
| | kind: Secret | +-------------+
| | metadata: | |
| | name: service-secret1 | |
| | namespace: openstack | |
| | stringData: | +----------------------------------+
| | snippet1: | | | 04-secret-<service>.conf |
| | [logger_root] | | |
| | level=WARNING | | [logger_root] |
| | handlers=stdout | | level=WARNING |
| | snippet2: | | | handlers=stdout |
| | ################## | | ################## |
| | # Log Formatters # | | # Log Formatters # |
| | ################## | | ################## |
| | [formatter_normal] | | [formatter_normal] |
| | format=(%(name)s): s%(message)s | | format=(%(name)s): s%(message)s |
| +------------------------------------+ | [formatters] |
| | keys=normal |
+-------------------------+ +----------------------------------+
| apiVersion: v1 | |
| kind: Secret | |
| metadata: | |
| name: service-secret2 | |
| namespace: openstack | 1. Mounted as Volume by Service Pod
| stringData: |
| snippet3: | | 2. `kolla_set_configs && kolla_start`: sync
| [formatters] | the resulting secret as done for the other
| keys=normal | regular config files
+-------------------------+
```

As a result of this strategy, the service presents a layout similar to the
following:

```
00-default.conf => default configs generated by operator. This is stored in ConfigMap
01-deployment-secrets.conf => default configs generated by operator, which contains credentials such as [database] connection . This is stored in Secret
02-global.conf => custom configs provided by users via top-level customServiceConfig. This is stored in ConfigMap
03-service.conf => custom configs provided by users via service level customServiceConfig. This is stored in ConfigMap
04-secrets.conf => custom configs provided by users via service level customServiceConfigSecrets, which contains credentials. This is stored in Secret and would not be present if no secrets are provided
```


The service will pass the `--config-dir` parameter to point to the
`<service>.conf.d` directory where all the config files listed above are
rendered.

```
"command": "/usr/bin/<service> --config-dir /etc/<service>/<service>.conf.d"
```

If this strategy is not available, the `init container` that executes a start
script (e.g., `init.sh`) won't be removed, and the logic that generates the
layout mentioned above will be implemented in the init script.

An example of the `customServiceConfigSecret` usage can be found in Manila,
where this parameter has been used to test the
[NetApp backend](https://gist.github.com/gouthampacha/1b5681104ee066b5bd2c702b29376199)


## Conclusion

The model described here allows to reach many goals:
- remove (entirely) the `initContainer` and the related scripts that are no
longer required to start the service: **01-deployment-secret** is generated
by the operator according to the parameters defined in the service CR and the
data retrieved by the initial `osp_secret`;

- `Kolla` is still used to copy files from `src` to `dst`, as they are
rendered and mounted accordingly with the right permissions in the
destination directory (*)

- Operators’ controllers are able to parse many secrets referenced by the
`CustomServiceConfigSecrets` parameter and merge them into a **single**
`Secret` which is passed to the deployment and mounted to the `Pod` in the
target directory (**)

Due the reasons mentioned above, `kolla` is still the target tool used to start
the service.

(\*) Mounting `Secrets` in the same destination directory where the `Configmap`
files are synced currently generates permission related issues, and passing the
`SubPath` to the `VolumeMount` doesn’t solve the problem. Removing Kolla from
the picture doesn’t add much value rather than keeping it

(\*\*) 04-secret-<service>.conf is generated by the operator, and the `data`
field is nothing more than the concatenation of the data retrieved by the list
of the secrets specified in the service CR


## Resources

The work described in this document is supported by the `Glance` patch that has
been tested via the `meta-operator` driven deployment:

* [Glance PoC](https://github.com/openstack-k8s-operators/glance-operator/pull/221)

0 comments on commit cdd75e2

Please sign in to comment.