Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite vault core, issue AppRoles to minions #62684

Merged
merged 104 commits into from
Dec 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
104 commits
Select commit Hold shift + click to select a range
5449abf
Rewrite vault core, orchestrate AppRoles for minions
lkubb Sep 14, 2022
8cc6f2f
Fix linting for rewritten vault integration
lkubb Sep 18, 2022
60bddc6
Add pytest unit tests for utils.vault, fix found issues
lkubb Sep 22, 2022
99260fe
Fix old vault runner tests
lkubb Sep 22, 2022
7182b19
Rewrite vault sdb tests, migrate to pytests
lkubb Sep 22, 2022
7365f6e
Adapt vault ext_pillar tests
lkubb Sep 22, 2022
edf378b
Adapt vault execution module tests, migrate to pytests
lkubb Sep 22, 2022
f09a016
Add more vault execution module unit tests
lkubb Sep 23, 2022
9fa4274
Support python <3.7 (vault util), time-independent tests
lkubb Sep 23, 2022
aa61b88
Add/migrate vault runner unit tests (pytest)
lkubb Sep 26, 2022
e881749
Add vault state module pytests
lkubb Sep 26, 2022
48a2256
Fix tests lint
lkubb Sep 26, 2022
1cfcc30
Refactor Vault container fixture, move to session scope
lkubb Sep 27, 2022
c233158
Fix for existing vault execution/sdb module integration tests
lkubb Sep 27, 2022
32090e0
Improve existing vault runner integration tests
lkubb Sep 28, 2022
bf62a15
Fix vault test support, add list policies
lkubb Sep 28, 2022
e43f53a
Add more functional execution module tests, fix deprecated warning
lkubb Sep 28, 2022
71b764d
Refactor vault pytest support
lkubb Sep 28, 2022
4282caa
Add integration tests, improve/fix caching/issue_params
lkubb Sep 30, 2022
fd9b796
Improve caching behavior, fix tests
lkubb Oct 3, 2022
f0942e1
Allow to autodiscover platform default CA bundle
lkubb Oct 4, 2022
4a28ab3
Remove runner approle param overrides
lkubb Oct 4, 2022
88ba07d
Add clear_cache runner function
lkubb Oct 4, 2022
78db0f1
Also manage token metadata for issued secret IDs
lkubb Oct 4, 2022
4d00606
Cleanup tests
lkubb Oct 4, 2022
b1b6884
Cleanup code, pylint logging suggestions
lkubb Oct 4, 2022
4d6e89f
Do not always invalidate config when verify=default
lkubb Oct 4, 2022
cf16697
Ensure concatted metadata lists are sorted
lkubb Oct 4, 2022
52e07c3
Add changelog (partly)
lkubb Oct 4, 2022
f4d0d50
Merge branch 'master' into approle-minions-vault
Oct 5, 2022
72b647d
Work with legacy peer_run configuration as well
lkubb Oct 5, 2022
8c8d331
Consume a token use regardless of status code
lkubb Oct 6, 2022
07a3586
Correct verify semantics
lkubb Oct 6, 2022
db0706e
Refine token uses handling, add changelog/tests for old issues
lkubb Oct 6, 2022
3f31f1f
Add changelog for main features
lkubb Oct 6, 2022
d3dc2e1
Add test for issue 58580
lkubb Oct 8, 2022
519ee59
Fix vault docs
lkubb Oct 9, 2022
ced497b
Provide all old make_request functionality, add tests
lkubb Oct 9, 2022
363751a
Merge branch 'master' into approle-minions-vault
lkubb Oct 11, 2022
d7bebb2
Merge branch 'master' into approle-minions-vault
lkubb Oct 13, 2022
a7ed73e
Allow token use override, add docstrings to query funcs
lkubb Oct 14, 2022
8f70487
Merge branch 'master' into approle-minions-vault
lkubb Oct 19, 2022
cd2369c
Merge branch 'master' into approle-minions-vault
Oct 20, 2022
cc84901
Simplify config_location merge
lkubb Oct 20, 2022
297e687
Cleanup
lkubb Oct 20, 2022
a7e775a
Merge branch 'master' into approle-minions-vault
lkubb Oct 20, 2022
cb97d83
Merge branch 'master' into approle-minions-vault
lkubb Nov 29, 2022
47fee10
Merge branch 'master' into approle-minions-vault
lkubb Dec 5, 2022
80a22f8
Fix make_request warning
lkubb Dec 5, 2022
4bb7b46
Attempt to fix memory issues during CI test run
lkubb Dec 7, 2022
0e78728
Merge branch 'master' into approle-minions-vault
lkubb Dec 20, 2022
8568e45
Increase documented version
lkubb Dec 20, 2022
78d6bae
Improve lease handling
lkubb Dec 20, 2022
b7e94fd
Refine lease ttl handling/add token lifecycle management
lkubb Jan 3, 2023
c318c32
Merge branch 'master' into approle-minions-vault
lkubb Jan 3, 2023
17fa4e2
Fix docs build
lkubb Jan 3, 2023
ca8322f
Adapt formatting
lkubb Jan 3, 2023
ad35e70
Fix issue param overrides
lkubb Jan 6, 2023
e1a8d42
Introduce session-scoped cache
lkubb Jan 6, 2023
f617a09
Tokens with a single use left are unrenewable
lkubb Jan 6, 2023
88e30de
Allow override of flushing of cached leases during lookup
lkubb Jan 7, 2023
0f68834
Refactor cache classes, save lease data
lkubb Jan 8, 2023
83458fa
Rename session token cache key
lkubb Jan 8, 2023
618a3e8
Add lease management utility
lkubb Jan 9, 2023
95e2fcb
Fix runner integration tests
lkubb Jan 9, 2023
77e2772
Do not overwrite data of cached leases after renewal
lkubb Jan 9, 2023
d6d77b2
Pass token_lifecycle to minions
lkubb Jan 9, 2023
5633841
Do not fail syncing multiple approles/entities with pillar templates
lkubb Jan 11, 2023
55abe4e
Ensure config cache expiration can be disabled
lkubb Jan 11, 2023
2af084d
Merge branch 'master' into approle-minions-vault
lkubb Apr 12, 2023
e2f908c
Rename changelog files (.md)
lkubb Apr 12, 2023
1d37cc5
Declare vaultpolicylexer as parallel read safe
lkubb Apr 12, 2023
008d730
Correct meta[data] payload key
lkubb Apr 12, 2023
9f58c41
Reuse TCP connection
lkubb Apr 14, 2023
7fe512a
Refactor utils module
lkubb Apr 14, 2023
520ef97
Ensure client is recreated after clearing cache
lkubb Apr 14, 2023
dc0b617
Always use unwrap_client config as expected server
lkubb Apr 17, 2023
788a07a
Ensure client is recreated after clearing cache 2
lkubb Apr 17, 2023
6e4da71
Simulate patch for KV v1 or missing `patch` capability
lkubb Apr 17, 2023
4f2e13c
Add `patch` option to Vault SDB driver
lkubb Apr 17, 2023
69fef51
Reduce lease validity when revocation fails
lkubb Apr 18, 2023
60f7286
Extract AppRole/Identity API from runner into utils
lkubb Apr 18, 2023
108a95c
Revoke tokens, fire events, improve cache/exception handling
lkubb Apr 20, 2023
439b330
Allow updating cached config w/o closing session
lkubb Apr 23, 2023
35b0e1a
Homogenize funcs, update docs, cleanup
lkubb Apr 23, 2023
a86cfc7
Merge branch 'master' into approle-minions-vault
lkubb Apr 23, 2023
f5943ea
Minor internal fixes
lkubb Apr 24, 2023
5bb2de9
Merge branch 'master' into approle-minions-vault
lkubb May 2, 2023
15944a6
Add release note
lkubb May 2, 2023
5a302dc
Address review remarks
lkubb May 2, 2023
0310a82
Merge branch 'master' into approle-minions-vault
lkubb May 11, 2023
32f9773
Fix release notes
lkubb May 11, 2023
0b77146
Remove loading minion_mods from factory
lkubb May 29, 2023
e20c496
Address other review remarks
lkubb May 29, 2023
4ee39da
Add inline specification of trusted CA root cert
lkubb May 29, 2023
60bd374
Small QoL additions
lkubb May 29, 2023
753417d
Merge branch 'master' into approle-minions-vault
lkubb May 29, 2023
101c65d
Fix lint
lkubb May 29, 2023
90750ce
Merge branch 'master' into approle-minions-vault
lkubb Jun 11, 2023
e940419
Fix lint for Python >=3.8 support
lkubb Jun 11, 2023
0182a00
Merge branch 'master' into approle-minions-vault
lkubb Dec 12, 2023
e1903de
Add missing fixes
lkubb Dec 12, 2023
f6f397c
Fix unit tests
lkubb Dec 12, 2023
a0a7fce
Merge branch 'master' into approle-minions-vault
lkubb Dec 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog/51986.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed Salt master does not renew token
1 change: 1 addition & 0 deletions changelog/57561.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed vault module fetching more than one secret in one run with single-use tokens
1 change: 1 addition & 0 deletions changelog/58174.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed Vault verify option to work on minions when only specified in master config
1 change: 1 addition & 0 deletions changelog/58580.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed vault command errors configured locally
1 change: 1 addition & 0 deletions changelog/60779.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed sdb.get_or_set_hash with Vault single-use tokens
1 change: 1 addition & 0 deletions changelog/62380.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Fixed Vault session storage to allow unlimited use tokens
1 change: 1 addition & 0 deletions changelog/62823.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added Vault AppRole and identity issuance to minions
1 change: 1 addition & 0 deletions changelog/62825.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added Vault AppRole auth mount path configuration option
1 change: 1 addition & 0 deletions changelog/62828.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added distribution of Vault authentication details via response wrapping
1 change: 1 addition & 0 deletions changelog/63406.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added Vault token lifecycle management
1 change: 1 addition & 0 deletions changelog/63440.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added Vault lease management utility
1 change: 1 addition & 0 deletions changelog/64096.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added patch option to Vault SDB driver
1 change: 1 addition & 0 deletions changelog/64379.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Added inline specification of trusted CA root certificate for Vault
26 changes: 26 additions & 0 deletions doc/_ext/vaultpolicylexer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
from pygments.lexer import bygroups, inherit
from pygments.lexers.configs import TerraformLexer
from pygments.token import Keyword, Name, Punctuation, Whitespace


class VaultPolicyLexer(TerraformLexer):
aliases = ["vaultpolicy"]
filenames = ["*.hcl"]
mimetypes = ["application/x-hcl-policy"]

tokens = {
"basic": [
inherit,
(
r"(path)(\s+)(\".*\")(\s+)(\{)",
bygroups(
Keyword.Reserved, Whitespace, Name.Variable, Whitespace, Punctuation
),
),
],
}


def setup(app):
app.add_lexer("vaultpolicy", VaultPolicyLexer)
return {"parallel_read_safe": True}
1 change: 1 addition & 0 deletions doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@
"saltrepo",
"myst_parser",
"sphinxcontrib.spelling",
"vaultpolicylexer",
#'saltautodoc', # Must be AFTER autodoc
]

Expand Down
48 changes: 48 additions & 0 deletions doc/topics/releases/templates/3007.0.md.template
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,54 @@ A new ``package`` grain was added in 3007.0 This detects how Salt was installed
the directory. If you are building packages of Salt you need to ensure this file is set to the correct package type
that you are building. The options are ``pip``, ``onedir``, or ``system``. By default this file is already set to ``pip``.

## Improved Vault integration
This release features a much deeper integration with HashiCorp Vault, for which
many parts of the implementation core were improved. Among other things, the Salt
daemons now attempt to renew/revoke their access tokens and can manage long-lived leases,
while the Salt master now distributes authentication secrets using response wrapping.
An important new feature concerns the way Vault policies can be managed.

In versions before 3006, the Salt master only issued tokens to minions, whose policies
could be templated with the minion ID and (insecure) grain values.
3006 introduced secure templating of those policies with pillar values, as well as
templating of Vault external pillar paths with pillar values. These improvements reduced the
overhead of managing Vault policies securely.

In addition, the Salt master can now be configured to issue AppRoles
to minions and manage their metadata using a similar templating approach.
Since this metadata can be taken advantage of in templated policies on the Vault side,
the need for many boilerplate policies is reduced even further:
{%- raw %}

```vaultpolicy
path "salt/data/minions/{{identity.entity.metadata.minion-id}}" {
capabilities = ["create", "read", "write", "delete", "patch"]
}

path "salt/data/roles/{{identity.entity.metadata.role}}" {
capabilities = ["read"]
}
```
{%- endraw %}

Although existing configurations will keep working without intervention after upgrading
the Salt master, it is strongly recommended to adjust the `peer_run` configuration to
include the new issuance endpoints in order to avoid unnecessary overhead:

```yaml
peer_run:
.*:
- vault.get_config
- vault.generate_new_token
```

Please see the [Vault execution module docs](https://docs.saltproject.io/en/3007.0/ref/modules/all/salt.modules.vault.html) for
details and setup instructions regarding AppRole issuance.

.. note::
The Vault modules are being moved to a [Salt extension](https://github.com/salt-extensions/saltext-vault), but this improvement
has still been merged into core for a smoother transition.

<!--
Do not edit the changelog below.
This is auto generated
Expand Down
Loading
Loading