Improve Vault templating (policies, ext_pillar paths)

[lkubb]

What does this PR do?

1. Allows to securely template Vault policies with pillar values
2. Allows to template Vault ext_pillar paths with pillar values from previously rendered pillars

I am aware of the several attempts to introduce the behavior found in (1). The way it is implemented in this PR should prevent the mentioned issues. For reference:
#43288
#49343
#48829

This builds on the ext_pillar fix found in #62653.

What issues does this PR fix or reference?

Fixes: #43287
Reference: #61821

Previous Behavior

Assigned policies had to be templated using insecure grains or could only use the minion ID. You would have to live with insecure policy assignment or create a policy for each minion.

The pillar module only allowed to template paths with the minion ID. This necessitated a separate secret path for each minion, causing secret redundancy, or all minions to have access to all secrets defined in the static ext_pillar path.

New Behavior

Roles can be securely assigned in the pillar and used to template Vault policies. These pillar-sourced roles can also be used to template Vault ext_pillar paths, reducing redundancy/increasing security. It is sufficient to create a policy for each role.

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

• Docs
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

Yes

Notes

The improvements in this PR are extended by #62684. It has a similar idea behind it, but represents a major rewrite to simplify further Vault integration. Noticeably, it allows to issue AppRoles and entities to minions and template their metadata using pillar values. This allows to push the ACL policy templating to the Vault side, reducing the amount of policy overhead further.

[github-actions]

Hi! I'm your friendly PR bot!

You might be wondering what I'm doing commenting here on your PR.

Yes, as a matter of fact, I am...

I'm just here to help us improve the documentation. I can't respond to
questions or anything, but what I can do, I do well!

Okay... so what do you do?

I detect modules that are missing docstrings or "CLI Example" on existing docstrings!
When I was created we had a lot of these. The documentation for these
modules need some love and attention to make Salt better for our users.

So what does that have to do with my PR?

I noticed that in this PR there are some files changed that have some of these
issues. So I'm leaving this comment to let you know your options.

Okay, what are they?

Well, my favorite, is that since you were making changes here I'm hoping that
you would be the most familiar with this module and be able to add some other
examples or fix any of the reported issues.

If I can, then what?

Well, you can either add them to this PR or add them to another PR. Either way is fine!

Well... what if I can't, or don't want to?

That's also fine! We appreciate all contributions to the Salt Project. If you
can't add those other examples, either because you're too busy, or unfamiliar,
or you just aren't interested, we still appreciate the contributions that
you've made already.

Whatever approach you decide to take, just drop a comment here letting us know!

Detected Issues (click me)

Check Known Missing Docstrings...........................................Failed
- hook id: invoke
- exit code: 1
/home/runner/.cache/pre-commit/repon79d_ti0/py_env-python3/lib/python3.9/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.

warnings.warn("Setuptools is replacing distutils.")

The function 'read_secret' on 'salt/modules/vault.py' does not have a 'CLI Example:' in its docstring

Found 1 errors

---

Thanks again!

[github-actions]

Hi! I'm your friendly PR bot!

You might be wondering what I'm doing commenting here on your PR.

Yes, as a matter of fact, I am...

I'm just here to help us improve the documentation. I can't respond to
questions or anything, but what I can do, I do well!

Okay... so what do you do?

I detect modules that are missing docstrings or "CLI Example" on existing docstrings!
When I was created we had a lot of these. The documentation for these
modules need some love and attention to make Salt better for our users.

So what does that have to do with my PR?

I noticed that in this PR there are some files changed that have some of these
issues. So I'm leaving this comment to let you know your options.

Okay, what are they?

Well, my favorite, is that since you were making changes here I'm hoping that
you would be the most familiar with this module and be able to add some other
examples or fix any of the reported issues.

If I can, then what?

Well, you can either add them to this PR or add them to another PR. Either way is fine!

Well... what if I can't, or don't want to?

That's also fine! We appreciate all contributions to the Salt Project. If you
can't add those other examples, either because you're too busy, or unfamiliar,
or you just aren't interested, we still appreciate the contributions that
you've made already.

Whatever approach you decide to take, just drop a comment here letting us know!

Detected Issues (click me)

Check Known Missing Docstrings...........................................Failed
- hook id: invoke
- exit code: 1
/home/runner/.cache/pre-commit/repon79d_ti0/py_env-python3/lib/python3.9/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.

warnings.warn("Setuptools is replacing distutils.")

The function 'read_secret' on 'salt/modules/vault.py' does not have a 'CLI Example:' in its docstring

Found 1 errors

---

Thanks again!

[lkubb]

@Ch3LL Any chance of getting this merged soon? That would settle all the requisites for #62684, which will need a final merge conflict resolving from my side after this.

#62889 seems OK with this PR going forward as well.