Run all builtins as plugins #5536

tyrannosaurus-becks · 2018-10-17T20:33:09Z

This PR is to allow Vault to run all builtins as plugins. This PR is ready for review.

Overview

To understand how this is helpful, consider this scenario:

You are running the aws auth builtin
You wish to run a custom version while still otherwise remaining on the current version of Vault

To achieve this given this code, you would:

Customize the aws builtin
Run $ go build on main.go in its directory to make a binary
Place this binary in your configured plugin directory
Add it to the plugin catalog as per normal
Run $ auth enable aws
Vault would then give precedence to enabling your custom plugin over the builtin version of the aws plugin

This hopefully illustrates the new behavior this PR introduces:

All builtins are plugins
Plugins in the plugin directory are preferred over builtin versions

Because of possible naming clashes across plugins, we had to introduce the concept of a plugin type. This prevents the aws secrets plugin's name from clashing with the aws auth plugin's name.

Other side effects of these actions include:

The plugin catalog used to be a LIST /sys/plugins/catalog which is DEPRECATED by this PR
The same functionality, by plugin type, will now be at /sys/plugins/catalog/database, /sys/plugins/catalog/auth, and /sys/plugins/catalog/secret
The plugin catalog now has READ at /sys/plugins/catalog which, under "data", returns a map of plugin type to the names of the plugins that fall into that type
The syntax of $ vault secrets enable -plugin-name='ad' plugin will now be DEPRECATED (though still supported) and now all plugins will be enabled via $ vault secrets enable ad

TODO

I will create a follow-up PR immediately after this is merged updating all docs to correspond with these changes

vault/auth.go

builtin/credential/app-id/backend_test.go

builtin/logical/database/dbplugin/plugin.go

calvn · 2018-10-30T19:18:36Z

builtin/plugin/backend.go

-// backend is a thin wrapper around plugin.BackendPluginClient
-type backend struct {
+// PluginBackend is a thin wrapper around plugin.BackendPluginClient
+type PluginBackend struct {


Does this have to be exported? The func Backend factory func above returns an interface so having this as backend seems to be fine (and follows the pattern on how other backends are laid out, e.g. credentials/aws having its own internal type backend struct).

This is exported so it can be used in the plugin_test package residing in the same directory, here.

The reason that was changed to its own package was to resolve an extremely persistent import cycle. This is the import cycle that results if I revert it:

$ make test ==> Checking that build is using go version >= 1.10... ==> Using go version 1.11... # github.com/hashicorp/vault/builtin/plugin import cycle not allowed in test package github.com/hashicorp/vault/builtin/plugin (test) imports github.com/hashicorp/vault/http imports github.com/hashicorp/vault/vault imports github.com/hashicorp/vault/builtin/plugin FAIL github.com/hashicorp/vault/builtin/plugin [setup failed]

I experimented with multiple ways of breaking this import cycle, ultimately selecting this one because it was the least code change.

I'm open to taking other approaches, or adding a comment to that effect.

calvn · 2018-10-30T19:19:57Z

builtin/plugin/backend_test.go

@@ -1,4 +1,4 @@
-package plugin
+package plugin_test


Any reason why this has to be broken out to its own package?

Replied to this one in the above comment.

helper/consts/plugin_types.go

command/plugin.go

briankassouf

I think we need to think through some of the upgrade cases still, i worry that existing plugins mounts wont upgrade correctly. Also i think the catalog's storage needs to be broken out by type.

command/auth_enable.go

briankassouf · 2018-10-31T16:36:13Z

command/base_predict.go

+	i := 0
+	for pluginName := range pluginNames {
+		plugins[i] = pluginName
+		i++


Feels like the dedup should be happening in the API

I agree - and the sorting of the list too. So, I added this commit and what that does is re-add the ability to list all plugins at sys/plugins/catalog. At that endpoint, it sorts and dedupes them.

There's a catch, though. I can't move all the sorting and deduping to the API because when I re-add path prediction for the secret mounts, I need to predict all database and secrets plugins. So there need to be two API calls - one listing each type, and then some logic sorting and deduping them.

command/commands.go

command/secrets_enable.go

vault/auth.go

vault/logical_system.go

vault/mount.go

vault/plugin_catalog.go

briankassouf · 2018-10-31T18:00:48Z

vault/plugin_reload.go

@@ -77,8 +75,7 @@ func (c *Core) reloadMatchingPlugin(ctx context.Context, pluginName string) erro
 		if ns.ID != entry.Namespace().ID {
 			continue
 		}
-
-		if entry.Config.PluginName == pluginName && entry.Type == "plugin" {
+		if entry.Type == pluginName {


What does entry.Type equal now?

In the command, $ vault auth enable my-custom-plugin, the entry.Type == "my-custom-plugin".

command/plugin_info.go

vault/logical_system.go

Co-Authored-By: tyrannosaurus-becks <[email protected]>

briankassouf · 2018-11-06T21:26:30Z

command/commands.go

+		"database": logicalDb.Factory,
+		// This is also available in the plugin catalog, but is here due to the need to
+		// automatically mount it.
+		"kv": logicalKv.Factory,


Was this necessary to add?

* Determine plugin type if using the old untyped API * remove logger * add comment * Consolidate plugin paths

* Reuse the same path spec * Re-arrange paths to proper order

This reverts commit f6d2ee9.

Try list if get fails

tyrannosaurus-becks requested a review from briankassouf October 18, 2018 01:33

tyrannosaurus-becks changed the title ~~Early reviews: Run all builtins as plugins~~ Run all builtins as plugins Oct 18, 2018

tyrannosaurus-becks commented Oct 18, 2018

View reviewed changes

vault/auth.go Outdated Show resolved Hide resolved

briankassouf added this to the 0.12 milestone Oct 19, 2018

run all builtins as plugins

97c24d7

tyrannosaurus-becks force-pushed the run-all-buitins-as-plugins branch from 74a052f to 97c24d7 Compare October 23, 2018 16:35

merge master

8cd2d92

tyrannosaurus-becks changed the title ~~Run all builtins as plugins~~ Not Ready: Run all builtins as plugins Oct 23, 2018

fix tests

7ac1b95

tyrannosaurus-becks changed the title ~~Not Ready: Run all builtins as plugins~~ Run all builtins as plugins Oct 23, 2018

Becca Petrin added 4 commits October 23, 2018 14:58

Merge branch 'opensource-master' into run-all-buitins-as-plugins

fc3343a

Merge branch 'opensource-master' into run-all-buitins-as-plugins

fd9a1b8

strip extra space from import

4c66440

fix some tests

e8f4954

tyrannosaurus-becks mentioned this pull request Oct 26, 2018

vault_auth_backend: Add plugin parameter hashicorp/terraform-provider-vault#213

Closed

calvn reviewed Oct 30, 2018

View reviewed changes

builtin/credential/app-id/backend_test.go Outdated Show resolved Hide resolved

calvn reviewed Oct 30, 2018

View reviewed changes

builtin/logical/database/dbplugin/plugin.go Outdated Show resolved Hide resolved

calvn reviewed Oct 30, 2018

View reviewed changes

Becca Petrin added 3 commits October 30, 2018 14:12

remove unnecessary plugin type db argument

887eb24

Merge branch 'opensource-master' into run-all-buitins-as-plugins

d8fe340

Merge branch 'opensource-master' into run-all-buitins-as-plugins

ad70d44

kalafut reviewed Oct 30, 2018

View reviewed changes

helper/consts/plugin_types.go Show resolved Hide resolved

strip field for auth plugin requests

c258561

chrishoffman reviewed Oct 31, 2018

View reviewed changes

command/plugin.go Outdated Show resolved Hide resolved

command/plugin.go Outdated Show resolved Hide resolved

briankassouf reviewed Oct 31, 2018

View reviewed changes

Becca Petrin added 3 commits October 31, 2018 11:46

add backend type to the token store

6ec7fd4

support plugin-name for auth enablement

376039f

support plugin-name for secrets enablement

1534d44

calvn reviewed Oct 31, 2018

View reviewed changes

command/plugin_info.go Outdated Show resolved Hide resolved

Becca Petrin added 4 commits November 6, 2018 09:01

change PluginType const to uint32

6d57143

re-add plugin_name to logical_system_paths

363abe3

fix plugin list command

a9d491f

merge master

8aaf887

briankassouf reviewed Nov 6, 2018

View reviewed changes

vault/logical_system.go Outdated Show resolved Hide resolved

briankassouf reviewed Nov 6, 2018

View reviewed changes

vault/logical_system.go Outdated Show resolved Hide resolved

briankassouf reviewed Nov 6, 2018

View reviewed changes

vault/logical_system.go Outdated Show resolved Hide resolved

briankassouf reviewed Nov 6, 2018

View reviewed changes

vault/logical_system.go Outdated Show resolved Hide resolved

briankassouf and others added 4 commits November 6, 2018 10:21

update config.PluginNameDeprecated to logicalType

1a8e3f8

Co-Authored-By: tyrannosaurus-becks <[email protected]>

update config.PluginNameDeprecated to logicalType

ef5e51f

Co-Authored-By: tyrannosaurus-becks <[email protected]>

update config.PluginNameDeprecated to logicalType

0e093ba

Co-Authored-By: tyrannosaurus-becks <[email protected]>

update config.PluginNameDeprecated to logicalType

a446776

Co-Authored-By: tyrannosaurus-becks <[email protected]>

briankassouf reviewed Nov 6, 2018

View reviewed changes

Becca Petrin and others added 8 commits November 6, 2018 14:35

automatically sort plugins from dev flag

0931816

merge master

aab2e8c

Determine plugin type if using the old untyped API (#5713)

ce8cb57

* Determine plugin type if using the old untyped API * remove logger * add comment * Consolidate plugin paths

Reuse the same path spec (#5715)

8d32f46

* Reuse the same path spec * Re-arrange paths to proper order

support listing at parent plugin catalog

f6d2ee9

Revert "support listing at parent plugin catalog"

442b667

This reverts commit f6d2ee9.

try listing if getting fails

ba57fa0

Merge pull request #5717 from hashicorp/backwards-compat

5b6d88c

Try list if get fails

briankassouf approved these changes Nov 7, 2018

View reviewed changes

tyrannosaurus-becks merged commit fb89af7 into master Nov 7, 2018

tyrannosaurus-becks deleted the run-all-buitins-as-plugins branch November 7, 2018 01:21

tyrannosaurus-becks mentioned this pull request Nov 7, 2018

Update docs to match running builtins as plugins #5727

Merged

catsby mentioned this pull request Nov 15, 2018

Remove Plugin from detailed output #5797

Closed

This was referenced Aug 1, 2019

Determine from the API if an Auth Backend is a plugin #7229

Closed

Support enabling auth plugins with vault_auth_backend hashicorp/terraform-provider-vault#54

Closed

cpick mentioned this pull request Dec 7, 2019

Vault provider does not support mount for plugins hashicorp/terraform-provider-vault#623

Open

tomhjp mentioned this pull request Nov 24, 2023

cli: Improve error handling for plugin commands #24250

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run all builtins as plugins #5536

Run all builtins as plugins #5536

tyrannosaurus-becks commented Oct 17, 2018 •

edited

Loading

calvn Oct 30, 2018

tyrannosaurus-becks Oct 30, 2018 •

edited

Loading

calvn Oct 30, 2018

tyrannosaurus-becks Oct 30, 2018

briankassouf left a comment

briankassouf Oct 31, 2018

tyrannosaurus-becks Oct 31, 2018

briankassouf Oct 31, 2018

tyrannosaurus-becks Oct 31, 2018

briankassouf Nov 6, 2018

Run all builtins as plugins #5536

Run all builtins as plugins #5536

Conversation

tyrannosaurus-becks commented Oct 17, 2018 • edited Loading

Choose a reason for hiding this comment

tyrannosaurus-becks Oct 30, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

briankassouf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tyrannosaurus-becks commented Oct 17, 2018 •

edited

Loading

tyrannosaurus-becks Oct 30, 2018 •

edited

Loading