add kep for kustomize secret generator plugins

[sethpollack]

No description provided.

[sethpollack]

/assign @monopole

[monopole]

Thanks Seth!

[monopole]

Suggest say what the goal is, and its probably not useful to assume reader has history, e.g.

kustomize wants to generate Secret objects (link to api?), which are general key-value pairs where the value is supposed to be a secret. Reading secret values from local files raises security concerns about file lifetime and access. Reading secret values from the execution of arbitrary commands in a kustomization file introduces concerns in a world where kustomization files can be used directly from the internet. This proposal describes obtaining secret values from specific kustomize plugins...

[monopole]

Please add an example of what the new secretGenerator stanza would look like in https://github.com/kubernetes-sigs/kustomize/blob/master/docs/kustomization.yaml

[monopole]

link to golang plugins, and maybe a sentence or two as to why they are different from any old executable

[mattfarina]

From the go plugins doc:

Currently plugins are only supported on Linux and macOS.

Shouldn't the solution work on Windows since kustomize ships for Windows?

[monopole]

kubernetes-sigs/kustomize#791

[monopole]

How would you frame an attack? If it sounds implausible, all the better.

[monopole]

Not sure what to put here since we cannot get user data at this point - other than noting the number of people who chimed in on kubernetes-sigs/kustomize#692

• A higher level feature test (like those in pkg/target)
• Documentation of fields in the canonical example file
• Usage example

[monopole]

PS, just flesh out those points, and we can immediately pull as provisional.

Wait a few days for comments?

Then go to implementable (not just the code - need examples and tests too).

[sethpollack]

Ok awesome, I'll get to work on this. Thanks for the feedback!

[sethpollack]

@monopole Ok updated. Let me know if it needs anymore changes?

[monopole]

@seans3 pingaroo

[mattfarina]

Has this been discussed in SIG CLI? If not, I would suggest discussing it there as a next step.

[mattfarina]

Can you add the related SIGs section to the metadata? Some are listed as labels on the PR. Also, can you add SIG Apps.

[monopole]

Has this been discussed in SIG CLI? If not, I would suggest discussing it there as a next step.

Done on Feb 13th

[mattfarina]

Should the approvers be the kustomize admin and/or maintainers? https://github.com/kubernetes-sigs/kustomize/blob/master/OWNERS_ALIASES

[calebamiles]

to answer a question from @lavalamp, it is my intention that approvers and reviewers are the people who did not the people who can which is the problem that the OWNERS mechanism addresses

[lavalamp]

@calebamiles Then the fields should be called "reviewed-by" and "approved-by"? And it is weird to have those people added by the author rather than add themselves. I would not be surprised if we have many KEPs created with a different understanding of those fields.

[mattfarina]

From the go plugins doc:

Currently plugins are only supported on Linux and macOS.

Shouldn't the solution work on Windows since kustomize ships for Windows?

[mattfarina]

If kustomize is going to add plugins should it try the same model as kubectl for management?

[monopole]

What do you think?

[monopole]

Some options for leveraging kubectl plugin model (discussed during/after sig-cli meeting):

• Require installation of kubectl, and have kustomize execute the secret generation code as a kubectl sub-process, i.e. exec.Command(“kubectl”, “foo”, …) using the plugin via kubectl.

• Arrange via cli-runtime refactoring to vendor kubectl plugin code into kustomize, to track changes to argument and flag parsing and the notion of plugin subcommands.

• Vendor no code, and use a simplified kubctl plugin model: given a secret generation command foo bar, just look for a binary called kubectl-foo and run it, passing bar as an argument. This works because the kustomize use case doesn’t have to worry about command collisions, sub-commands, listing plugins, etc.

The downside to all these options is that a plugin intended only for kustomize secret generation would appear in kubectl help, and plugin lists, etc.

[monopole]

The above comments assume familiarity with how kubectl plugins work. if interested, see https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins

[sethpollack]

I like option 3 the best.

[anguslees]

I think there are risks of command collisions here. Not right now, because nobody uses kubectl plugins ;) but imagine a world where it is normal to have several unknown kubectl-something commands in $PATH. It is highly likely that these plugins have interesting side effects when invoked, because that's the whole point of plugins.

Given that kustomization.yaml files will come from relatively-untrusted external sources, I would much prefer that this specific category of plugin was cleanly namespaced off from other possible kubectl plugins. This should be as simple as invoking kubectl-kustomize-plugin-foo ..., rather than just kubectl-foo.

[pohly]

Do those extensions have to be in $PATH? If users are not meant to invoke them directly, then they shouldn't be.

[monopole]

@anguslees I don't want to use kubectl style plugins. Was just listing the options as a thought experiment. Your point taken though; if we did, we could avoid collisions in kubectl with more ugly namespace. When I said the kustomize use case doesn't have to worry about command collisions, i was referring to colliding with other native kustomize commands, which isn't possible since the kustomize plugin wouldn't be used as a cobra-style command. The only trouble would be plugin-plugin collision, and we just error out if the loader detects that.

I'd never use a kustomize.yaml file from an untrusted source. Is there something in the docs that suggests such behavior is recommended?

[anguslees]

I'd never use a kustomize.yaml file from an untrusted source.

I think the larger kustomize proposal is to have 3rd parties publish remote manifest repositories that may include kustomize.yaml (since they are in turn derived from manifests coming from another 4th party). That means we are pulling kustomize.yaml from a sort-of-untrusted source.

I agree that actually applying the remote manifests necessarily means we are trusting the remote source to give us code/config that will be executed in our local cluster. There are differences between what (and when) the remote source is able to execute in the cluster vs on the local machine, however. In particular, we need to be able to do "--dry-run" -style kubectl operations on remote yaml without endangering the local user.

(See also: the reasons the previous arbitrary-command-based kustomize secret generator was rolled back)

[mattfarina]

Can I suggest you look at the XDG spec for directory naming? https://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

[monopole]

In that spirit, would be fine with ~/.config/kustomization_plugins

[mattfarina]

If I remember correctly, plugins also have the issue where the path needs to be the same, from the root of the filesystem, for them across systems and developers. This may be fixed but it has stopped others from using them in the past. Has this been looked at?

This is a developer experience risk and I'm curious if it's no longer an issue or if there is a mitigation.

[monopole]

do you have a reference?

[mattfarina]

It appears that windows support for go plugins is unplanned (issue here).

Helm plugins might provide a style worth looking at. Basic plugins are similar to what kubectl does. There are downloader plugins that can hook into some situations. Docs are at https://docs.helm.sh/using_helm/#downloader-plugins. Just another idea to consider. Happy to share more if anyone is interested.

[monopole]

yes, that's a general model. we looked at it back when thinking about kubectl plugins v1.
the use case here is far more constrained (just generate a secret). we can start with Golang plugins, and allow different kinds later as the need arises.

[monopole]

suggest goplugins

i.e. tie the keyword to golang shared object file plugins, leaving the door open to other kinds of plugins later.

[mattfarina]

@monopole Right now kubectl is built for Windows and has install instructions. Instead of improving test coverage or fixing some bugs for windows, the most popular developer operating system, the suggestion is to bake in a feature that doesn't work on Windows and has no plan for windows support.

Is this correct?

[akutz]

Hi All,

So Go plug-ins. Where do I even begin. @mattfarina is correct, there's no planned support for Windows as far as I'm aware, but more than that, Go plug-ins themselves are fraught with overestimated expectations that run quickly into some unfortunate barriers of entry due to the way Go's typing system works.

A few notes:

• Everything I'm going to say derives from the in-depth discussion on Go plug-ins at plugin: Utility of Go plug-ins diminished by vendored dependencies golang/go#20481, plugin: Utility of Go plug-ins diminished by vendored dependencies golang/go#20481 (comment), and plugin: Utility of Go plug-ins diminished by vendored dependencies golang/go#20481 (comment).
• I proposed a standard pattern/approach to Go plug-ins two years ago called Gpds https://github.com/akutz/gpds.
• All of the above is related to long-running host processes, but can absolutely be applied to programs with a much shorter happy-path.

When to use Go plug-ins

Go plug-ins make sense in three scenarios:

1. Plug-ins are built at the same time as the host program in order to provide a base program and additional functionality that can be enabled at runtime by loading plug-ins. Building the plug-ins at the same time as the host program, using the exact same sources, can ensure the exact same dependency graph and ensure that the plug-ins will load into the host program and use shared types to share data.
2. Shared types are not used to share data unless:
   1. The types belong to stdlib.
   2. The types are interface{} references that can be asserted as Go interfaces defined in both the host program and in the plug-in.
3. Information is shared by marshaling/unmarshaling data via gRPC/JSON.

Portable Go plug-ins

The approach described at https://github.com/akutz/gpds illustrates using gRPC to build stand-alone binaries that may also be compiled as Go plug-ins. A host program either loads the archives as plug-ins and controls the lifecycle of the plug-in's built-in, memory backed gRPC server, or a plug-in runs as a stand-alone process, exposing the same gRPC endpoint.

Either way, a host program simply accesses plug-ins a well-defined gRPC endpoints via a well-defined object model and API.

Problems with plug-ins

• Go plug-ins currently only work on Linux and macOS. As far as I'm aware, there are no current plans to get plug-ins working on Windows.
• People want to think of Go plug-ins as the same type of loadable-at-runtime feature that exists in .NET and Java, but this simply isn't true. It's not trivial for @mattfarina to build a program that accepts Go plug-ins and for me to come along a year later and create a plug-in and expect it to work. There are too many gotchas. Without a full understanding of the constraints involved or an adoption of data marshaling as the means to share information, Go plug-ins may be more trouble than they're worth.

Conclusion

I personally think Go plug-ins are really cool, and I'd love to see them in kubectl. However, because kubectl is a short-lived program, I think external binaries at a well-defined location, ex. ${HOME}/.kubectl/plugins, similar to how Terraform works, is probably a more sensible solution. However, even in this approach, I still think the gRPC pattern outlined at https://github.com/akutz/gpds lays the foundation for building extensible support as independent binaries or as Go plug-ins loaded into the kubectl host process.

[sethpollack]

@monopole @pwittrock @soltysh Should we just use the terraform approach? It's a nice middle ground between go plugins and kubectl plugins.

1. Binaries are prefixed and stored in a dedicated directory.
2. They use grpc for a strict interface with well-defined responsibilities
3. Works on all platforms and plugins can be written in any language.

[anguslees]

3. Works on all platforms and plugins can be written in any language.

This. I'm slightly confused where we stand on this wrt kustomize. I think we're proposing (in other KEPs) making kustomization.yaml a standard part of the "Kubernetes Experience". If we make golang plugins a user-visible part of kustomize, then we're also saying that all tools that read "kubernetes yaml" files must be implemented in golang (or call out to an external tool implemented in golang).

Everything else in the k8s public API is language-neutral (with some minor leakage around the quirks of golang's encoding semantics). We (as a k8s project) should be super-clear on whether we're ok binding all our users/implementers to a single language or not. If we want choice of programming environment then exposing a golang plugin API is obviously a non-starter, and it would be great if we stated that out loud so future (sig-cli most likely) KEPs can avoid rediscovering this objection.

For this use-case in particular, we want the freedom to choose languages, because the whole point is to integrate with some existing secrets-management system. It is quite likely that an existing system has APIs that suggest some languages over others - eg: pkcs11 modules are defined as C ABIs, which are particularly difficult to use from golang; macos credentials stores might be easier to access in a proprietary Apple language.

---

Personally, I would much prefer we just use simple exec-based plugins here. grpc is huge overkill for something that just takes string arguments, and returns key/value pairs(*) - in particular requiring grpc prevents me from just writing my plugin as a simple bit of shell glue around an existing command (which is what almost all of these secret plugins will be).

(*) Eg: Return value could be base64-encoded values in json to stdout

[anguslees]

If we must go with golang plugins, please don't just allow the kustomization.yaml file to invoke any symbol that happens to be exported.

A better way to do this is to have each plugin export a well-known symbol (perhaps KustomizationSecretGenerators) that is a map (or array for simplicity) of name => function. That way only explicitly marked functions will be accessible.

As a bonus, it allows the same plugin .so to be used to provide other sets of entry points (with different function signatures) in the future. This includes versioning the well-known symbol so we can cleanly evolve the ABI over time.

[monopole]

@anguslees Policy around exposed vars sounds reasonable.

[pwittrock]

@sethpollack yes the doc should reflect the design we plan to implement. which are the new design details you are referring to? I think we had reached agreement to move forward with a go-plugin approach for iteration 1, and leave exec for iteration 2.

[monopole]

@pwittrock We were only discussing field names, i.e. type vs pluginType.

That's a level of nitpicking best left to the implementation PR. This KEP is done (modulo spelling?)

[monopole]

/lgtm

[donbowman]

given that most of the above comments are against doing go-plugins, and given that we seem to instead be going with go-plugins, can we at least provide a reference plugin that calls exec and wraps so those of us with existing tools aren't broken?

[sethpollack]

@donbowman existing tools are already broken, the exec functionality was removed a while ago. We do plan on adding other plugin types like exec as well.

[mattfarina]

@sethpollack This KEP should capture the plan for plugins. With exec plugins are not listed in the plans. If they are in the plan shouldn't that be listed here? Maybe as part of a multi-phase approach.

[monopole]

@sethpollack Please address above comments by making the following changes:

1. Update The proposed API would look like this: with what we agreed on above:

secretGenerator:
- name: app-tls
  kvSources:
  - pluginType: builtin  // builtin is the default value of pluginType
    name: files
    args:
    - secret/tls.cert
    - secret/tls.key
- name: env_file_secret
  kvSources:
  - name: env  // this is a builtin
    args:
    - env.txt
- name: myJavaServerEnvVars
  kvSources:
  - name: literals    // this is a builtin
    args:
    - JAVA_HOME=/opt/java/jdk
    - JAVA_TOOL_OPTIONS=-agentlib:hprof
- name: secretFromPlugins
  kvSources:
  - pluginType: go      // described by this KEP
    name: someLocalGoPlugin
    args:
    - someArg
    - someOtherArg
  - pluginType: kubectl      // some future KEP can write this
    name: someKubectlPlugin
    args:
    - anotherArg
    - yetAnotherArg

2. Update Risks and Mitigations

Risks and Mitigations

Several people want an exec-style plugin

exec-style means execute arbitrary code from some file.

Most the lines of code written to implement this KEP
accomodate the notion of KV generation via a generic
notion of a plugin, and a generic interface, in the
kustomization file and associated structs and code.

This code recharactizes the three existing KV
generation methods as pluginType: builtin
(literals, env and files), introduces the new
pluginType: Go, and leaves room for someone to easily
add, say pluginType: kubectl to look for a kubectl
plugin, and pluginType: whatever to handles some
completely new style, e.g. look for the plugin name as
an executable in some hardcoded location.

Actual implementation of these other kinds of plugins
are out of scope for this KEP, however this KEP's
implementation will make it much easier to create other
kinds of plugins.

Go Plugins that become popular have a clear
path to becoming a builtin - so if someone writes, say,
a general exec-style plugin, we can easily promote it to a
builtin (by virtue of the fact that it's written in Go,
and because of the choices made in this KEP for
describing a plugin in the kustomization file).

No current windows support for golang plugins.

This may be implemented by someone later in which
case goplugins will work on windows. Go itself was
not supported on windows when first introduced.

Also, as noted above, someone can write an exec style plugin
which will work on Windows.

General symbol exection from the plugin

A Go plugin will be cast to a particular hard-coded interface, e.g.

type KvSourcePlugin {
  Get []kvPairs
}

and accessed exclusively through that interface.

Existing KV generators continue as an option

We leave in place the existing three mechanisms
(literals, env and files) for generating KV pairs, but
additionally allow these mechanisms to be expressed as
builtin plugins (see example above).

If plugins - both Go and other styles - prove unpopular
or problematic, we can remove them per API change
policies.

If they do prove popular/useful, we can deprecate
the legacy form for (literals, env and files),
and help people convert to the new builtin form.

[monopole]

Ok, that addresses the final concerns from @donbowman and @mattfarina, thanks!

/lgtm

[monopole]

/hold cancel

[monopole]

/lgtm

thanks for the spelling fixes

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: monopole, pwittrock, sethpollack

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-cli/OWNERS [pwittrock]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[monopole]

/retest

[monopole]

/test pull-enhancements-verify

[donbowman]

the kep is closed and merged, but... Here is a sample https://www.agilicus.com/safely-secure-secrets-a-sops-plugin-for-kustomize/ to show how to use it with sops https://github.com/mozilla/sops (since this is the top result in search and i didn't find one)