Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core Feature][Flytekit] Flytekit should have a consistent API to request for secrets #796

Closed
kumare3 opened this issue Mar 3, 2021 · 1 comment
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@kumare3
Copy link
Contributor

kumare3 commented Mar 3, 2021

Motivation: Why do you think this is important?
As we write many plugins in flytekit, secrets are very important for many plugins. For non Cloud cases, where IAM can help access the resource without needing explicit secrets, other services need secret to access these resources. We recommend using Vault, which writes kube-secret files, but users may use Confidant, kube-secrets, and other potential solutions. For non K8s cases, users may use AWS KMS etc.
The best way to ensure that for plugin-authors and users, the way the secrets are provisioned an attained should be an implementation detail. Flytekit should provide a canonical way of accessing secrets

Goal: What should the final outcome look like, ideally?
Users/Plugin writers should be able to provide secret keys during declaration of a task.
At runtime, the actual values for these secrets should be accessible through the context.

SECRET1_KEY = "secret1"
SECRET2_KEY="secret2"

@task(secrets=[SECRET1_KEY, SECRET2_KEY])
def foo(x: int) -> int:
  ...
   v1 = flytekit.current_context().secrets.get(SECRET1_KEY)
   v2 =flytekit.current_context().secrets.get(SECRET2_KEY)
  ...

# For not function tasks, it would mostly not make sense to expose the api of arbitrary secrets to users
# but the required secrets should be documented

# But for plugin writers it should be possible to do

class MySQLTask(SQLTask):
  _SEC1_KEY="secret1"
  def _init__(...):
     super().__init__(..., secrets=[_SEC1_KEY])


# Every plugins should have a table of various config parameters available and various secrets required

Describe alternatives you've considered
Implicit usage of Vault of pod by users

@kumare3 kumare3 added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers and removed untriaged This issues has not yet been looked at by the Maintainers labels Mar 3, 2021
@kumare3 kumare3 added this to the 0.12.0 milestone Mar 3, 2021
@kumare3 kumare3 self-assigned this Mar 3, 2021
@kumare3
Copy link
Contributor Author

kumare3 commented Mar 17, 2021

This should be merged tomorrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants