New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

secrets: avoid repeated calls to k8s API #455

Closed

mdonadoni opened this issue May 3, 2024 · 0 comments · Fixed by reanahub/pytest-reana#131, #456, reanahub/reana-server#686, reanahub/reana-workflow-controller#583 or reanahub/reana-job-controller#451

Assignees

Labels

system/refactor system/scalability type/enhancement

Member

mdonadoni commented May 3, 2024 •

edited

Loading

As of now, each call to get_secret_value results in an API call made to k8s. Let's refactor the code so that it's easy to fetch all the secrets once, without making repeated requests. This will also be useful to cache secrets in reana-job-controller, to avoid calling the k8s API many times for each new job.

As an example, the k8s API is called twice when creating the k8s configuration for the kerberos-related pods:

reana-commons/reana_commons/k8s/kerberos.py

Lines 52 to 53 in ac2a01b

    
           keytab_file = secrets_store.get_secret_value("CERN_KEYTAB") 
        
           cern_user = secrets_store.get_secret_value("CERN_USER")

The text was updated successfully, but these errors were encountered:

mdonadoni added type/enhancement system/refactor system/scalability labels

mdonadoni self-assigned this

mdonadoni added a commit to mdonadoni/pytest-reana that referenced this issue


          refactor(fixtures): update secrets-related fixtures (reanahub#131)

e91d085

Update secrets-related fixtures to be compatible with latest changes to
`reana-commons`.

Closes reanahub/reana-commons#455

mdonadoni mentioned this issue

fix(fixtures): correctly encode secrets in base64 (#131) reanahub/pytest-reana#131

Merged

mdonadoni added a commit to mdonadoni/reana-commons that referenced this issue


          perf(k8s): avoid fetching secrets multiple times (reanahub#456)

6c02ed7

Closes reanahub#455

mdonadoni mentioned this issue

perf(k8s): avoid fetching secrets multiple times (#456) #456

Merged

mdonadoni added a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

63d2206

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

mdonadoni mentioned this issue

refactor(secrets): adapt to reana-commons secret-handling changes (#686) reanahub/reana-server#686

Merged

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

82afe28

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni mentioned this issue

refactor(secrets): adapt to reana-commons secret-handling changes (#583) reanahub/reana-workflow-controller#583

Merged

mdonadoni added a commit to mdonadoni/pytest-reana that referenced this issue


          fix(fixtures): correctly encode secrets in base64 (reanahub#131)

da2ae0c

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-commons that referenced this issue


          perf(k8s): avoid fetching secrets multiple times (reanahub#456)

e26308c

Closes reanahub#455

mdonadoni added a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

e4c00d3

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

39d460c

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

dc94f04

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this issue


          perf(secrets): avoid fetching k8s secrets multiple times (reanahub#451)

a8d4a77

Cache user secrets to avoid fetching them multiple times when creating
many jobs.

Closes reanahub/reana-commons#455

mdonadoni mentioned this issue

perf(secrets): avoid fetching k8s secrets multiple times (#451) reanahub/reana-job-controller#451

Merged

mdonadoni added a commit to mdonadoni/pytest-reana that referenced this issue


          fix(fixtures): correctly encode secrets in base64 (reanahub#131)

5d6d1f2

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-commons that referenced this issue


          perf(k8s): avoid fetching secrets multiple times (reanahub#456)

e9cc2d9

Closes reanahub#455

mdonadoni added a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

7b0c890

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

f0929b3

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

92485ef

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

0a58b47

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

ecbcc9b

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this issue


          perf(secrets): avoid fetching k8s secrets multiple times (reanahub#451)

ecc05ca

Cache user secrets to avoid fetching them multiple times when creating
many jobs.

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

81a489e

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

mdonadoni added this to 0.95.0

mdonadoni moved this to In review in 0.95.0

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

9165c73

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-commons that referenced this issue


          perf(k8s): avoid fetching secrets multiple times (reanahub#456)

6d0f7f8

Closes reanahub#455

mdonadoni added a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

3a56d7d

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

9696d57

…anahub#583)

Closes reanahub/reana-commons#455

mdonadoni added a commit to mdonadoni/reana-job-controller that referenced this issue


          perf(secrets): avoid fetching k8s secrets multiple times (reanahub#451)

9eaf650

Cache user secrets to avoid fetching them multiple times when creating
many jobs.

Closes reanahub/reana-commons#455

tiborsimko closed this as completed in


      reanahub/pytest-reana@d4a07cf

github-project-automation bot moved this from In review to Done in 0.95.0

tiborsimko pushed a commit to mdonadoni/reana-commons that referenced this issue


          perf(k8s): avoid extraneous refetching of user secrets (reanahub#456)

77d263e

Closes reanahub#455

github-actions bot mentioned this issue

chore(master): release 0.95.0 #450

Open

tiborsimko pushed a commit to mdonadoni/reana-workflow-controller that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

d9512fb

…anahub#583)

Closes reanahub/reana-commons#455

tiborsimko pushed a commit to mdonadoni/reana-server that referenced this issue


          refactor(secrets): adapt to reana-commons secret-handling changes (re…

3596ee0

…anahub#686)

Refactor secrets-related endpoints to adapt to performance-related
refactor of `reana-commons`. Improve validation of endpoints'
parameters.

Closes reanahub/reana-commons#455

tiborsimko pushed a commit to mdonadoni/reana-job-controller that referenced this issue


          perf(secrets): avoid extraneous refetching of user secrets (reanahub#451

d8180d0

)

Cache user secrets to avoid fetching them multiple times when creating
many jobs.

Closes reanahub/reana-commons#455

tiborsimko pushed a commit to mdonadoni/reana-job-controller that referenced this issue


          perf(secrets): avoid extraneous refetching of user secrets (reanahub#451

e046989

)

Cache user secrets to avoid fetching them multiple times when creating
many jobs.

Closes reanahub/reana-commons#455

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment