Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ensure k8s-conform service accounts have objectAdmin rights #1890

Merged
merged 6 commits into from
Apr 9, 2021

Conversation

spiffxp
Copy link
Member

@spiffxp spiffxp commented Apr 8, 2021

This ensures that for each conformance bucket k8s-conform-foo, the same write privileges are given to:

The code before this PR looked like it would probably do this for any newly provisioned GCS bucket. However ensure-conformance-storage.sh was written in a way that it wasn't always refreshing resources and IAM bindings. So I rewrote it to refresh as much as possible.

While I was here I fixed a few other bugs in the course of troubleshooting / verifying this refactor:

  • ensure-conformance-storage.sh can no longer provision arbitrary buckets if invoked with a typo'ed argument
  • lib_iam.sh doesn't crash on empty IAM policies
  • "repo" seemed like the wrong term for what we're provision conformance buckets for, so it's now "offering"
  • K8S_INFRA_DEBUG=true ./ensure-whatever.sh will list and leave behind the contents of $TMPDIR instead of removing it
  • lib*.sh is no longer executable (these files should be sourced, not executed)

Fixes #1850

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. wg/k8s-infra labels Apr 8, 2021
@k8s-ci-robot k8s-ci-robot requested review from dims and thockin April 8, 2021 20:31
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 8, 2021
spiffxp added 6 commits April 8, 2021 20:49
specifically

- add secret_full_name: generates "projects/p/secrets/s"
- add ensure_secret: creates a secret if it doesn't already exist
- add ensure_seviceaccount_key_secret: if a secret doesn't exist, create
  it and pre-populate with a newly created serviceaccount key
specifically

- reorganize into functions
- use lib_gsm functions
- redo to ensure all resources (except one-time private key
  provisioning)
- switch from "repo" to "offering" as the term for what we're
  provisioning for, borrowed from CNCF's conformance language
- prevent provisioning of unrecognized offerings
@spiffxp spiffxp force-pushed the redo-ensure-conformance branch from 0360438 to 5890959 Compare April 9, 2021 00:51
@spiffxp
Copy link
Member Author

spiffxp commented Apr 9, 2021

Ran K8S_INFRA_ENSURE_ONLY_SERVICES_WILL_FORCE_DISABLE=true ./ensure-conformance-storage.sh as of 5890959

@spiffxp spiffxp changed the title [wip] ensure k8s-conform service accounts have objectAdmin rights ensure k8s-conform service accounts have objectAdmin rights Apr 9, 2021
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 9, 2021
@spiffxp
Copy link
Member Author

spiffxp commented Apr 9, 2021

/cc @ameukam @saschagrunert

@spiffxp
Copy link
Member Author

spiffxp commented Apr 9, 2021

It looks like this decided to create new keys for two service accounts, because neither of these had corresponding secrets provisioned beforehand

for email in $(gcloud iam service-accounts list --project=k8s-conform --filter='email ~ "iam.gserviceaccount.com"' --format='value(email)'); do 
  echo "${email}"...; 
  gcloud iam service-accounts keys list --projct=k8s-conform --iam-account=${email} --managed-by=user; 
done
[email protected]...
KEY_ID                                    CREATED_AT            EXPIRES_AT
ce8a54a39915f738a85a28d41d521b998fac5166  2020-02-27T01:30:44Z  9999-12-31T23:59:59Z
d07efa6acb769eac354120680c011168326368c7  2021-04-08T20:43:11Z  9999-12-31T23:59:59Z
[email protected]...
KEY_ID                                    CREATED_AT            EXPIRES_AT
6ffe09fb3aa6a13c62d778eb4d3441b1f14059b7  2021-02-23T06:37:03Z  9999-12-31T23:59:59Z
[email protected]...
KEY_ID                                    CREATED_AT            EXPIRES_AT
045c845ce8d8e1a5bc5671fac07c71353987596c  2020-09-23T21:18:10Z  9999-12-31T23:59:59Z
[email protected]...
KEY_ID                                    CREATED_AT            EXPIRES_AT
92b2b472b09b1714aa69fb4742b9ec5776ade47d  2021-02-15T15:18:07Z  9999-12-31T23:59:59Z
[email protected]...
KEY_ID                                    CREATED_AT            EXPIRES_AT
6ac4a3cd42ef8b169502c363f2c32485005c2785  2020-02-26T12:52:14Z  9999-12-31T23:59:59Z
16204156ed33ab4865f83a4ba0766de992e90998  2021-04-08T20:32:12Z  9999-12-31T23:59:59Z

The two accounts that have new keys were created before 2020-05-28, when #873 added the provision-to-secret pattern. The previous code gated creating new secrets on service account existence, where the code now gates on secret existence.

I opened #1895 as a followup issue to get the key users to switch to the new keys, but I don't consider it a big security risk to leave both keys enabled.

@spiffxp
Copy link
Member Author

spiffxp commented Apr 9, 2021

#1892 reflects basically all of the changes these scripts caused. It was too early to capture the latest run I did in #1890 (comment). I'll point those changes out in the next audit PR

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 9, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: saschagrunert, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@saschagrunert
Copy link
Member

saschagrunert commented Apr 9, 2021

/hold
for further reviews since I'm not aware of the full code context within this repo

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 9, 2021
@spiffxp
Copy link
Member Author

spiffxp commented Apr 9, 2021

/hold cancel
#1897 (comment) shows services getting removed

/assign @thockin
if you want to review post-merge

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 9, 2021
@k8s-ci-robot k8s-ci-robot merged commit d52b5a5 into kubernetes:main Apr 9, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Apr 9, 2021
@thockin
Copy link
Member

thockin commented Apr 9, 2021

Aaron, your bash is so pleasant to read that I'm going to invite you to collab on my bash-only rewrite of Kubernetes.

/lgtm

@spiffxp spiffxp deleted the redo-ensure-conformance branch April 9, 2021 16:33

local project="${1}"
local secret="${2}"
local serviceaccount="${3}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spiffxp you mean email ?

# $3: The service-account (e.g. "[email protected]")
function ensure_serviceaccount_key_secret() {
if [ ! $# -eq 3 -o -z "$1" -o -z "$2" -o -z "$3" ]; then
echo "ensure_serviceaccount_key_secret(project, secret, serviceaccountt) requires 3 arguments" >&2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

email instead of serviceaccountt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

audit followup: give k8s-conform serviceaccounts objectAdmin privileges for their respective gcs buckets
5 participants