Unable upload conformance result to bucket

[ydcool]

I have updated my email in groups.yaml (kubernetes/k8s.io#1600) , but still have no access to upload files to the bucket

What happened:

python3 ./test-infra/testgrid/conformance/upload_e2e.py --junit=./k8s-conformance/v1.18/inspur-iop-amd64/junit_01.xml --log=./k8s-conformance/v1.18/inspur-iop-amd64/e2e.log --bucket=gs://k8s-conform-inspur
Uploading entry to: gs://k8s-conform-inspur/1628824828
Run: ['gsutil', '-q', '-h', 'Content-Type:text/plain', 'cp', '-', 'gs://k8s-conform-inspur/1628824828/started.json'] stdin={"timestamp": 1628824828}
ResumableUploadAbortException: 403 [email protected] does not have storage.objects.create access to k8s-conform-inspur/1628824828/started.json.
Traceback (most recent call last):
  File "./test-infra/testgrid/conformance/upload_e2e.py", line 326, in <module>
    main(sys.argv[1:])
  File "./test-infra/testgrid/conformance/upload_e2e.py", line 316, in main
    upload_string(gcs_dir+'/started.json', started_json, args.dry_run)
  File "./test-infra/testgrid/conformance/upload_e2e.py", line 176, in upload_string
    "Failed to upload with exit code: %d" % proc.returncode)
RuntimeError: Failed to upload with exit code: 1

What you expected to happen:

upload file success with using the command above.

How to reproduce it (as minimally and precisely as possible):

gcloud auth login [email protected]

python3 ./test-infra/testgrid/conformance/upload_e2e.py \
    --junit=./k8s-conformance/v1.18/inspur-iop-amd64/junit_01.xml \
    --log=./k8s-conformance/v1.18/inspur-iop-amd64/e2e.log \
    --bucket=gs://k8s-conform-inspur

Please provide links to example occurrences, if any:

Anything else we need to know?:

[ydcool]

/help

[spiffxp]

/assign @BenTheElder
/remove-help
/wg k8s-infra
/sig testing
/sig architecture
/area conformance
/milestone v1.21
/priority important-soon

[BenTheElder]

The current GCP user must have access to the bucket. There's not much we can do about that here in test-infra..

ResumableUploadAbortException: 403 [email protected] does not have storage.objects.create access to k8s-conform-inspur/1628824828/started.json.

This seems to be an issue with the k8s.io configuration? I don't remember if this is automated at all.

[spiffxp]

Can't troubleshoot the user getting access through groups because of kubernetes/k8s.io#1695

$ gcloud policy-troubleshoot iam \
  --project=k8s-staging-e2e-test-images \
  //storage.googleapis.com/projects/_/buckets/k8s-conform-inspur \
  [email protected] \
  --permission=storage.objects.create
access: UNKNOWN_INFO_DENIED

Looking at IAM for the bucket in question

# gs://k8s-conform-inspur/
{
  "bindings": [
    {
      "members": [
        "group:[email protected]",
        "projectEditor:k8s-conform",
        "projectOwner:k8s-conform"
      ],
      "role": "roles/storage.legacyBucketOwner"
    },
    {
      "members": [
        "projectViewer:k8s-conform"
      ],
      "role": "roles/storage.legacyBucketReader"
    },
    {
      "members": [
        "group:[email protected]"
      ],
      "role": "roles/storage.objectAdmin"
    },
    {
      "members": [
        "allUsers"
      ],
      "role": "roles/storage.objectViewer"
    }
  ],
  "etag": "CAg="
}

It's missing permissions for group:[email protected]. This is scripted, but not automatically run. See https://github.com/kubernetes/k8s.io/blob/e0aa798c58a4cc10dd4c798ff90d8cd4bc1a0e6e/infra/gcp/ensure-conformance-storage.sh#L99

Guessing this was missed when kubernetes/k8s.io#1426 merged. Running it now

spiffxp@spiffxp-macbookpro:k8s.io (main %)$ ./infra/gcp/ensure-conformance-storage.sh inspur
Ensuring project exists: k8s-conform
billingAccountName: billingAccounts/018801-93540E-22A20E
billingEnabled: true
name: projects/k8s-conform/billingInfo
projectId: k8s-conform
Enabling the GCS API
Enabling the Secret Manager API
Ensuring all conformance buckets
  Configuring conformance bucket for inspur
  Configuring bucket: gs://k8s-conform-inspur
  Ensuring the bucket exists and is world readable
  Enabling Bucket Policy Only for gs://k8s-conform-inspur...
  No changes made to gs://k8s-conform-inspur/
  Ensuring the GCS bucket retention policy is set: k8s-conform
  Setting Retention Policy on gs://k8s-conform-inspur/...
  Empowering GCS admins
  Updated IAM policy for project [k8s-conform].
  bindings:
  - members:
    - serviceAccount:[email protected]
    role: roles/compute.serviceAgent
  - members:
    - serviceAccount:[email protected]
    - serviceAccount:[email protected]
    role: roles/editor
  - members:
    - user:[email protected]
    role: roles/owner
  - members:
    - group:[email protected]
    role: roles/viewer
  etag: BwW7-yR7NZM=
  version: 1
  No changes made to gs://k8s-conform-inspur/
  No changes made to gs://k8s-conform-inspur/
  Empowering [email protected] to GCS
  Creating service account: service-inspur
  Created service account [service-inspur].
  Empowering service account: service-inspur to GCS
  Creating private key for service account: service-inspur
  created key [6ffe09fb3aa6a13c62d778eb4d3441b1f14059b7] of type [json] as [/tmp/service-inspur.l3VyPg/key.json] for [[email protected]]
  Creating secret to store private key
  Created secret [service-inspur-key].
  Adding private key to secret
  Created version [1] of the secret [service-inspur-key].
  Empowering [email protected] for read secret
  Updated IAM policy for secret [service-inspur-key].
  bindings:
  - members:
    - group:[email protected]
    role: roles/secretmanager.secretAccessor
  etag: BwW7-yW-U1I=
  version: 1
Done

[spiffxp]

This looks correct.

$ gsutil iam get gs://k8s-conform-inspur/
{
  "bindings": [
    {
      "members": [
        "group:[email protected]",
        "projectEditor:k8s-conform",
        "projectOwner:k8s-conform"
      ],
      "role": "roles/storage.legacyBucketOwner"
    },
    {
      "members": [
        "projectViewer:k8s-conform"
      ],
      "role": "roles/storage.legacyBucketReader"
    },
    {
      "members": [
        "group:[email protected]",
        "serviceAccount:[email protected]"
      ],
      "role": "roles/storage.legacyBucketWriter"
    },
    {
      "members": [
        "group:[email protected]",
        "group:[email protected]",
        "serviceAccount:[email protected]"
      ],
      "role": "roles/storage.objectAdmin"
    },
    {
      "members": [
        "allUsers"
      ],
      "role": "roles/storage.objectViewer"
    }
  ],
  "etag": "CA4="
}

Ideally if you're uploading these automatically, you should use the service account, e.g.

# do this once, and store the key someplace safe
gcloud secrets versions access latest --secret=service-inspur-key --project=k8s-conform > service-inspur-key.json

# then everytime you call the script, get the key from that safe place and use it
upload_e2e.py --key-file=service-inspur-key.json

[spiffxp]

/assign @ydcool
please /close once you've verified this works

[ydcool]

Yes, it works now, thank you so much! @spiffxp

[BenTheElder]

Thank you @spiffxp !