Disable Vulnerability Scanning

[dims]

Just got a nice note from the friendly GCP folks:

[Action Required] Free Container Scanning ends May 19, 2021; Normal billing begins July 1, 2021

I was able to click on disable for k8s-artifacts-prod, but looks like we need to do this across the board automatically.

[dims]

cc @thockin @spiffxp

[spiffxp]

/assign

[ameukam]

/assign @thockin

[spiffxp]

Are we concerned the costs are really going to be that outsized? tl;dr it looks like ~2.5% of spend as currently deployed

• Pricing is $0.26 per scanned image https://cloud.google.com/container-analysis/pricing
• Looking at the billing report, I believe the dollars spent there are representative of what we would pay, they're just being covered with gratis GCP credits instead of our own. So, ballpark numbers:
  • April: 148K total, 3.7K vuln scanning (2.5%) (2K of which was k8s-staging-ci-images (54%))
  • YTD: 584K total, 16K vuln scanning (2.7%) (10K of which was k8s-staging-ci-images (62%))
  • Last Year: 700K total, 11K vuln scanning (1.5%) (3K of which was k8s-staging-ci-images (27%), 0.6K of which was k8s-artifacts-prod (5%)

Regardless, I'm interested in pruning services that shouldn't be enabled anyway, so I'm using this as a reason to push more on that work.

WIP PR is here: #2016

[spiffxp]

It's now May 19 and I haven't been able to land #2016 yet. I will reprioritize this.

In the meantime, let's manually disable it for the project that generates the majority of costs

$ gcloud services disable containerscanning.googleapis.com --project=k8s-staging-ci-images
Provide the --force flag if you wish to force disable services.
ERROR: (gcloud.services.disable) FAILED_PRECONDITION: The service containerscanning.googleapis.com has usage in the last 30 days. Please specify check_if_service_has_usage=SKIP if you want to proceed with disabling the service.

OK, let's use --force as suggested

$ gcloud services disable --force containerscanning.googleapis.com --project=k8s-staging-ci-images
Operation "operations/acf.p17-731599680865-1c03d2a4-d620-4b90-b319-65f44b3ac2c6" finished successfully

[justaugustus]

Thanks for the heads-up, Aaron!
@kubernetes/release-engineering -- We should be fine here, but please keep an eye out for anything strange in our GCP projects.

[spiffxp]

#2016 disabled containerscanning.googleapis.com for all projects

there is leftover detritus in the form of a container scanning service agent IAM binding

[spiffxp]

/priority awaiting-more-evidence
/remove-priority important-soon
/milestone v1.23
to remove the service agent IAM binding

[spiffxp]

Based on a survey of all staging projects (ref: #1675 (comment)), I'm also going ahead and disabling the containeranalysis API

#!/usr/bin/env bash

gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-addon-manager
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-apisnoop
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-artifact-promoter
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-autoscaling
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-bootkube
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-boskos
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-build-image
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-capi-docker
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-capi-kubeadm
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-capi-openstack
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-capi-vsphere
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-ci-images
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cip-test
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cloud-provider-gcp
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cluster-addons
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cluster-api
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cluster-api-aws
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cluster-api-azure
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cluster-api-do
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cluster-api-gcp
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-coredns
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cpa
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-cri-tools
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-csi
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-csi-secrets-store
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-descheduler
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-dns
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-e2e-test-images
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-etcd
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-etcdadm
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-examples
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-experimental
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-external-dns
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-gateway-api
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-git-sync
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-infra-tools
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-ingress-nginx
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-ingressconformance
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-k8s-gsm-tools
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kas-network-proxy
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kind
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kops
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kube-state-metrics
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kubeadm
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kubernetes
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kubetest2
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-kustomize
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-metrics-server
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-mirror
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-multitenancy
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-networking
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-nfd
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-npd
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-provider-aws
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-provider-azure
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-provider-openstack
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-publishing-bot
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-releng
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-releng-test
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-scheduler-plugins
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-scl-image-builder
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-sig-docs
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-sig-storage
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-slack-infra
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-sp-operator
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-storage-migrator
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-test-infra
gcloud services disable --force containeranalysis.googleapis.com --project=k8s-staging-txtdirect

[spiffxp]

The audit PR #2689 picked up some of the changes from above

[spiffxp]

Opened #2697 to remove the service agent binding

[spiffxp]

Waiting on an audit PR to prove the service agent has been removed and then I think we can call this done

[spiffxp]

#2699 is that audit PR

/close
calling this done!

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

#2699 is that audit PR

/close
calling this done!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thockin]

Revisiting this - is there any way to enable scanning for "recent" images (e.g. uploaded in the last 6 months) or something? I had a question about git-sync CVEs and I ... don't have a report!

[BenTheElder]

Seems like it wouldn't be that expensive if we just enabled it on one AR backend?

I don't think the pricing analysis above is accurate, unless something changed:

https://cloud.google.com/container-analysis/pricing

If you choose to enable the vulnerability scanning feature for your container images, the price is $0.26 per scanned container image.

1. When you enable the Container Scanning API, billing begins immediately.

2. Container Analysis automatically scans an image the first time you push it to the registry and charges you for the initial scan. This charge applies only to the initial scan and is not affected by the number of layers on the base image. Once the image has been uploaded and initially scanned, subsequent scans of the same image are free.

3. Each image is uniquely identified by a digest. Images with the same digest are considered the same image.

4. Container Analysis does not automatically scan existing images after you enable the API. To scan an existing image, you must push it again.

Each new version of an image is associated with a digest, a unique identifier that is created when you push the image to Container Analysis. Tags that you add to an image are labels and do not change the content of the image, therefore you are not billed for adding or editing tags.

(emphasis mine) ... so I read that as $0.26 per new digest pushed, one-time cost, we can't be adding that many new digests annually?

That said, we are still well-over budget as-is :/

[ameukam]

@thockin if you only use on-demand scanning, you should be able to do scans on recent builds. You can also use tools like https://github.com/aquasecurity/trivy to get similar results.