audit: reduce noise in logging export

[spiffxp]

#2102 introduced export of logging resources to the audit script. Based on review of the first audit job PR that used this (#2094) there is some noise we should filter out to ease review burden.

empty metrics.json

Currently there are lots of services/logging/metrics.json files with content []. If there are no metrics, we shouldn't export them.

e2e test logs

Logs appear to be showing up for all pods used in e2e tests. For example audit/projects/k8s-infra-e2e-boskos-010/services/logging/logs.json has a diff that looks like:

   "projects/k8s-infra-e2e-boskos-010/logs/events",
   "projects/k8s-infra-e2e-boskos-010/logs/externalname-service",
-  "projects/k8s-infra-e2e-boskos-010/logs/externalsvc",
-  "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-0d464eed-160c-4f37-963b-36a011030391",
-  "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-31c7c414-74fa-49ca-8a8f-48c571c143a3",
   "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-3638847c-8c6c-47dc-9c89-32c571411622",
   "projects/k8s-infra-e2e-boskos-010/logs/filler-pod-b6d92649-31d0-4d9a-8634-f0f8fe06ebbe",
# ...
   "projects/k8s-infra-e2e-boskos-010/logs/test-container-subpath-projected-fwgf",
+  "projects/k8s-infra-e2e-boskos-010/logs/test-container-subpath-projected-2v7w",
+  "projects/k8s-infra-e2e-boskos-010/logs/test-container-subpath-projected-fwgf",

We should either choose to ignore/filter these out, or determine how to configure our e2e tests to not send any logs. I swear we had done this a while ago, but we only ever verified by way of costs going down.

/wg k8s-infra
/area infra/auditing
/priority important-longterm
/milestone v1.22

[spiffxp]

#2133 removed export of logs for k8s-infra-e2e projects as a start

Should survey remaining log churn in audit PR's to track down what can be done

[spiffxp]

Log noise that seems like it shouldn't be present:

• addition of projects/{project}/logs/cloudaudit.googleapis.com%2Fsystem_event
  • shouldn't this log always be present?
  • example projects: k8s-staging-sig-storage, k8s-release-test-prod
• removal of projects/{project}/logs/cloudaudit.googleapis.com%2Fsystem_event
  • shouldn't this log always be present?
  • example projects: k8s-artifacts-prod, k8s-cip-test-prod

gcloud logging logs list --help says Only logs that contain log entries are listed.

Every project has at least these two buckets

$ gcloud logging buckets list
LOCATION  BUCKET_ID  RETENTION_DAYS  LIFECYCLE_STATE  LOCKED  CREATE_TIME  UPDATE_TIME
global    _Default   30              ACTIVE
global    _Required  400             ACTIVE           True

And at least these two sinks that route to them

$ gcloud logging sinks list --format=yaml
---
destination: logging.googleapis.com/projects/spiffxp-gke-dev/locations/global/buckets/_Required
filter: LOG_ID("cloudaudit.googleapis.com/activity") OR LOG_ID("externalaudit.googleapis.com/activity")
  OR LOG_ID("cloudaudit.googleapis.com/system_event") OR LOG_ID("externalaudit.googleapis.com/system_event")
  OR LOG_ID("cloudaudit.googleapis.com/access_transparency") OR LOG_ID("externalaudit.googleapis.com/access_transparency")
name: _Required
---
destination: logging.googleapis.com/projects/spiffxp-gke-dev/locations/global/buckets/_Default
filter: NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT LOG_ID("externalaudit.googleapis.com/activity")
  AND NOT LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT LOG_ID("externalaudit.googleapis.com/system_event")
  AND NOT LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT LOG_ID("externalaudit.googleapis.com/access_transparency")
name: _Default

So are we losing system_event logs because nothing has happened to generate a log entry there in 400 days?

[spiffxp]

/remove-priority important-longterm
/priority backlog
It's annoying but it's not really creating a lot of additional review burden for me at this point

[spiffxp]

/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale
/lifecycle frozen