audit: redo organizations, storage-api and add container/clusters, logging, monitoring #2102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
area/audit
This was referenced May 26, 2021
spiffxp
force-pushed
the
audit-add-services
branch
from
ca29784 to
b46e46e
Compare
k8s-ci-robot
added
size/L
|
/hold |
k8s-ci-robot
added
the
do-not-merge/hold
spiffxp
force-pushed
the
audit-add-services
branch
2 times, most recently
from
608cb26 to
94903bb
Compare
spiffxp
changed the title
Fix a few bugs that were introduced by the last PR: - prevent bq from aborting audit script early - fix leftover variable name from refactor - avoid importing infra/gcp/lib.sh since it automatically looks for yq, which is not in the image used by the job that runs this script
This is mostly a straight copy-paste of the bash-script embedded in the ci-k8sio-audit job defined in: - repo: https://github.com/kubernetes/test-infra - sha: 624c7344bdb56bd25e119a8b1ccc284939861938 - file: config/jobs/kubernetes/wg-k8s-infra/trusted/wg-k8s-infra-trusted.yaml Added boilerplate header, comments up top, and put line breaks in between each block of code beginning with an echo statement
Export to audit/organizations/kubernetes.io/... Add a migrate_audit_format function that commits a migration from old format to new. Hoping this makes it easier to follow changes during review
include migration code to move to new format
Try to make bucket auditing faster by avoiding multiple gsutil calls for each possible setting; only make them if they're not the default setting. Add support for auditing lifecycle and retention configurations if present.
spiffxp
force-pushed
the
audit-add-services
branch
from
94903bb to
1f47041
Compare
|
/hold |
|
/hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
|
It might also be easier to instead merge #2103 which will close this out as well |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cpanato, spiffxp The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is part of a series of PRs:
Now that #2099 has deployed, the audit job should be able to export more resources. Take a crack at
loggingandmonitoringwith this PR.Makes the following layout changes / resource additions by service type
organizations:org_kubenetes.iotoorganizations/kubernetes.ioorganizations/kubernetes.io/description.jsoncontainers:projects/{project}/services/container/clusters.txtprojects/{project}/sevices/container/clusters/{cluster}.jsonlogging:projects/{project}/services/logging/logs.jsonprojects/{project}/services/logging/metrics.jsonprojects/{project}/services/logging/sinks.jsonmonitoring:projects/{project}/services/monitoring/dashboards/{dashboard}.jsonstorage-api:gsutilcalls by listing buckets, avoiding specific configurations unless presentprojects/{project}/buckets/{bucket}/metadata.txtprojects/{project}/buckets/{bucket}/logging.json(if configured)projects/{project}/buckets/{bucket}/lifecycle.json(if configured)projects/{project}/buckets/{bucket}/retention.json(if configured)projects/{project}/buckets/{bucket}/bucketpolicyonly.txt(now redundant givenmetadata.txt)projects/{project}/buckets/{bucket}/cors.txt(now redundant givenmetadata.txt)projects/{project}/buckets/{bucket}/logging.txt(now redundant givenlogging.json,metadata.txt)The actual migration to the new format is in followup PR #2103