Add GKE 1.6 CIS benchmark for GCP environment #1672

ttousai · 2024-09-02T05:39:39Z

Implements #1662

CLAassistant · 2024-09-02T05:39:45Z

All committers have signed the CLA.

deven0t · 2024-09-02T07:30:33Z

Hi @ttousai
We will need to update cis version selection based on k8s version here https://github.com/aquasecurity/kube-bench/blob/main/cmd/util.go#L497
can you check and update, so new version will selected as per k8s version

ttousai · 2024-09-03T10:48:07Z

Hello @deven0t I have added the selection based on k8s version and also added updates to various documents about gke-1.6.0 support.

guyjerby

Hi @ttousai
there are different errors while running the benchmark - tests are not completed succsfully - can you make the changes and verify again on GKE cluster that the tests are completed succsfully?

guyjerby · 2024-09-15T15:16:57Z

cfg/gke-1.6.0/node.yaml

+            - flag: "--anonymous-auth"
+              path: '{.authentication.anonymous.enabled}'
+              compare:
+                op: eq


@ttousai
.authentication.anonymous.enabled should not appears in kubelet-config.yaml as enabled

guy_jerby@gke-gke-test-cluster-bas-default-pool-ba74cdf0-qbln /etc/kubernetes $ cat kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
webhook:
enabled: false
authorization:
mode: AlwaysAllow
enableServer: false
podCIDR: 10.42.0.0/24
staticPodPath: /etc/kubernetes/manifests
staticPodURL: http://metadata.google.internal/computeMetadata/v1/instance/attributes/google-container-manifest
staticPodURLHeader:
Metadata-Flavor: [Google]
cgroupDriver: systemd

guyjerby · 2024-09-15T15:25:43Z

cfg/gke-1.6.0/node.yaml

+            - flag: --streaming-connection-idle-timeout
+              path: '{.streamingConnectionIdleTimeout}'
+              set: false
+          bin_op: or


@ttousai , the result that the kube-bench shows is not logical => '{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present

This check should PASS:

if --streaming-connection-idle-timeout is set to any value not equal to 0 on the command line or,

if streamingConnectionIdleTimeout is set to any value not equal to 0 in the config file or,

if --streaming-connection-idle-timeout is not set on the command line or,

if streamingConnectionIdleTimeout is not set in the config file.

In our case it should pass because --streaming-connection-idle-timeout is not set on the command line and it is also not set in the config file (the correct config file is /home/kubernetes/kubelet-config.yaml).

guyjerby · 2024-09-16T09:54:45Z

cfg/gke-1.6.0/node.yaml

+        tests:
+          test_items:
+            - flag: --make-iptables-util-chains
+              path: '{.makeIPTablesUtilChains}'


@ttousai , here is the result,
the iptables-util-chains is exists in the kubelet-config , but the test is failed and exoected result is empty, probably a typo in the test?

{
"test_number": "3.2.6",
"test_desc": "Ensure that the --make-iptables-util-chains argument is set to true (Automated)",
"audit": "/bin/ps -fC kubelet",
"AuditEnv": "",
"AuditConfig": "/bin/cat /etc/kubernetes/kubelet-config.yaml",
"type": "",
"remediation": "Remediation Method 1:\nIf modifying the Kubelet config file, edit the kubelet-config.json file\n/etc/kubernetes/kubelet/kubelet-config.json and set the below parameter to\ntrue\n\n "makeIPTablesUtilChains": true\n\nEnsure that /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf\ndoes not set the --make-iptables-util-chains argument because that would\noverride your Kubelet config file.\n\nRemediation Method 2:\nIf using executable arguments, edit the kubelet service file\n/etc/systemd/system/kubelet.service.d/10-kubelet-args.conf on each\nworker node and add the below parameter at the end of the KUBELET_ARGS variable\nstring.\n\n --make-iptables-util-chains:true\n\nRemediation Method 3:\nIf using the api configz endpoint consider searching for the status of\n"makeIPTablesUtilChains.: true by extracting the live configuration from the nodes\nrunning kubelet.\n\nSee detailed step-by-step configmap procedures in Reconfigure a Node's Kubelet in a\nLive Cluster (https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/),\nand then rerun the curl statement from audit process to check for kubelet\nconfiguration changes\n\n kubectl proxy --port=8001 \u0026\n export HOSTNAME_PORT=localhost:8001 (example host and port number)\n export NODE_NAME=gke-cluster-1-pool1-5e572947-r2hg (example node name from\n "kubectl get nodes")\n curl -sSL "http://${HOSTNAME_PORT}/api/v1/nodes/${NODE_NAME}/proxy/configz"\n\nFor all three remediations:\nBased on your system, restart the kubelet service and check status\n\n systemctl daemon-reload\n systemctl restart kubelet.service\n systemctl status kubelet -l\n",
"test_info": ["Remediation Method 1:\nIf modifying the Kubelet config file, edit the kubelet-config.json file\n/etc/kubernetes/kubelet/kubelet-config.json and set the below parameter to\ntrue\n\n "makeIPTablesUtilChains": true\n\nEnsure that /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf\ndoes not set the --make-iptables-util-chains argument because that would\noverride your Kubelet config file.\n\nRemediation Method 2:\nIf using executable arguments, edit the kubelet service file\n/etc/systemd/system/kubelet.service.d/10-kubelet-args.conf on each\nworker node and add the below parameter at the end of the KUBELET_ARGS variable\nstring.\n\n --make-iptables-util-chains:true\n\nRemediation Method 3:\nIf using the api configz endpoint consider searching for the status of\n"makeIPTablesUtilChains.: true by extracting the live configuration from the nodes\nrunning kubelet.\n\nSee detailed step-by-step configmap procedures in Reconfigure a Node's Kubelet in a\nLive Cluster (https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/),\nand then rerun the curl statement from audit process to check for kubelet\nconfiguration changes\n\n kubectl proxy --port=8001 \u0026\n export HOSTNAME_PORT=localhost:8001 (example host and port number)\n export NODE_NAME=gke-cluster-1-pool1-5e572947-r2hg (example node name from\n "kubectl get nodes")\n curl -sSL "http://${HOSTNAME_PORT}/api/v1/nodes/${NODE_NAME}/proxy/configz"\n\nFor all three remediations:\nBased on your system, restart the kubelet service and check status\n\n systemctl daemon-reload\n systemctl restart kubelet.service\n systemctl status kubelet -l\n"],
"status": "FAIL",
"actual_value": "apiVersion: kubelet.config.k8s.io/v1beta1\nkind: KubeletConfiguration\nauthentication:\n webhook:\n enabled: false\nauthorization:\n mode: AlwaysAllow\nenableServer: false\nmakeIPTablesUtilChains:true\npodCIDR: 10.42.0.0/24\nstaticPodPath: /etc/kubernetes/manifests\nstaticPodURL: http://metadata.google.internal/computeMetadata/v1/instance/attributes/google-container-manifest\nstaticPodURLHeader:\n Metadata-Flavor: [Google]\ncgroupDriver: systemd",
"scored": true,
"IsMultiple": false,
"expected_result": ""
}

No it was a bad test, the test should PASS if the makeIPTablesUtilChains is not set on either the command line or in the config file which is our case so it should pass.

I have added a fix for the test.

afdesk · 2024-10-07T05:04:29Z

Hi guys!
Is this PR ready for review and merge? thanks!

guyjerby · 2024-10-08T18:45:49Z

Hi @afdesk - we are still validating the benchmark and fixing the last founded issues, soon we will be able to merge that and I will notify you, will you able to merge it and also to build a new kube-bench release and image?

guyjerby · 2024-10-08T18:53:17Z

cfg/gke-1.6.0/node.yaml

+            systemctl daemon-reload
+            systemctl restart kubelet.service
+            systemctl status kubelet -l
+        scored: false


@ttousai - scored should be true for all automatic tests

afdesk · 2024-10-09T05:13:52Z

Hi @afdesk - we are still validating the benchmark and fixing the last founded issues, soon we will be able to merge that and I will notify you, will you able to merge it and also to build a new kube-bench release and image?

hi @guyjerby!
yes, I'll try to review and merge it ASAP after your approve.

thanks for the answer.

guyjerby · 2024-10-09T13:32:44Z

@afdesk , @ttousai - The validation has been completed , we can merge the PR and build a new kube-bench release

afdesk

hi guys! I left a few comments.

as for me, only a mark about 3.1.3 is critical.

thanks for understanding.

afdesk · 2024-10-10T06:07:57Z

cfg/gke-1.6.0/controlplane.yaml

+      - id: 2.1.1
+        text: "Client certificate authentication should not be used for users (Manual)"
+        type: "manual"


this check marked as automation.
If there is no way to automate it, we should add a command to test in remediation section:

$ kubectl get secrets --namespace kube-system # Look for secrets with names starting with gke-. These secrets contain the client certificates.

afdesk · 2024-10-10T06:09:45Z

cfg/gke-1.6.0/node.yaml

+    text: "Worker Node Configuration Files"
+    checks:
+      - id: 3.1.1
+        text: "Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Manual)"


this check is marked as automated.

afdesk · 2024-10-10T06:10:37Z

cfg/gke-1.6.0/node.yaml

+        scored: true
+
+      - id: 3.1.2
+        text: "Ensure that the proxy kubeconfig file ownership is set to root:root (Manual)"


this check is marked as automated and there is audit command here

afdesk · 2024-10-10T06:11:07Z

cfg/gke-1.6.0/node.yaml

+        scored: true
+
+      - id: 3.1.3
+        text: "Ensure that the kubelet configuration file has permissions set to 600 (Manual)"


this check is marked as automated and there is audit command

afdesk · 2024-10-10T06:11:38Z

cfg/gke-1.6.0/node.yaml

+        scored: true
+
+      - id: 3.1.4
+        text: "Ensure that the kubelet configuration file ownership is set to root:root (Manual)"


this check is marked as automated and there is audit command

afdesk · 2024-10-10T06:12:20Z

cfg/gke-1.6.0/node.yaml

+            - flag: "permissions"
+              compare:
+                op: bitmask
+                value: "644"


it was changed to 600

@ttousai - can you change to 600?

afdesk · 2024-10-10T06:34:39Z

cfg/gke-1.6.0/node.yaml

+          test_items:
+            - flag: "--read-only-port"
+              path: '{.readOnlyPort}'
+              set: false
+            - flag: "--read-only-port"
+              path: '{.readOnlyPort}'
+              compare:
+                op: eq
+                value: 0
+          bin_op: or


if --read-only-port isn't set, the check will pass, right?
is it correct behavior?
just make sure

Verify that the --read-only-port argument exists and is set to 0.

@afdesk yes this is the correct behavior. According to the kubelet config doc if readOnlyPort is not set, it defaults to 0, which is disable.

afdesk · 2024-10-10T06:47:13Z

cfg/gke-1.6.0/policies.yaml

+      - id: 4.1.1
+        text: "Ensure that the cluster-admin role is only used where required (Automated)"
+        type: "manual"
+        remediation: |


does it make sense to add audit tip into remediation block? wdyt?

Audit:
Obtain a list of the principals who have access to the cluster-admin role by reviewing
the clusterrolebinding output for each role binding that has access to the cluster-
admin role.
kubectl get clusterrolebindings -o=custom-
columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[*].name
Review each principal listed and ensure that cluster-admin privilege is required for it.

@afdesk I think the idea makes sense, I'm not sure if we want to add it to the remediation block though.
@guyjerby wdyt?

guyjerby · 2024-10-10T14:46:45Z

Thanks @afdesk for your review - regarding the automated test that cannot be added - this is because we never run kubectl commands from the kube-bench pod in the previous releases and yes - if we converted it to manual - the remediation must cover it.

in any case - I am going to test the kubectl functionality from the kube-bench pod - if it works well - we will have create another PR to automate them

@ttousai - can you make the relevant fixes for the remediations if missing and than we will finalize it?

Thanks!

ttousai · 2024-10-10T16:32:21Z

@guyjerby @afdesk I am done with the changes we can make at the moment.

afdesk · 2024-10-11T04:49:58Z

@guyjerby @ttousai thanks for your efforts.
merged.

I'll take a look at another PR and try to cut a new release today

guyjerby · 2024-10-11T11:10:30Z

@afdesk , @ttousai Thank you very much! for assisting with GKE 1.6 and having the new release ready!

guyjerby · 2024-10-11T11:27:42Z

@afdesk , how can I contact you? can you share your mail address or send it to me? [email protected] ?

* Add config entries for GKE 1.6 controls * Add gke1.6 control plane recommendations * Add gke-1.6.0 worker node recommendations * Add gke-1.6.0 policy recommendations * Add managed services and policy recommendation * Add master recommendations * Fix formatting across gke-1.6.0 files * Add gke-1.6.0 benchmark selection based on k8s version * Workaround: hardcode kubelet config path for gke-1.6.0 * Fix tests for makeIPTablesUtilChaings * Change scored field for all node tests to true * Fix kubelet file permission to check for --------- Co-authored-by: afdesk <[email protected]>

ttousai added 6 commits August 9, 2024 07:08

Add config entries for GKE 1.6 controls

50f4779

Add gke1.6 control plane recommendations

7794be3

Add gke-1.6.0 worker node recommendations

5c27cb2

Add gke-1.6.0 policy recommendations

aee2710

Add managed services and policy recommendation

69222c1

Add master recommendations

ebd34d5

Fix formatting across gke-1.6.0 files

f3e0d50

ttousai force-pushed the gke1.6 branch from 3821796 to f3e0d50 Compare September 2, 2024 05:49

Add gke-1.6.0 benchmark selection based on k8s version

e96f308

guyjerby self-requested a review September 10, 2024 12:07

TastefulToast mentioned this pull request Sep 11, 2024

Adding GKE-1.6.0 CIS benchmark #1677

Open

guyjerby requested changes Sep 16, 2024

View reviewed changes

ttousai and others added 3 commits September 23, 2024 13:56

Workaround: hardcode kubelet config path for gke-1.6.0

e30c1e6

Fix tests for makeIPTablesUtilChaings

ef1968b

Merge branch 'main' into gke1.6

ec9b88a

guyjerby requested changes Oct 8, 2024

View reviewed changes

Change scored field for all node tests to true

fcda8cf

guyjerby approved these changes Oct 9, 2024

View reviewed changes

Merge branch 'main' into gke1.6

0d13ece

afdesk requested changes Oct 10, 2024

View reviewed changes

Fix kubelet file permission to check for

3d4ce59

afdesk self-requested a review October 11, 2024 04:48

afdesk approved these changes Oct 11, 2024

View reviewed changes

afdesk merged commit a15e8ac into aquasecurity:main Oct 11, 2024
5 checks passed

afdesk mentioned this pull request Oct 11, 2024

GKE 1.6 CIS benchmark for GCP environment #1662

Closed

ttousai deleted the gke1.6 branch October 11, 2024 08:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GKE 1.6 CIS benchmark for GCP environment #1672

Add GKE 1.6 CIS benchmark for GCP environment #1672

ttousai commented Sep 2, 2024

CLAassistant commented Sep 2, 2024 •

edited

Loading

deven0t commented Sep 2, 2024

ttousai commented Sep 3, 2024

guyjerby left a comment

guyjerby Sep 15, 2024

guyjerby Sep 15, 2024

ttousai Sep 23, 2024 •

edited

Loading

guyjerby Sep 16, 2024

ttousai Sep 23, 2024

afdesk commented Oct 7, 2024

guyjerby commented Oct 8, 2024

guyjerby Oct 8, 2024

afdesk commented Oct 9, 2024

guyjerby commented Oct 9, 2024 •

edited

Loading

afdesk left a comment •

edited

Loading

afdesk Oct 10, 2024

afdesk Oct 10, 2024

afdesk Oct 10, 2024

afdesk Oct 10, 2024

afdesk Oct 10, 2024

afdesk Oct 10, 2024

guyjerby Oct 10, 2024

ttousai Oct 10, 2024

afdesk Oct 10, 2024

ttousai Oct 10, 2024

afdesk Oct 10, 2024

ttousai Oct 10, 2024

guyjerby commented Oct 10, 2024

ttousai commented Oct 10, 2024

afdesk commented Oct 11, 2024

guyjerby commented Oct 11, 2024 •

edited

Loading

guyjerby commented Oct 11, 2024

Add GKE 1.6 CIS benchmark for GCP environment #1672

Add GKE 1.6 CIS benchmark for GCP environment #1672

Conversation

ttousai commented Sep 2, 2024

CLAassistant commented Sep 2, 2024 • edited Loading

deven0t commented Sep 2, 2024

ttousai commented Sep 3, 2024

guyjerby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttousai Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afdesk commented Oct 7, 2024

guyjerby commented Oct 8, 2024

Choose a reason for hiding this comment

afdesk commented Oct 9, 2024

guyjerby commented Oct 9, 2024 • edited Loading

afdesk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guyjerby commented Oct 10, 2024

ttousai commented Oct 10, 2024

afdesk commented Oct 11, 2024

guyjerby commented Oct 11, 2024 • edited Loading

guyjerby commented Oct 11, 2024

CLAassistant commented Sep 2, 2024 •

edited

Loading

ttousai Sep 23, 2024 •

edited

Loading

guyjerby commented Oct 9, 2024 •

edited

Loading

afdesk left a comment •

edited

Loading

guyjerby commented Oct 11, 2024 •

edited

Loading