Make SCC more restrictive #513

hvaghani221 · 2022-08-12T12:21:30Z

The agent doesn't write in the root directory. So it makes sense to enable readOnlyRootFilesystem.
By default, CRI-O will add the following capabilities. That can be dropped as well.

CHOWN
DAC_OVERRIDE
FSETID
FOWNER
SETGID
SETUID
SETPCAP
NET_BIND_SERVICE
KILL

You can checked current capabilities using:

$oc rsh sck-otel-splunk-otel-collector-agent-pgg77 bash
bash-4.4# cat /proc/1/status | grep Cap
CapInh: 0000000000000000
CapPrm: 00000000000005fb
CapEff: 00000000000005fb
CapBnd: 00000000000005fb
CapAmb: 0000000000000000

capsh --decode=00000000000005fb
0x00000000000005fb=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service

dmitryax · 2022-08-12T20:35:17Z

The agent doesn't write in the root directory. So it makes sense to enable readOnlyRootFilesystem.

The agent writes log checkpoints to /var/addon/splunk/otel_pos. Please make sure it's still working.

@jvoravong, can you please validate this change from Splunk O11y side? We need to make sure that all host metrics are collected correctly with the dropped capabilities?

Also we need to make sure that container and journald logs are collected using both log engines fluend/otel.

jvoravong · 2022-08-12T20:50:32Z

I'll validate these changes early next week.

mtcolman · 2022-08-17T13:35:28Z

Hi, I can see that you're dropping the default capabilities provided when CRI-O is the container engine (as per the "default_capabilities" section of https://github.com/cri-o/cri-o/blob/main/docs/crio.conf.5.md).

Docker however, grants additional capabilities as default, see here: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities. Can you please assess these additional capabilities and look at dropping as many of these that aren't required?

The additional ones granted by Docker are:

MKNOD
NET_RAW
SETFCAP
SYS_CHROOT
AUDIT_WRITE

Alternatively, dropping "ALL" and then adding back in only the capabilities the otel-collector needs would be best practice.

Thank you!

hvaghani221 · 2022-08-18T12:52:22Z

I have executed functional tests manually with these changes on Red Hat OpenShift Local.

All tests are passing except 3

These are expected failures since the hostname would be different and docker/containerd logs wouldn't be there.

jvoravong · 2022-08-18T22:15:27Z

Finished up some testing:

Metrics -> Look Good
Traces -> Look Good
Logs -> Found an issue. When using the the otel log engine everything is good. When using the fluentd log engine, the agent goes into a crash loop.

Using Kuberenetes 1.24 and Openshift 4.11 (Rosa)

splunk-otel-collector-chart % k version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-08-17T18:46:11Z", GoVersion:"go1.19", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0+9546431", GitCommit:"0a57f1f59bda75ea2cf13d9f3b4ac5d202134f2d", GitTreeState:"clean", BuildDate:"2022-07-08T19:55:26Z", GoVersion:"go1.18.4", Compiler:"gc", Platform:"linux/amd64"}

Warning log printed with a Helm Install

W0818 15:34:17.467292   41736 warnings.go:70] would violate PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true), hostPort (container "otel-collector" uses hostPorts 14250, 14268, 4317, 4318, 55681, 8006, 9080, 9411, 9943), allowPrivilegeEscalation != false (containers "prepare-fluentd-config", "fluentd", "otel-collector" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "prepare-fluentd-config", "fluentd", "otel-collector" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "varlog", "varlogdest", "journallogpath", "host-dev", "host-etc", "host-proc", "host-run-udev-data", "host-sys", "host-var-run-utmp" use restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "prepare-fluentd-config", "fluentd", "otel-collector" must set securityContext.runAsNonRoot=true), runAsUser=0 (containers "prepare-fluentd-config", "fluentd" must not set runAsUser=0), seccompProfile (pod or containers "prepare-fluentd-config", "fluentd", "otel-collector" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0818 15:34:17.553879   41736 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "otel-collector" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "otel-collector" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "otel-collector" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "otel-collector" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Fluentd containers entering a crashloop

k get pod splunk-otel-collector-agent-zz4ld
splunk-otel-collector-agent-zz4ld                            1/2     CrashLoopBackOff   5 (2m3s ago)    5m11s   XX.X.XXX.XX   ip-XX-XX-XXX-XX.ec2.internal   <none>           <none>
k describe pod splunk-otel-collector-agent-zz4ld
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  91s                default-scheduler  Successfully assigned monitoring/splunk-otel-collector-agent-27wr9 to ip-XX-XX-XXX-XX.ec2.internal by ip-XX-XX-XXX-XX
  Normal   Pulled     91s                kubelet            Container image "splunk/fluentd-hec:1.2.8" already present on machine
  Normal   Created    91s                kubelet            Created container prepare-fluentd-config
  Normal   Started    91s                kubelet            Started container prepare-fluentd-config
  Normal   Pulled     90s                kubelet            Container image "quay.io/signalfx/splunk-otel-collector:0.57.0" already present on machine
  Normal   Created    90s                kubelet            Created container otel-collector
  Normal   Started    90s                kubelet            Started container otel-collector
  Warning  Unhealthy  89s                kubelet            Readiness probe failed: Get "http://XX.X.XXX.XXX:13133/": dial tcp XX.X.XXX.XXX:13133: connect: connection refused
  Normal   Created    47s (x4 over 90s)  kubelet            Created container fluentd
  Normal   Started    47s (x4 over 90s)  kubelet            Started container fluentd
  Normal   Pulled     47s (x4 over 90s)  kubelet            Container image "splunk/fluentd-hec:1.2.8" already present on machine
  Warning  BackOff    31s (x6 over 86s)  kubelet            Back-off restarting failed container

@harshit-splunk please look into the Fluentd crashes.

CHANGELOG.md

dmitryax · 2022-08-18T22:36:06Z

Logs -> Found an issue. When using the the otel log engine everything is good. When using the fluentd log engine, the agent goes into a crash loop.

I believe this is caused by disabling readOnlyRootFilesystem. I suspect that otel collector also cannot write checkpoints to /var/addon/splunk/otel_pos, just doesn't fail. @harshit-splunk please validate that checkpoints can be written.

jvoravong · 2022-08-18T23:43:43Z

I did specifically miss mentioning that. I observed the receiver offset files in /var/addon/splunk/otel_pos were successfully being written to. Which was surprising...

dmitryax · 2022-08-19T16:15:34Z

Interesting, if this cannot be fixed for fluentd, we can just update the template and readOnlyRootFilesystem based on logsEngine configuration.

Anyway, first we need to know why fluentd is failing.

hvaghani221 · 2022-08-19T16:27:06Z

Actually, the checkpoint directory is mounted as the volume when otel logs engine is used. So, it is able to write there.

splunk-otel-collector-chart/helm-charts/splunk-otel-collector/templates/daemonset.yaml

Lines 383 to 384 in 0f63517

    
           - name: checkpoint 
        
             mountPath: {{ .Values.logsCollection.checkpointPath }}

It's missing in fluentd. I'll include that in this PR.

dmitryax · 2022-08-19T20:03:39Z

@harshit-splunk fluentd configuration is a bit confusing but it's also mounted at /var/log

splunk-otel-collector-chart/helm-charts/splunk-otel-collector/templates/daemonset.yaml

Lines 200 to 201 in 0f63517

    
           - name: varlog 
        
             mountPath: {{ .Values.fluentd.config.containers.path }}

.

Co-authored-by: jvoravong <[email protected]>

hvaghani221 · 2022-08-22T12:26:19Z

@dmitryax @jvoravong Fluentd internally writes to /tmp/fluentd directory. Since it was read-only, it wasn't able to create the directory. So I have mounted an empty directory to /tmp

jvoravong

Fluentd works with these latest changes. Good work. Giving my approval, let's also give dmitryax a chance to respond.

dmitryax

LGTM

Make SCC more restrictive

a939ae2

hvaghani221 requested review from a team as code owners August 12, 2022 12:21

harshit-splunk added 2 commits August 12, 2022 17:58

Updated changelog

a64318f

Fix pre-commit

320b30b

drop all capabilities

4f71695

Merge branch 'main' into restrict-scc

e3717cc

jvoravong reviewed Aug 18, 2022

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

hvaghani221 and others added 2 commits August 22, 2022 12:32

Update CHANGELOG.md

283b055

Co-authored-by: jvoravong <[email protected]>

Mounted emptyDir volume to /tmp in fluentd container

08bb83a

hvaghani221 requested a review from jvoravong August 22, 2022 12:33

jvoravong approved these changes Aug 22, 2022

View reviewed changes

dmitryax approved these changes Aug 22, 2022

View reviewed changes

dmitryax merged commit 3819317 into signalfx:main Aug 22, 2022

hvaghani221 deleted the restrict-scc branch August 23, 2022 04:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make SCC more restrictive #513

Make SCC more restrictive #513

hvaghani221 commented Aug 12, 2022 •

edited

Loading

dmitryax commented Aug 12, 2022

jvoravong commented Aug 12, 2022

mtcolman commented Aug 17, 2022

hvaghani221 commented Aug 18, 2022

jvoravong commented Aug 18, 2022 •

edited

Loading

dmitryax commented Aug 18, 2022

jvoravong commented Aug 18, 2022

dmitryax commented Aug 19, 2022

hvaghani221 commented Aug 19, 2022

dmitryax commented Aug 19, 2022

hvaghani221 commented Aug 22, 2022

jvoravong left a comment

dmitryax left a comment

Make SCC more restrictive #513

Make SCC more restrictive #513

Conversation

hvaghani221 commented Aug 12, 2022 • edited Loading

dmitryax commented Aug 12, 2022

jvoravong commented Aug 12, 2022

mtcolman commented Aug 17, 2022

hvaghani221 commented Aug 18, 2022

jvoravong commented Aug 18, 2022 • edited Loading

dmitryax commented Aug 18, 2022

jvoravong commented Aug 18, 2022

dmitryax commented Aug 19, 2022

hvaghani221 commented Aug 19, 2022

dmitryax commented Aug 19, 2022

hvaghani221 commented Aug 22, 2022

jvoravong left a comment

Choose a reason for hiding this comment

dmitryax left a comment

Choose a reason for hiding this comment

hvaghani221 commented Aug 12, 2022 •

edited

Loading

jvoravong commented Aug 18, 2022 •

edited

Loading