SElinux experiences for who wants to know #107
Comments
|
@coeki I don't think we technically document Fedora installations, I assume you used the Centos7 repos? Which docker did you use, the one in Fedora or one from Docker Inc? Could you include details on the denials? spc_t should be correct per http://danwalsh.livejournal.com/2016/10/03/ @jasonbrooks do you have any thoughts on this, potential fallout from kubernetes/kubernetes#37327 I will try to find some time to see if I can reproduce this week. |
|
Reproduced: Does not happen on CentOS 7 as far as I know. |
|
Hi, Yes, that are the type of errors I'm seeing. For clarification, yes I used the Centos repos and I used the docker from Fedoras repo. I will do a quick check if this happens with Centos7 later today. Thanks, |
|
Hi, Actually happens on centos7 too: type=AVC msg=audit(1484065309.021:634): avc: denied { entrypoint } for pid=12204 comm="exe" path="/usr/local/bin/etcd" dev="dm-9" ino=4194436 scontext=system_u:system_r:spc_t:s0:c390,c679 tcontext=system_u:object_r:svirt_sandbox_file_t:s0:c390,c679 tclass=file Regarding Dan Walsh's post, I also saw this http://danwalsh.livejournal.com/75011.html, so I think stuff changed. Further things I'm seeing: [root@localhost vagrant]# ls -Z /var/lib/ |grep container [root@localhost vagrant]# ls -Z /var/lib/kubelet/pods/3a26566bb004c61cd05382212e3f978f/containers/etcd/ The same on Centos and Fedora. Not sure what the way to go is, but I think we don't need to specify SElinux types anymore in any manifests. At least for kubeadm, the pod networking is another story. Thoughts? Thanks, |
|
I'm poking at this now. |
|
Thanks @jasonbrooks. FWIW CentOS7 was clean for me with: kubeadm init was working and this was the env I was doing my testing with to verify things were ok. I then thought to check if my vagrant machines were running latest policy and that picked up a bunch of selinux updates, after which kubeadm init now fails with the denial provided by @coeki. So something has gone wrong with the latest policy. After my update: |
|
Look at this. CentOS 7 w/ docker-selinux: And after installing container-selinux: |
|
Yep, I added |
|
ping @rhatdan |
|
So we should document |
|
I'm thinking we'll need an selinux fix to address this.
…On Jan 10, 2017 22:41, "Lucas Käldström" ***@***.***> wrote:
So we should document exclude=container-selinux as a solution to having
SELinux on (setenforce 1) and running kubeadm?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#107 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAG52vBqKyMAcSj62oGqazI2hWETd-RPks5rRHmZgaJpZM4Le4yP>
.
|
|
After reading mentioned post from Dan Walsh ( http://danwalsh.livejournal.com/75011.html) a little better, either there's en error in typealiasing the docker_t types to the new generic name for container types container_t, or it didn't get pulled yet. Another train of thought. I'm not sure how docker containers are launched by kubernetes, but a fix could also be to launch the containers with the new container types. |
|
Ingnore the last remark, it's the OS contianer runtime of course, so yes @jasonbrooks remark is right, we need a selinux fix. |
|
@lsm5 can we get an updated version of container-selinux for Fedora 24. Seems to be a little out of date there. |
|
What version of docker, docker-selinux, container-selinux does Centos 7 have? |
|
From above: In my comment above you can also see the previous versions I had where everything was working, and then it starts failing after a yum update to these versions. |
|
@dgoodwin can you try the latest container-selinux for CentOS 7? You can get it using this repo: |
|
Specifically, http://cbs.centos.org/koji/buildinfo?buildID=15216 |
|
Still failing @lsm5 |
|
Could you make sure container-selinux successfully installed? dnf reinstall container-selinux |
|
I tried with that latest version of container-selinux yesterday on CentOS
and also I tried the container-selinux that's in updates-testing for f25,
same issue.
…On Jan 11, 2017 07:32, "Daniel J Walsh" ***@***.***> wrote:
Could you make sure container-selinux successfully installed?
dnf reinstall container-selinux
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#107 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAG52tRhNh6En1hfbZJzzGISzNUnhVfKks5rRPYUgaJpZM4Le4yP>
.
|
|
I think this block is silently failing. optional_policy(` |
|
Still failing after reinstall. |
|
I am setting up a CENTOS machine to check what is going wrong. |
|
Thanks for the help everyone, it would be cool if we would be able to have |
|
Hi, I tried with fedora 25 cloud base, the same result as in, it not installing, but different messages: time->Sat Jan 21 13:55:37 2017
|
|
Oops, sorry for the markup in the last post, have no clue how that happened ;) |
|
@coeki it's because of the # at the beginning of those lines. I suggest indenting the lines with four spaces, which lets GH know that it's code. |
|
Yes, many thanks for this investigation! I'm a noob at selinux (haven't ever used it, I'm more of a debian guy), so this effort is worth its weight in gold. I guess we're hoping for a resolution to this in k8s v1.6, right? Also, do network providers like weave, flannel, etc. add these selinux rules as well in order for their hostpath mounts to work properly as well? If that's the case, we have to inform them in time for the next release. |
|
Thanks all for having a look at this. So to recap, tell me if I read it wrong: 1 Issue with the format of the label ":" vs"=" worked on here: #kubernetes/kubernetes#40179 @luxas I was first trying to fix kubeadm and then have a look at what exactly goes wrong with the network providers like weave, flannel, canal, calico and romana, probably the same isssues, but we'll have to pinpoint first I think, so we can tell them what to fix. |
this is just for docker > 1.12.6 - docker in Fedora/RHEL has 1.12.6 which correctly applies label |
Automatic merge from submit-queue (batch tested with PRs 38443, 40145, 40701, 40682) Move kubeadm etcd SELinux options from container to pod. **What this PR does / why we need it**: Works around a bug that surfaces in Docker 1.12+ related to the pause container's namespace and selinux labels being transferred to the etcd container when it runs. At present it appears that applying selinux options to a container may be broken, or perhaps shouldn't be supported at all. Moving these to the pod causes all containers (including pause) to run with the correct labels. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # **Special notes for your reviewer**: Related to and partial fix for kubernetes/kubeadm#107 This is one of several selinux related fixes in flight for upcoming releases, and newer versions of Docker. To successfully run kubeadm with selinux enforcing right now would like require a recent container-selinux build as uncovered in kubernetes/kubeadm#107, a bugfix for the format labels in #40179, and finally this fix. **Release note**: ```release-note Fixed an SELinux issue in kubeadm on Docker 1.12+ by moving etcd SELinux options from container to pod. ```
This is correct. It is done in the security context provider by the
@pmorie @mrunalp @eparis - help me understand the correct direction for the SC provider in this scenario. It sounds like since we always share the IPC namespace with the pause container that:
if item 2 is true then I think that is a strong argument to only have pod level settings. |
|
I'd argue that docker needs to start the pause container with the pod level security context (if defined). It needs to start the actual containers with the container security context (if defined) fallback to the pod level context (if defined) and then fall back to 'shared with pause'. (if undefined) I do not understand why the shared IPC namespace should be relevant to this discussion. If the user defines a combination of contexts that selinux won't allow to work, then things shouldn't work. But docker should never 'overwrite' anything the user asked for. |
Then I think that there isn't anything to do on the provider side. It should already be setting the pause container settings to the pod level SC if defined and actual containers get the merge of pod + container overrides which means it should share settings with pause if things are defined on the pod level and not the container level.
Agree. Thanks! |
|
Yes @eparis is correct. I have to get a patch to docker to handle the labeling correctly. We need to allow user flexibility (IE The ability to break his system if he so chooses.) The default case should be that all container processes sharing the same IPC namespace have share the same SELinux label, but if a user asks to override the label, it should be allowed. |
|
The problem in this case, although I can't put my finger on it, seems a little bit more complex. You can actually see it a little when this fix is not applied: 1 Issue with the format of the label ":" vs"=" worked on here: #kubernetes/kubernetes#40179 docker inspect the pause container: Docker inspect the etcd container: The label with the format "label:" is coming from the manifest with which we started the deployment of the etcd pod. At least I think so. Kubelet or docker seems to add the others, which kinda make sense as they are for the image files placed in /var/lib/docker/containers and /var/lib/kubelet/pods From the discussion in #kubernetes/kubernetes#40179 we know docker is not preventing adding more security opts then actually should be allowed. |
|
@eparis @rhatdan Could you help check this issue too? Why |
|
I have a pull request to help fix this issue in for docker We have already described the issue. In standard docker if you add a container with one label and then add another container which you want to join the same IPC Namespace, docker will take the label of the first container and assign it to the second container. This way SELinux will not block access to IPC created by the first container that needs to be used by the second container. The SELinux type/mcs label at the Pod Level sets the label for the Bottom Line: Old docker, if With my patch above we will allow users to specify alternate selinux labels for each container added, but this should only be done by people who understand how SELinux confinement works. IE You might want to set up a multi container pod where one container has more/less rights then the container it is joining. The docker patch would have fixed the k8s problem, also but it does not eliminate the fact that the pause container probably should be running with the same SELinux label as the containers joining it. |
|
@rhatdan thanks so much |
|
So.........how to test? @dgoodwin @jasonbrooks @rhatdan @jessfraz? I'm sorry not well versed in building stuff, but willing to do so....I have some clue, but any help welcome. Thanks |
|
Ok figured it out over the weekend, but it still not working. |
|
Ok, build kubeadm and kubelet (this part I forgot, hence my latest comment) myself from master branch with kubernetes/kubernetes#40903 merged and it works with selinux enforcing, even with weave running, although weave is not running correctly. That's a weave issue to deal with later as well as the other providers I still have to test. Regarding the double entries referenced in kubernetes/kubernetes#37807: I think that stems from some validation when DetermineEffectiveSecurityContext is run, as @pweil- mentioned, and just added rather then replaced as docker security option. Since docker runs with all options passed to but only uses the last security options passed, this seems to work. Not sure if that is expected behavior for docker, but that's a diff matter. @rhatdan might want to take another look at that, regarding moby/moby#30652 Not sure if we should close the issue, till we get the rpms from official repo's, test again and so forth, test the pod networking first, and fix it, update docs etc, before closing. So let me know. Thanks all for fixing this :) |
|
This looks correct, although with the patch I have for upstream docker you would end up with only one "label=type:spc_t" field. |
|
@rhatdan question, because I build docker with your patch, well maybe I didn't do it right, but I think it did. And I still see this. this behavior didn't change: snip "SecurityOpt": [ it runs with your patch, none the wiser. Docker accepts that, is the patch breaking that? Is there a sane use case that docker should be able to do that? From this issue, if docker changes that, seeing we (kubernetes pod runtime ) end up we double labels, which I think come from the validation in kubelet, if you break the fact docker accepts more options then it should...well it breaks again. Look if sane, we should be able to run a pod with diff security options for containers within a pod, and I think that was the intention, which should be possible, but an alignment is needed. Docker and kubernetes can not have diff expectations about this. I might be wrong, adding @eparis @dgoodwin @jasonbrooks @luxas @pweil- @pmorie |
|
My change was not to prevent that although we probably should. My change was to block on container joining the IPC Namespace or pid namespace of another The second container should be running as spc_t. Where the old code would have had it running as container_t. This basically would mimic two containers running in the same pod with different SELinux labels. |
|
Thanks @rhatdan for clarying. |
|
Closing this one, as the rbac enabled changes for pod networking are tracked #143 Not sure about the double labels, it seems interaction between kubernetes/kubelet and docker as mentioned above. Anyway filed this for docker #moby/moby#31328 Thanks all, |
|
Im running with |
Hi all,
So been battling with my special setup all weekend, but I have to give up and let other folks look at it.
So on windows, vagrant and ansible, I can share my stuff if you like.
Facts:
latest version from repo in the guide.
using fedora 24 as images in virtualbox with vagrant
So following the guide, I got it working, with setenforce permissive, kubeadm runs with calico. In my case I have a small machine for running ansible, setting up a master and a node.
First I tried with setenforce enforcing, and got stuck at the famous, "waiting for the control plane to become ready" , and then used new terminal to look at it. It seems that actually etcd is clashing with selinux due to diff types. In the static manifest, in the code in kubernetes/cmd/kubeadm/app/master/manifest.go, I can see that it's run with type spc_t, while everything run from docker, etcd itself is run under svirt_sandbox_file_t, so yes it clashes. I think it has to to do with the volumes attached or the process actually trying to write to the volume on the master host.
I see similar problems with the pod networking scripts.
So anyone any ideas?
Just trying to get this working with SElinux from the get go ;)
Thanks,
The text was updated successfully, but these errors were encountered: