add openshift/installer 'bazel build tarball' test to prow

[sallyom]

@bbguimaraes another installer job for prow, thanks

[bbguimaraes]

That's the default value, so it's not really necessary.

[bbguimaraes]

/lgtm

[openshift-ci-robot]

@sallyom: Updated the job-config configmap using the following files:

• key openshift-installer-presubmits.yaml using file ci-operator/jobs/openshift/installer/openshift-installer-presubmits.yaml

In response to this:

@bbguimaraes another installer job for prow, thanks

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wking]

This job is failing with:

+ bazel --output_base=/tmp build tarball
Error: $USER is not set, and unable to look up name of current user: (error: 0): Success

I'm looking into it, but thought I'd post to this PR in case someone more familiar with the Prow execution environment understood what was going on without digging ;). Here's a successful run in Travis.

[wking]

The error message is coming from this block, where Bazel is finding an unset $USER, falling back on getpwuid(getuid()) and getting either a NULL struct pointer or pw_name. I'm not sure how that would happen in Prow but not Travis, because in both cases Bazel is running inside a container. Maybe Prow has stricter seccomp settings or some such?

[bbguimaraes]

@wking: from the man page:

The getpwnam() and getpwuid() functions return a pointer to a passwd structure,
or NULL if the matching entry is not found or an error occurs.  If an error
occurs, errno is set appropriately.

It means there's no entry for that uid in the passwd file, which makes sense
because containers in Openshift execute with arbitrary uids.

That seems like a strange requirement for a build system (but Bazel never
ceases to amaze me).

[wking]

It looks like the container is explicitly setting a user:

# podman pull quay.io/coreos/tectonic-builder:bazel-v0.3
# podman history quay.io/coreos/tectonic-builder:bazel-v0.3 | grep USER
<missing>      5 months ago   /bin/sh -c #(nop) USER [bazel]                  0B

I'm not quite sure where that version of the image comes from, but I think the associated Dockerfile line is here. The USER Dockerfile command does not result in the USER environment variable being set:

# podman run --rm -it quay.io/coreos/tectonic-builder:bazel-v0.3 id
uid=1000(bazel) gid=1000(bazel) groups=1000(bazel)
# podman run --rm -it quay.io/coreos/tectonic-builder:bazel-v0.3 env | grep USER

[bbguimaraes]

That is ignored when the pod is assigned a uid by the security policy in
Openshift (not sure about podman). I'm not familiar with Bazel, but you can
always set the USER variable yourself in the job/script if its value is
irrelevant.

[wking]

That is ignored when the pod is assigned a uid by the security policy in Openshift...

Why would OpenShift care about the container-side UID (I can understand caring about the host-side UID)?

I'm not familiar with Bazel, but you can always set the USER variable yourself in the job/script if its value is irrelevant.

Grepping, it looks like the value may only be used for an output directory name. We're explicitly setting the output directory, and we aren't installing anything, so I expect we are safe to explicitly set USER. I'll work up a PR.

[wking]

... so I expect we are safe to explicitly set USER...

Once we have a fix, I expect we'll want to update the Prow job config in this repo. But for testing fixes, I'll use local adjustments in the hack script itself. Setting USER did get us past the earlier error. Now we're dying with:

+ USER=bazel-testing bazel --output_base=/tmp build tarball
Error: mkdir('/.cache/bazel/_bazel_bazel-testing'): (error: 13): Permission denied

I'll keep adding command-line arguments and environment variables until I get the script to pass...

[wking]

Setting both USER and HOME was sufficient to get Bazel up off the ground. But then a Python invocation crashed with:

ERROR: /home/prow/go/src/github.com/openshift/installer/BUILD.bazel:106:1: error executing shell command: 'bazel-out/host/bin/external/bazel_tools/tools/build_defs/pkg/build_tar --flagfile=bazel-out/k8-fastbuild/bin/tf_bin.args' failed (Exit 1)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site.py", line 554, in <module>
    main()
  File "/usr/lib/python2.7/site.py", line 536, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/lib/python2.7/site.py", line 272, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/lib/python2.7/site.py", line 247, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/lib/python2.7/site.py", line 237, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/lib/python2.7/sysconfig.py", line 582, in get_config_var
    return get_config_vars().get(name)
  File "/usr/lib/python2.7/sysconfig.py", line 533, in get_config_vars
    _CONFIG_VARS['userbase'] = _getuserbase()
  File "/usr/lib/python2.7/sysconfig.py", line 210, in _getuserbase
    return env_base if env_base else joinuser("~", ".local")
  File "/usr/lib/python2.7/sysconfig.py", line 196, in joinuser
    return os.path.expanduser(os.path.join(*args))
  File "/usr/lib/python2.7/posixpath.py", line 262, in expanduser
    userhome = pwd.getpwuid(os.getuid()).pw_dir
KeyError: 'getpwuid(): uid not found: 1000130000'

Python should be looking at $HOME first as well, so I'm not sure yet why it's falling back to the getpwuid call. Maybe we need to do something to get that environment variable through when Bazel drops to Python.

[stevekuznetsov]

@smarterclayton can comment on security implications of container-side UUID and why OpenShift cares

@BenTheElder might have some comments on running bazel in containers for testing -- much of upstream k8s testing does that, but I am not familiar with it

[wking]

Maybe we need to do something to get that environment variable through when Bazel drops to Python.

Bazel's build_tar invocation sets use_default_shell_env=True. That seems (at least from the design docs) to mean that only environment variables declared with --action_env are passed through to the action executable. I'll update the script to set HOME that way as well.

[wking]

With --action_env as well, the build succeeds. I'll have a PR in both repos for this issue shortly.

[BenTheElder]

k/k prow jobs do typically set $HOME and $USER. For local bazel containers I've also found a fake passwd entry to be useful for eg python and git.

[wking]

@smarterclayton can comment on security implications of container-side UUID and why OpenShift cares

Ah, I think I found the reason over here:

By default, all containers that we try and launch within OpenShift, are set blocked from “RunAsAny” which basically means that they are not allowed to use a root user within the container. This prevents root actions such as chown or chmod from being run and is a sensible security precaution as, should a user be able to perform a local exploit to break out of the container, then they would not be running as root on the underlying container host. NB what about user-namespaces some of you are no doubt asking, these are definitely coming but the testing/hardening process is taking a while and whilst companies such as Red Hat are working hard in this space, there is still a way to go until they are ready for the mainstream.

That blog post is from 2016; so I'm not sure the user-namespace mainstream-ness caveat still applies. But that's probably still why OpenShift is doing this.

[wking]

That blog post is from 2016; so I'm not sure the user-namespace mainstream-ness caveat still applies.

Looks like it does, since Kubernetes still has an open feature for user-namespace support: kubernetes/enhancements#127. That feature has been open for a while, but there's recent related activity in kubernetes/kubernetes#64005. Maybe it will land in 1.12...

[smarterclayton]

Yeah, user namespaces are still not real (in that kube doesn't fully support them being enabled, or deal with the implications correctly when you try to run host level workloads). We could grant the ability to run root containers to the ci infra, but I'm also hesitant to do that because a) that's like giving up on security) and b) would encourage jobs that are dependent on that particular tweak.

Seths team is driving the fixes as you note.

[stevekuznetsov]

All of the bazel jobs upstream use privileged containers FWIW. This gives us more reason to run a separate build cluster :)

[BenTheElder]

You should be able to run without privileged containers if you disabling the sandboxing IIRC, something like --spawn_strategy=local either as an extra flag to the build or in a .bazelrc as build --spawn_strategy=local. We don't do this because the sandboxing is important to ensuring correctness of known inputs for each step, without enforcing that we may cache incorrectly; and we use cluster-level isolation where we are concerned.