Allow running Linux System apps for foreign targets

[rmartin16]

Changes

• Adds support for running apps from within Docker for other distributions
• The strategy uses socat to run a TCP proxy as a spoofed X display to the current X display
  • Most X displays run through a UNIX socket; so, the TCP proxy will normal connect there
  • Although, in the case of SSH X11 forwarding, the X display will be a TCP port provided by a SSH channel
  • X authentication is managed via xauth and is assumed to be using MIT cookies

TODO

• Update bootstraps for new run requirements
• Bump Docker server minimum version to 20.10

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[freakboy3742]

I haven't run the code to check if it works - but if it does... OMGYES. This looks remarkably straightforward and elegant, and it could only be a benefit for testing purposes.

[rmartin16]

After coming to a better understanding of the ways an X11 server can be exposed, this approach seems like our best option.

An alternative straightforward approach is to run the container on the host network and allow the app to connect to the X11 server via SSH. However, this requires a few things of the host:

• Running an SSH daemon
• SSH daemon is configured for X11 forwarding
• X11 connections are enabled beyond the typical default permissions via a command like xhost +local:docker

[edit] Passing X11 access through to Docker does not require SSH.

Alternatively, there's many Docker wrappers that'll spawn a VNC server for the container or even ones that'll stand up a dedicated X11 server for a container.

All of these dependencies on the host machine seem particularly undesirable.

That said, the approach in this PR does not work with Docker Desktop...since it actually runs containers inside a Linux VM and you can't just bind-mount a UNIX socket from the host in to a VM and have things work out.

Additionally, a key piece that allows this to work is the fact the user in our containers masquerades as the host user; if the UID was different, then the host's X11 server would reject the connection with typical permissions configured.

[rmartin16]

The tests are driving me insane...but this implementation is stable if you have any general thoughts before a more thorough review.

note to self: there's concurrency issue with the tests (i.e. tox -e py-fast); something is replacing the real shutil.which

[freakboy3742]

A couple of minor nits, and one architectural question; but otherwise, this makes sense to me.

I still haven't been able to run this, because my current virtualisation setup won't really allow it. I'm running Linux in a VM, so I can't run Docker on Linux; and I can't run system apps on macOS. So, this review still comes with a "..but I haven't tried it" qualifier.

I agree that avoiding the requirement of an SSH daemon and port forwarding is preferable. I'd rather not get into the game of becoming network administrators for Briefcase users, especially given the level of experience that our user base generally has.

The Docker Desktop limitation is unfortunate - and I wonder if that might be what make this infeasible. I don't have a good feel for whether Linux users with Docker are using Docker Desktop or Docker Engine, but the Docker tutorial seems to direct Linux users towards Docker Desktop. Have you got any feel for whether the Desktop limitation will be a serious constraint in practice? Also - if this is a constraint, it feels like we need some sort of protection identifying if the user is on Docker Desktop so they're not left stranded with (I presume) some sort of gnarly bind error.

In terms of the specifics of the PR - the only part that doesn't make sense to me is the use of a context manager to process keyword args. Why isn't the interface a straight up call to Popen on the app context, with the X variable transformation being a straightforward function returning an updated kwargs set? There's no entry or exit logic, no setup or teardown to perform... I'm not sure I see what "context" we actually get.

[freakboy3742]

Minor nit: these should be broken out by line, and explain why webkit is commented out:

Suggested change

	"gtk3", "typelib-1_0-Gtk-3_0",
	# "libwebkit2gtk3", "typelib-1_0-WebKit2-4_1",
	"gtk3",
	"typelib-1_0-Gtk-3_0",
	# Needed if your app uses WebView
	# "libwebkit2gtk3",
	# "typelib-1_0-WebKit2-4_1",

[rmartin16]

n/a. Updated Dockerfile handling to use both system_requires and system_runtime_requires when it creates the image.

[rmartin16]

A couple of minor nits, and one architectural question; but otherwise, this makes sense to me.

I still haven't been able to run this, because my current virtualisation setup won't really allow it. I'm running Linux in a VM, so I can't run Docker on Linux; and I can't run system apps on macOS. So, this review still comes with a "..but I haven't tried it" qualifier.

hmm....nested virtualization on macOS does seem limited right now....although, QEMU on x86 should work...I'm sure that'd be a fun rabbit hole for ya 😉

I agree that avoiding the requirement of an SSH daemon and port forwarding is preferable. I'd rather not get into the game of becoming network administrators for Briefcase users, especially given the level of experience that our user base generally has.

The Docker Desktop limitation is unfortunate - and I wonder if that might be what make this infeasible. I don't have a good feel for whether Linux users with Docker are using Docker Desktop or Docker Engine, but the Docker tutorial seems to direct Linux users towards Docker Desktop. Have you got any feel for whether the Desktop limitation will be a serious constraint in practice?

Hard to say; anyone familiar with Docker would not understand using Docker Desktop on Linux outside of probably development workflows that explicitly use Docker Desktop for its features that specifically target those workflows. That said, Docker seems to like the idea of pushing a more platform-agnostic form of Docker....so, maybe it'll become predominant over time. Although, Docker knows bind-mounting a socket is a missing feature with Desktop...so, maybe they'll add a proxy to support that in the future.

Also - if this is a constraint, it feels like we need some sort of protection identifying if the user is on Docker Desktop so they're not left stranded with (I presume) some sort of gnarly bind error.

When I added support a while back for rootless Docker and Docker Desktop, I could not find a reliable way to determine how Docker is running. Docker created a very generalized infrastructure that could be used to create all kinds of front-ends (and back-ends) for Docker.....and then they made Docker Desktop as one of them.

That said, Docker does seem to use potentially reliable names of their default contexts:

❯ docker context list
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT                                    KUBERNETES ENDPOINT   ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   unix:///var/run/docker.sock                                              
desktop-linux       moby                Docker Desktop                            unix:///home/russell/.docker/desktop/docker.sock                         
rootless            moby                Rootless mode                             unix:///run/user/1000/docker.sock   

❯ docker context show
default

We could at least try to use this as a hint when someone is using Docker Desktop. (Although, maybe this isn't a unreliable as I'm making it out to be....)

If we try to catch the error afterwards, that'll probably be difficult. Docker Desktop doesn't complain about trying to bind-mount the socket....it does as instructed but the bind-mount is unusable as a socket. So, for GTK anyway, it just errors on finding a display to use.

In terms of the specifics of the PR - the only part that doesn't make sense to me is the use of a context manager to process keyword args. Why isn't the interface a straight up call to Popen on the app context, with the X variable transformation being a straightforward function returning an updated kwargs set? There's no entry or exit logic, no setup or teardown to perform... I'm not sure I see what "context" we actually get.

The context manager cleans up the xauth file that's created for the container.

[rmartin16]

Misconceptions

I did some more research today and it turns out the internet is a bit full of cargo culting and misinformation.

Initially, I mentioned that a Docker container can use the host's X server via SSH; this is not true....at least, not in the way it's being presented around the internet. And while I kept seeing this, it bothered me because my machine wasn't even running an SSH daemon...but I figured the X server and clients must be vendoring SSH implementations or something and were making it work.

My suspicion for all this confusion is that 1) "X11 forwarding" is commonly discussed with SSH and 2) a lot of people are trying to run graphical applications within Docker on a remote machine. Therefore, all the different pieces are jumbled together with all the copy/pasting happening everywhere.

Reality

Consider the widely recommended command to run a graphical application in Docker, e.g.:

docker run --rm -it --net host --env DISPLAY ubuntu bash -c "apt update && apt install -y x11-apps && xeyes"

The trick here is understanding what --net host is actually doing and how the X server accepts connections.

Superficially, it gives the container the same access to the network that the host has. But...on modern Linux, the network isn't just the network 🤔. Specifically, --net host configuration puts the container in the same "network namespace" as the host machine; while normally, Docker will create an isolated network namespace for a container to use.

Network namespaces manage access-control not only for network resources but also for "abstract UNIX domain sockets". These are relevant because not only does Xorg's X server expose a socket via the files at /tmp/.X11-unix/X#, but those filepaths are also used to define abstract sockets with the same behavior.

So, while my initial thought was --net host was allowing the container to talk to the X server with TCP, most (or all) distros disable TCP communication with their X server. Instead, this setting is actually allowing the container access to the X server's abstract socket (which you can see via netstat -x) and the DISPLAY environment variable obviously tells it what display to use.

Our Situation

My implementation avoids using --net host and instead directly exposes the relevant UNIX socket for the X server (combined with some X authentication stuff I'm leaving out so we can focus on even just getting a connection).

This approach allows both Docker Engine as well as rootless Docker to use the X server....however, this fails with Docker Desktop because a socket cannot be bind-mounted in to a container in Docker Desktop. Furthermore, though, --net host only works with Docker Engine because neither Docker Desktop nor rootless Docker can actually use the same network namespace as the host....in general, --net host does seem to expose some of the host network...but not at all in the same way.

Therefore, I really only see one solution right now for Docker Desktop: configure a proxy for the X server's socket.

Such a proxy could relatively straightforwardly be set up with socat or even ssh back to localhost. However, to perform this on an arbitrary machine programmatically starts to become a little much perhaps...

note: an example of someone explaining this properly: https://blog.cykerway.com/posts/2021/07/08/run-gui-applications-in-docker.html

[rmartin16]

For posterity, this will expose the X11 socket via TCP:

Run on the host and leave running:

DISPLAY_NUM=$(echo $DISPLAY | cut -d ':' -f 2 | cut -d '.' -f 1)
DOCKER_BRIDGE_IP=$(ifconfig docker0 | awk '/inet / {print $2}')
socat -ddd TCP-LISTEN:$((6000+$DISPLAY_NUM)),reuseaddr,fork,bind=$DOCKER_BRIDGE_IP ABSTRACT-CONNECT:/tmp/.X11-unix/X$DISPLAY_NUM

In a separate terminal, start the container in Docker Desktop:

DISPLAY_NUM=$(echo $DISPLAY | cut -d ':' -f 2 | cut -d '.' -f 1)
DOCKER_BRIDGE_IP=$(ifconfig docker0 | awk '/inet / {print $2}')
docker run --rm -it --net host --env DISPLAY=$DOCKER_BRIDGE_IP:$DISPLAY_NUM ubuntu bash -c "apt update && apt install -y x11-apps && xeyes"

Viola

[freakboy3742]

For posterity, this will expose the X11 socket via TCP:

Run on the host and leave running:

DISPLAY_NUM=$(echo $DISPLAY | cut -d ':' -f 2 | cut -d '.' -f 1)
DOCKER_BRIDGE_IP=$(ifconfig docker0 | awk '/inet / {print $2}')
socat -ddd TCP-LISTEN:$((6000+$DISPLAY_NUM)),reuseaddr,fork,bind=$DOCKER_BRIDGE_IP ABSTRACT-CONNECT:/tmp/.X11-unix/X$DISPLAY_NUM

That... doesn't seem so bad - and it would seem to lend itself to being something that Briefcase manages as aPopen that is cleaned up when the process exits...

Also, throwing another spanner in the works: Wayland.

(you're welcome :-)

[rmartin16]

That... doesn't seem so bad - and it would seem to lend itself to being something that Briefcase manages as aPopen that is cleaned up when the process exits...

Yeah....although, I'm trying to replicate this on Fedora and running in to issues with sharing the socket over TCP through the docker0 bridge interface. Potentially resolvable...

Also, throwing another spanner in the works: Wayland.

True; however, Wayland runs a Xwayland process that mimics an X environment including these sockets. So, I imagine for the foreseeable future, this will continue being the case....that said, my initial testing with Fedora 39 isn't going well with wayland sooooo.....more fun I'm sure

[rmartin16]

I was able to resolve my networking woes....but it required a much deeper understanding of networking with Docker Desktop.

Background

The original design of Docker includes the docker0 bridge interface; by default, all containers are attached to it and it facilitates communication among the containers as well as the host (in limited ways anyway) and the broader internet. Abstractions on top of Docker, e.g. docker-compose, will create separate networks for each container and in many ways obviate docker0....since arguably, having any random container be able to talk to any other random container by default doesn't really scream "isolated and secure".

At any rate, Docker Desktop (notoriously at this point) runs its containers in a Linux VM; therefore, the default docker0 bridge interface is a virtual interface inside of the VM....so, this is why you can't bind a server to the docker0 interface on the host and have it available inside the container on the docker0 interface....because they are two completely different and isolated interfaces. (It was only working on my development machine because my host docker0 interface is using a different subnet than the docker0 in Docker Desktop and my host's iptables were routing the requests....that said, even when I changed the subnet for docker0 in Docker Desktop on Fedora, I still couldn't access the host...so, this seems unreliable at best).

Accessing network-based services on the host

Finally, I discovered Docker's solution for this. In Docker Desktop, a DNS server is running in the VM and it is mapping host.docker.internal to the network assigned IP of the host machine. Docker provides this facility specifically to support development workflows where a server is running on the host and containers need access to it.

However, it is only available by default when using Docker Desktop; so, when just using Docker Engine, the docker run command needs to be augmented with --add-host host.docker.internal:host-gateway. This adds an entry to the /etc/hosts file in the container that directs host.docker.internal to the IP of docker0. (Although, if you add this while using Docker Desktop, it is the IP assigned to the host machine from the network.)

Conclusion

The X socket can be exposed via TCP but it must be binded to an address available to the container. For Docker Desktop, this will need to be the host's network assigned IP. For Docker Engine, this would need to be the IP of the default Docker bridge (unless we create a dedicated bridge interface for this purpose). Alternatively, we could just bind to 0.0.0.0.

Additionally, the minimum Docker version must be bumped to 20.10 (released dec 2020) to ensure host.docker.internal is available.

[rmartin16]

I modified the implementation to use a TCP proxy for the host's X socket. However, my initial testing is showing that X authorization is seemingly being ignored entirely now. While the proxy was open on my main machine, I could open windows on it from a VM running on an entirely different machine 👀 sooo....I'll need to figure out what's going on...or accept this is just really insecure, I guess.

[edit]
I am getting authentication failures when I try to connect to other machines....so, I must have auth disabled on my main machine currently or something.

❯ DISPLAY=192.168.122.191:0 xeyes
Invalid MIT-MAGIC-COOKIE-1 keyError: Can't open display: 192.168.122.191:0

Another thought is users running Briefcase over SSH....I'll have to see how this could even be adapted for X11 forwarding...

[rmartin16]

Yet more evolved thoughts on this design

1. If an existing TCP server for the X display exists, it should be used
   • The X11 standard reserves ports 6000 + <$DISPLAY> for this; so, for $DISPLAY=:10, the port is 6010
   • Such a TCP server should be exposed to the Docker container and specified in the container's DISPLAY env var
   • The most likely scenario for this is an SSH connection with X11 forwarding configured
   • However, such an existing TCP server would likely be bound to the loopback interface...therefore, it would likely be necessary to setup a TCP proxy bound to the other interfaces back to the loopback interface
   • Doing so may require spoofing a new X display....which complicates authentication but that can be worked around with a little parsing of the xauth commands
2. Otherwise, if a /tmp/.X11-unix/X# socket is available, use that
   • Initially, I had the TCP proxy connecting to the abstract socket, but those aren't portable beyond Linux; so, just connect the proxy to the file-based UNIX socket

Notes

• The use case of proxying an existing TCP connection for X is debatable
  • Although, I'd like to see this all working even for X11 forwarding
  • In theory, it shouldn't be too must more difficult than the proxy for the UNIX socket....I'll have a better idea once I have it working
• TCP proxy cannot just be bound to the loopback interface
  • From my testing and research, the proxy will likely have to be exposed to the network so the Docker container can access it
  • For Docker Engine and rootless Docker, it would be possible to just bind the proxy to the docker0 interface
  • But for Docker Desktop, AFAICT, the host's actual network interface is attached to a network defined for the VM; from there, a network bridge is created in Docker to expose this network from the VM.
    • So, unless a service on the host is binding to the "real" network interface, it won't be available in the container
    • This is obviously in contrast to Docker Engine where the network interface, docker0, is literally the same thing inside of a Docker container and outside

[freakboy3742]

Why "in docker"? Won't this be a requirement regardless - just one that virtually every Debian install will have by default?

[rmartin16]

Won't this be a requirement regardless

It's only a requirement to run the app....not to build it. If you aren't using Docker, then Briefcase will tell you the system_runtime_requires list needs to be installed first.

But I guess this brings up a larger point....

By adding these runtime requirements to system_requires, we're really co-opting its purpose which to specify the build-time requirements.

Maybe we should just have the Dockerfile install both...

[rmartin16]

Instead of duplicating the runtime requirements in to system_requires, I just updated the Dockerfile invocation to install both system_requires and system_runtime_requires. This makes so much more sense in retrospect.

[freakboy3742]

A couple of tweaks to the internal docs, and two related comments related to the socat/xauth error message - but otherwise, this is looking really solid.

The socat/xauth comments came as a result of being able to actually test this on a real live Ubuntu machine. It worked perfectly. socat wasn't installed, and it returned the expected error message; once it was installed, the app started as expected.

However - when I got the error, I wasn't 100% certain what needed to be installed. It wasn't too hard to guess apt install socat, but that might not be obvious to all, and the equivalent might not be correct on Fedora/Arch etc.

[freakboy3742]

A couple of minor markup and wording tweaks; but otherwise this looks good to go!

Nice work - this is a complex piece of plumbing, but the end result is really elegant.