[Feature Request] Deploying externally from LocalArchitect

[JackUrb]

Overview

At the moment, we're able to launch via the LocalArchitect to run tasks for testing purposes, or potentially over an internal network where everyone can directly access your address. We'd also like to be able to directly expose a server deployed by the LocalArchitect to the internet, thus if you're running Mephisto on a server that can be web accessible, you don't need to set up a separate routing server.

Details

This setup has a few requirements:

1. The running server needs to be exposed on port 443 (and perhaps also port 80).
2. The running server needs SSL support
3. Mephisto needs to know the public hostname

Each has a few things to address to work properly, and below are some possible approaches.

Running on ports 443/80

Generally it's not advised to run a Python process directly on these ports. In fact, without running python as root, you're not allowed to bind to these ports in the first place. As such ideally we run the Mephisto server itself on some given port (like 8080) and then route the traffic to root elsewhere.

In the end we'll likely want to set up a bash script that configures nginx as a reverse proxy, going through the process of writing configuration files if they don't already exist, testing them, and rebooting the system. We'll have to sudo run the bash script for setup, and people will need to enter their password for the script to run. Note, it's possible that those running Mephisto already have a setup for having the reverse proxy running, in which case we'll want to let people skip this step if they want.

Note, someone may even get to the point of configuring their nginx setup to serve Mephisto at the same server that they're already hosting other things, but with a different domain name. We need to not stomp the nginx (or other) configuration just in case there's already something there, even if someone accidentally tries to use the automated setup.

Setting up SSL support

Setting up SSL is a more complicated step, and may require that the user configure their domain or hostname. This may be split into a few different cases, depending on if the person setting up their system already has a domain name in mind. For now we can provide a few different configuration options. Realistically this gets pretty complicated based on how complicated the underlying server is. We'll assume a one-host setup.

Don't have a domain name yet

We should be directing users to a domain name that they can use for all of their tasks moving forward. At the moment it's not possible to directly assign certificates for the public domains on an EC2 instance (or other cloud providers for that matter), so we need people to get their own domains for this to work. It's possible that we refer people to manually get a domain name if they haven't configured one yet, perhaps directing them to a free domain source? Should probably vet further.

After getting a domain name

If the above situation doesn't work (ie, we're not running on an Amazon EC2 instance, or we don't have a domain), the user will need to get a domain from somewhere, and then can register it using Certbot. This can be invoked as part of the setup script.

Knowing the public hostname

This ultimately can just be a config option, but it's required that during the setup Mephisto ensures that it's possible to reach itself via the public address at both of the publicly available ports. It's also possible that knowing the hostname will be useful in helping setup both the SSL support and the reverse proxy.

Other

It's also important to note that one will likely only be able to run a single Mephisto job from a publicly accessible LocalArchitect server at a time, due to the way we have to set up routing. For this if someone tries to launch a second job for a publicly accessible LocalArchitect, we should error out.

Discussion:

Some early discussion so far can be found in #303, and even earlier in ParlAI here.