Consider using unix sockets instead of localhost for Tor Hidden Services

[garrettr]

Background

SecureDrop currently uses the "standard" configuration of "expose service on localhost, then connect that to a hidden service via torrc" (see install_files/ansible-base/roles/tor-hidden-services/templates/torrc).

This configuration is problematic, because applications sometimes (fairly) assume that localhost has a different threat model than an externally exposed service. Most recently, this common configuration lead to some hidden services exposing sensitive information through the Apache mod_status module (link). SecureDrop was never vulnerable to this particular issue because we have been aware of it for years and accordingly disable mod_status as well as the rest of the default Apache site configuration.

Nonetheless, I think this is a generally problematic configuration due to the mismatch in the intended use of localhost vs. its common (but frankly, inappropriate) use for externally exposed Tor Hidden Services. I am concerned that other, similar issues may be lurking.

Alternatives to localhost

Riseup has a nice page on deploying hidden services (down right now, here's a cached copy) that discusses this issue in depth. Here are their proposed alternatives to localhost:

---

Be careful of localhost bypasses!

You should take very careful care to not accidentally expose things on your server that are restricted to the local machine. For example, if you provide /server-status in apache (from mod_status) to monitor the health of your apache webserver, that will typically be restricted to only allow access from 127.0.0.1, or you may have .htaccess rules that only allow localhost, etc.

There are a few ways you can solve this problem:

different machine: consider running the onion service on a different machine (real or virtual) than the actual service. This has the advantage that you can isolate the service from the onion service (a compromise of one doesn’t compromise the other) and helps with isolating potential information leaks

isolation: similarly to the above, you can also isolate tor and the service so it will run on a different network namespace than the service.

public ip: configure the onion service to connect to the public IP address of the service instead of localhost/127.0.0.1, this should make tor not pick 127.0.0.1 as the source address and avoid most misconfigurations. For example like this:
bc. HiddenServiceDir /var/lib/tor/hidden/ftp/
HiddenServicePort 80 192.168.1.1:81

unix socket: consider using unix socket support instead of a TCP socket (requires 0.26 or later tor) – if you do this, then the onion service will be running on the same server as the service itself. With a socket approach, you should be able to run with privatenetwork=yes in systemd unit which gets you some really great isolation, for example:
bc. HiddenServicePort 80 unix:/etc/lighttpd/unix.sock

but then the service itself needs to support unix sockets, otherwise you have to setup some socat redirection from tcp <→ unix (nginx, twisted, lighttpd all support this)

audit carefully: carefully audit, and regularly re-audit your system for configurations that allow localhost/127.0.0.1, but prohibit everywhere else and configure those to work around the problem (for example make /server-status operate on a different IP; make the webserver listen on a different port for /server-status; make it password protected, etc.)

---

Unix Domain Sockets

Of the options above, I am most interested in Unix domain sockets because they can provide an additional layer of isolation protection through privatenetwork=yes.

I initially thought this would provide a layer of defense in depth for attackers attempting to exfiltrate sensitive data (since they would no longer be able to make arbitrary outgoing Tor connections), but I think they could still use the web application itself as a mechanism to exfiltrate, so this might not be as helpful as I thought it was when I started writing this. This warrants further research.

[eloquence]

Given that network access from within the application server is required to exploit the use of localhost, we're not considering to prioritize a change to the current architecture, consistent with our threat model.

[cwebber]

Funny timing with this being closed, I was just looking up how to try to lock down a system I'm building using tor hidden services to only working through unix domain sockets and accidentally ended up here.

@eloquence, while the statement you made about "requires network access to localhost already" seems to make sense, you may be surprised at how many confused deputy attacks are found through assuming localhost + port is an adequate security model. Web clients tend to be open to performing confused deputy attacks against them. Here's some examples:

• my favorite (though disclosure that I was involved in uncovering it), an attack that allowed any browser-using user to have arbitrary scheme code run against them if they were using the guile live repl and were directed to a malicious webpage
• likewise, exfiltrating databases

I bet there is a confused deputy attack here.

[emkll]

Reopening this issue as it would be worth investigating for defense-in-depth purposes:

While it is true that, in the current model, these types of attacks would require a local attacker (due to hardware firewall requirements, iptables rules and apache configuration), using unix sockets would allow to further enforce least privilege using filesystem permissions, and defense-in-depth in the event of a network or webserver misconfiguration.

[kushaldas]

Here are the minimal steps for doing this using nginx + uwsgi, for the PoC purpose, I am modifying a Focal staging instance.

Install nginx via apt

sudo apt install nginx -y

We have to use nginx for this as Apache can not listen on an unix socket, but only on ports on IPv4 or IPv6.

Install uwsgi in our virtualenv

But, first we had to install a few more packages because we had to build the wheel. Will not be required for production.

apt install python3-pip python3-dev build-essential libssl-dev libffi-dev python3-setuptools

Then installing the actual package from pypi.

source /opt/venvs/securedrop-app-code/bin/activate
python3 -m pip install whell
python3 -m pip install uwsgi

Create sourceng.py and journalistng.py for entry point

Path: /var/www/securedrop/sourceng.py

import os
import sys

os.environ["CRYPTOGRAPHY_ALLOW_OPENSSL_102"] = "True"
sys.path.insert(0,"/var/www/securedrop")

import logging
logging.basicConfig(stream=sys.stderr)

from source import app
if __name__ == "__main__":
    app.run()

Path: /var/www/securedrop/journalistng.py

import os
import sys

os.environ["CRYPTOGRAPHY_ALLOW_OPENSSL_102"] = "True"
sys.path.insert(0,"/var/www/securedrop")

import logging
logging.basicConfig(stream=sys.stderr)

from journalist import app
if __name__ == "__main__":
    app.run()

Create .ini configuration files for both the services

Path: /var/www/sourceng.ini

[uwsgi]
module = sourceng:app

master = true
processes = 5

socket = /var/run/wsockets/source.sock
chmod-socket = 660
vacuum = true

die-on-term = true

Path: /var/www/journalistng.ini

[uwsgi]
module = journalistng:app

master = true
processes = 5

socket = /var/run/wsockets/journalist.sock
chmod-socket = 660
vacuum = true

die-on-term = true

Important point here is the socket path.

Set systemd service for the both

Path: /etc/systemd/system/sourceng.service

[Unit]
Description=uWSGI instance to serve source interface
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/securedrop
Environment="PATH=/opt/venvs/securedrop-app-code/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
ExecStart=/opt/venvs/securedrop-app-code/bin/uwsgi --ini /var/www/sourceng.ini
RuntimeDirectory=wsockets

[Install]
WantedBy=multi-user.target

Path: /etc/systemd/system/journalistng.service

[Unit]
Description=uWSGI instance to serve journalist interface
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/securedrop
Environment="PATH=/opt/venvs/securedrop-app-code/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
ExecStart=/opt/venvs/securedrop-app-code/bin/uwsgi --ini /var/www/journalistng.ini
RuntimeDirectory=wsockets

[Install]
WantedBy=multi-user.target

Start the services one by one

systemctl daemon-reload
systemctl start sourceng
systemctl status sourceng
systemctl start journalistng
systemctl status journalistng

Remember to check for any error at this state.

Remove the default nginx settings.

rm /etc/nginx/conf.d/default

Create source nginx server setting

Path: /etc/nginx/sites-available/sourceng.conf

server {
    listen unix:/var/run/nginx.sock default_server;
    #listen 80 default_server;
    server_name 127.0.0.1;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/var/run/wsockets/source.sock;
    }

    location /static {
      root /var/www/securedrop;
    }
}

ln -s /etc/nginx/sites-available/sourceng.conf /etc/nginx/sites-enabled/

Path: /etc/nginx/sites-available/journalistng.conf

server {
    listen unix:/var/run/nginxj.sock default_server;
    #listen 80 default_server;
    server_name 127.0.0.1
                192.168.122.225;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/var/run/wsockets/journalist.sock;
    }

    location /static {
      root /var/www/securedrop;
    }
}

ln -s /etc/nginx/sites-available/journalistng.conf /etc/nginx/sites-enabled/

Problem in restarting nginx service

One thing I noticed that someimes the socket files are there in the /]var/run directory while stopping and starting the service, I had to remove those by hand before nginx could start properly.

Update torrc

HiddenServiceDir /var/lib/tor/services/sourcev3
HiddenServicePort 80 unix:/var/run/nginx.sock

Restart Tor. check for errors.

[emkll]

Thanks @kushaldas for the investigation and detailed writeup.

Based on the above, it seems like nginx is a hard requirement here? Is there a way to do this using Apache? Given the short timeframe for the update to Focal and the relatively large amounts of features to port (app separation, https certs, etc, it seems unrealistic to migrate the webserver to nginx from Apache. What do you think?

[kushaldas]

Based on the above, it seems like nginx is a hard requirement here? Is there a way to do this using Apache?

Yes, Apache can not listen on unix sockets, nginx can.

Given the short timeframe for the update to Focal and the relatively large amounts of features to port (app separation, https certs, etc, it seems unrealistic to migrate the webserver to nginx from Apache. What do you think?

The current timeline is hard for this move, and my personal suggestion would be not to try this.

Hopefully sometime next year, we will be in a better shape timeline wise to move all the logic from the postinstallation scrip into ansible playbooks inside of the app server, which would run in the post-inst. Thus making sure that the system is in a stable shape, without any input from the admins.

[kushaldas]

I still think that we should postpone this for a release during or after 2.0.