Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve SecureDrop loading experience over Tor Hidden Services #1152

Closed
4 tasks
garrettr opened this issue Oct 19, 2015 · 10 comments
Closed
4 tasks

Improve SecureDrop loading experience over Tor Hidden Services #1152

garrettr opened this issue Oct 19, 2015 · 10 comments

Comments

@garrettr
Copy link
Contributor

In production, all of SecureDrop's web interfaces (the Source Interface and the Document Interface) are only available as Tor Hidden Services, which have extremely restricted bandwidth. In addition, SecureDrop intentionally avoids caching any resources in the client's browser to avoid potential forensic infoleaks.

These qualities combine to make the SecureDrop web interface feel slow and glitchy, particularly when used over an especially slow hidden service connection:

securedrop source interface performance

Specific Issues to Fix

  • Seeing the unstyled page before the CSS loads. This looks glitchy and unprofessional.
  • Seeing the logo resize during the page load
  • Reduce or eliminate the delay in showing the icons (e.g. cloud icon next to "Submit Documents" on the home page)
  • Reduce or eliminate delay on page loads after the first in a session. This requires research, as it conflicts with our current position on avoiding caching web resources, and we would need to make sure that we can use some form of caching without the browser writing the resources to long term storage.

Potential Resolutions

  • Reordering or inlining resources (stylesheets, images)
  • Reduce the size of currently used resources to load them faster with the limited bandwidth we have, e.g.
    • Reduce the size of logos
    • Reduce the icons files to just the icons that are actually used in SecureDrop
    • Minify JS and CSS
  • Caching
@garrettr
Copy link
Contributor Author

I started working on this, but ran into an issue with the development cycle and our existing set of automated development environments. Rather than work around it, I think this is an instructive use case that could help us improve our development environments and automation.

Essentially, the problem is:

  1. The development environment synchronizes changes to the application code (securedrop/securedrop) to the VM, so it automatically synchronizes the type of changes I need to make (changes to HTML templates, CSS, images, JS, etc.). However, it does not make the site available via Tor Hidden Service.

  2. The staging environment does make the site available via Tor Hidden Service, but it installs securedrop-app-code from the production apt server. In order to use staging to test my changes, I would need to:

    1. Stand up a testing apt server, edit staging-specific.yml and associated files to use that instead of the production apt server.
    2. Make changes.
    3. Rebuild the securedrop-app-code deb locally, using the build vm
    4. Upgrade the testing apt server with the new, testing deb.
    5. Re-run the Ansible playbooks to install the new deb.

    This procedure is manual, complex, and time-consuming.

The install_local_packages Ansible role was intended to handle this development workflow use case, but it is currently broken and there has been some work to replace/remove it (#1040).

@conorsch What do you think the best next move is here?

@conorsch
Copy link
Contributor

@garrettr Ah, a subject near and dear to my heart. The lack of automated building as part of the staging environment makes it virtually useless for testing the app code—the staging environment is essentially a more convenient production, but with more verbose logging and direct SSH access. It's appropriate for testing the deployment process, but not the app code.

The solution is to trigger deb package builds during the staging deployment, and use that artifact during provisioning. I've tabled the install_local_pkgs rewrite until I can move many pieces around together. Most of the ansible reorg is finished, but not submitted for review yet, because it's a vicious diff, and would likely require several days of in-person review.

Let me take a look at providing a wrapper playbook with the current state of the ansible config that would essentially chain the build-deb-package-then-provision-staging workflow. That should be shippable with a far less burdensome review process, and provide you with a less hair-pulling process to test bleeding-edge code over Tor.

@garrettr
Copy link
Contributor Author

@conorsch The alternative is to make resolving this issue a dependency of the Ansible reorg, which I'm totally fine with. There's a lot of other stuff I could be working on, I just picked this one up since I spent so much time thinking about the issue while filing it and it seemed fairly straightforward.

@conorsch
Copy link
Contributor

@garrettr Upon closer inspection, there's a more direct path that works with the current config. Try to reproduce and let me know how it treats you.

# I say we take off and nuke the entire site from orbit.
# It's the only way to be sure.
vagrant destroy -f build /staging/

SECUREDROP_STAGING_SKIP_TAGS='untagged' vagrant up build /staging/

That's it. The build role will run first, and dump out new packages. Then the staging playbook will kick in, and since you've overridden the default of "don't install local packages", the local packages will be used during the staging provisioning run.

The SECUREDROP_STAGING_SKIP_TAGS='untagged' is a kludge (it actually only works because of the thorough tagging from #1004), and I suggest we remove the corresponding line from the Vagrantfile all together. Removing it would require that folks spinning up a staging environment also spin up a build VM, but I'm comfortable enforcing that as a development practice.

The current implementation of the install_local_pkgs role isn't great, definitely lacks a sufficient dollop of magic sauce, since it still installs the prod packages from the apt repo, then installs the deb packages over top of them, but that problem is better addressed in the overall ansible reorg. The solution described above should get you up and running for comparing development code performance over Tor and local HTTP via port forwarding.

@conorsch
Copy link
Contributor

Some rough benchmarking info: destroying and recreating build and /staging/ takes about 15 minutes on my machine (i5, 8GB, SSD). Running SECUREDROP_STAGING_SKIP_TAGS='untagged' vagrant provision build /staging/ takes only 3 minutes.

@garrettr
Copy link
Contributor Author

Tried to reproduce. I got an error:

TASK: [install_local_pkgs | if defined copy local deb packages to target system] *** 
fatal: [mon-staging] => input file not found at /Users/garrett/code/freedomofpress/securedrop/install_files/ansible-base/roles/build/ossec-server-2.8.2-amd64.deb or /Users/garrett/code/freedomofpress/securedrop/build/ossec-server-2.8.2-amd64.deb

FATAL: all hosts have already failed -- aborting

I will copy the ossec debs (which are built by an Ansible role currently maintained in the separate ossec repo) to securedrop/build and try again.

@conorsch
Copy link
Contributor

Discussion of the ossec debs is over in the ossec repo, with a proposed solution to the clunky error messages.

@conorsch
Copy link
Contributor

Changes to the install-local-packages role tracked in #1040. Have a branch ready to submit that will resolve the rebuilding-on-staging problems described in the comments above.

@heartsucker
Copy link
Contributor

Is using mod_pagespeed an option? I haven't used it yet, but we're considering using the Nginx flavor at work for something.

@eloquence
Copy link
Member

Closing; the behavior described here no longer occurs with current versions of SecureDrop, thanks to improvements that have since been made to templates and styling, and likely also in part due to upstream changes in Firefox ESR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants