Ansible intro, wishlist and bugs

[traumschule]

Since November 2017 we use ansible to test and deploy Hitchwiki. It can provision a local vagrant box or deploy to a remote server. Feel free to subscribe and add reports related to our ansible playbooks!

• HW master:
• master:
• ansible:
• testing:

Test it!

git clone https://github.com/traumschule/hitchwiki -b testing
./scripts/vagrant/install.sh
./scripts/deploy_remote.sh HOST

For details see INSTALL.md and ansible/README.md.

See it live on beta.hitchwiki.org (experimental: 6pna4byhdcdyprdc.onion)

Workflow

feature -> testing -> ansible -> HW master -> production

• feature branches should rebase testing, if test with travis and vagrant succeed, they are merged
• testing: runs CI build tests for added latest features (test latest features and make sure all methods install before merging into ansible)
• ansible: to be run with vagrant (development)
• master: (to be) synced with current HW master branch
• beta: beta.hitchwiki.orgis deployed from testing or ansible branch
• production: hitchwiki.org has it's own repository and is updated after some time when beta looks stable enough

To be merged into ansible branch

• branch:torhs set up a tor hidden service (why? facebook and NYT offer it, so we should too)
  • hidden service v2: 6pna4byhdcdyprdc.onion
  • hidden service v3: ncwpcjalalxyxj2247b6ts45dsqlt6ihi5dmb6uuv5c5e45pkfgxm6qd.onion (needs TBB 7.5.x)
  • [-] MW ignores the requested url and rewrites it to beta.hitchwiki.org. To preserve the .onion address a proxy is necessary $wgServer is commented out now because it is detected automatically
• turn off logging of IP addresses in apache (removeip module)
• add vagrant-cachier

In progress

Tasks for HWv3: project roadmap, upcoming: #136 HW gathering in Berlin

development is currently stalled because of a bug in vagrant

Would be nice

• re-enable maildev
  • status.sh: check why maildev and monit are not configured in docker and the started test for maildev and phpmyadmin fail
• 🚀 speed up installation
  • use docker compose to create a HW image
  • run jobs in parallell
  • apply perfomance tuning tipps
  • optimize downloads (caching)
• setup discourse with foreman - see scripts/ansible/roles/discourse/tasks/main.yml
• run rgrep 'TODO' scripts/ configs/to check the code for TODOs
• Remove Beta features menu #132 disable beta menus for prod
• test MW 1.30 (tar, .sig): upgrade
• apache configs for each domain (currently we have domain and domain2 in settings. if both are set it could make sense, to create separate apache config files with vhosts for port 80 and 443)
  • restructuring the domain variable(s) affects following files:

scripts/configs/settings-example.yaml
scripts/configs/parsoid_config.yaml
scripts/ansible/roles/discourse/tasks/main.yml
scripts/ansible/roles/hitchwiki/tasks/domain.yml
scripts/ansible/roles/hitchwiki/tasks/mediawiki.yml
scripts/ansible/roles/hitchwiki/tasks/dev.yml
scripts/ansible/roles/hitchwiki/tasks/tls.yml
scripts/ansible/roles/hitchwiki/tasks/letsencrypt.yml
scripts/ansible/roles/hitchwiki/tasks/tls_selfsigned.yml
scripts/ansible/roles/status/tasks/main.yml

• HW dump import
• honor dev: beta as a pre-step towards production with TLS and maildev enabled

Security and stability

• use ssh jail for deployment via travis
• check why secure db settings for production lock us out of the dbserver
• MW maps hack: scripts/ansible/roles/hitchwiki/tasks/mediawiki.yml: # TODO: any solution that is cleaner than this temporary dirty hack..
• explain ignore_errors: yes: run rgrep 'ignore_errors: yes'and check that every line has a comment with a good explanation
• [-] use vault for passwords, restructure variable tree as encrypted passwords are not visible to php (maybe db settings for production)

TLS

• test TLS config (see ssl-params.conf)
• add missing HTTP headers (see Content Security Policy)

Strict-Transport-Security | HTTP Strict Transport Security is an excellent feature to support on your site and strengthens your implementation of TLS by getting the User Agent to enforce the use of HTTPS. Recommended value "strict-transport-security: max-age=31536000; includeSubDomains".
Content Security Policy is an effective measure to protect your site from XSS attacks. By whitelisting sources of approved content, you can prevent the browser from loading malicious assets.
X-XSS-Protection sets the configuration for the cross-site scripting filter built into most browsers. Recommended value "X-XSS-Protection: 1; mode=block".
Referrer Policy is a new header that allows a site to control how much information the browser includes with navigations away from a document and should be set by all sites.

• read scripts/ansible/roles/hitchwiki/tasks/tls_selfsigned.yml:# TODO https://www.jeffgeerling.com/blog/2017/self-signed-certificates-ansible-local-testing-nginx
• scripts/ansible/roles/hitchwiki/tasks/tls.yml:# TODO check for release of: Let's Encrypt - apache mod md (We use Certbot which works well, so there currently is no reason to change it.)
  • scripts/ansible/roles/hitchwiki/tasks/tls_md.yml:# TODO untested - "unless it's at least beta-level, we should use just letsencrypt's certbot+cron"
• use ansible's letsencrypt module (not needed because we register via Certbot in one step and set a cron job to renew)

Tests

• run idempotence test on debian and ubuntu (stable and texting)
• test our playbook with ansible 2.4.1.2 and 2.5 (trunk)
• [-] test with MW trunk (install MW from git)
• [-] integrate tests with AnsibleCheck (HOWTO)
• [-] run docker and travis locally/on beta (travis client)

Later / low priority

• figure out why cached facts are not loaded automatically when stored in /etc/ansible/facts.d/file.yml
• we can switch from spyc to symfony (both are already implemented. it's merely a performance decision, see "Load Hitchwiki Config" in mediawiki.php), affected files:
  • mediawiki.php`
  • status.sh
  • hitchwiki/tasks/main.yml
  • system.yml
  • composer.json
  • mw-postgres: discourse needs postgres, to save ram it might be worth to use postgres for MW as well
  • nginx: is nginx an option and what would be the benefits?

Future OS

• currently xenial is LTS stable, to change later, uncomment other versions in .travis on a new branch and check if they build with travis
• check why php-apcu is not in ubuntu artful
• use package module if we deploy to non-debian derived systems (probably won't happen)

[guaka]

see #186