-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Troubleshooting and Performance Guide
For server-side issues, always check the HHVM and NGINX logs. These are located by default at:
/var/log/nginx/error.log
/var/log/hhvm/error.log
You can additionally modify: /etc/hhvm/server.ini to utilize a debugger with HHVM. Instructions can be found here.
For client-side issues, check the Development Tools Console in Firefox or Chrome for any warnings or errors.
First ensure the platform is configured correctly by utilizing the following steps:
- Ensure Ubuntu 16.04 x64 (Xenial) is being utilized. The platform is only supported on this distro.
- Ensure no other HTTP servers are installed, such as Apache. The platform utilizes NGINX. Restart the system after any HTTP server changes.
- Disable any Ad Blocker software on your client.
- Carefully read the provisioning script output to ensure there are no errors during installation.
- Examine the output of
curl -vk https://localhoston the platform system.
Next, review the server-side logs and client-side errors, by utilizing the details from the section above, Debug general issues with FBCTF.
In order for teams to capture levels, several options must be enabled in the admin panel:
- Ensure the Game is not Paused or Stopped.
- Ensure Scoring is enabled.
- Ensure the Team in question is enabled.
- Remind Teams that all Flags are case sensitive.
- Remember: Non-Visible Teams will not show up on the scoreboard.
In production mode, FBCTF utilizes HHVM in Repo Authoritative Mode. This compiles the PHP source files into a bytecode repo, increasing speed and efficiency. When code changes are made to the PHP source files, the HHVM repo must be rebuilt. After making your code changes, run the following commands to rebuild the HHVM repo:
sudo rm /var/cache/hhvm/hhvm.hhbc
sudo hhvm-repo-mode enable "/var/www/fbctf"
sudo chown www-data:www-data /var/cache/hhvm/hhvm.hhbc
sudo service nginx restart
sudo service hhvm restart
Client side code changes will require you to run grunt:
cd /var/www/fbctf
grunt --force
Many changes in the platform will require a flush of Memcached. This is utilized in order to increase performance. When certain actions are taken such as manual changes to the database or data imports, it should be flushed either through the controls admin page, or via command line:
echo 'flush_all' | nc localhost 11211
When troubleshooting issues with the FBCTF platform, you may find it helpful to enable the general query log. This log will show all queries for MySQL. To enable the general query log, edit the MySQL configuration file:
sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
Locate the lines below, ensure they are not commented out, and take note of the log file location.
general_log_file = /var/log/mysql/mysql.log
general_log = 1
Restart the MySQL process with the following command:
sudo service mysql restart
Please note that usage of the general query log will have a negative impact on performance, and should not be used in a live game.
While MySQL binary logs can assist with data recovery and allow for data replication, they also inhibit database performance. To disable bin logging, edit the MySQL configuration file:
sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
Comment out any lines beginning with log_bin, then save the MySQL configuration file. Restart the MySQL process with the following command:
sudo service mysql restart
By default, MySQL is set to a connection limit of around 100. It is recommended that you increase the value for larger events. To increase the MySQL connection limit, edit the MySQL configuration file:
sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
Locate the below line, ensure it is not commented out, and edit the max connections as desired. Note that a single user has approximately two connections at any given time. Save the MySQL configuration file.
max_connections = 100
Restart the MySQL process with the following command:
sudo service mysql restart
Production servers intended for large events must be tuned properly to perform up to expectations and avoid system limits.
Information on the supported kernel values can be found here.
Open the sysctl.conf file in a file editor:
sudo vi /etc/sysctl.conf
Edit the following options to suite your needs, and add the lines to the sysctl configuration file:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_syncookies = 1
net.core.somaxconn = 1024
net.core.netdev_max_backlog = 2000
kernel.sched_migration_cost_ns = 5000000
fs.file-max = 1000000
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_local_port_range = 10000 61000
Apply and save your sysctl changes:
sudo sysctl -p
Production servers intended for large events must be tuned properly to perform up to expectations and avoid system limits. Configure the appropriate interface(s) in order to better handle traffic.
Documentation for network tuning can be found here.
Edit the txqueuelen to suite your needs and run the following command to initiate immediate changes to your interface(s):
sudo ifconfig [interface] txqueuelen <value>
Note that this change will not persist through a reboot. In order to persist this change, Edit the rc.local file:
sudo vi /etc/rc.local
Add the following line and save:
/sbin/ifconfig [interface] txqueuelen <value>
Production servers intended for large events must be tuned properly to perform up to expectations and avoid system limits. Each network socket will require a file handler, so this limitation has a direct impact on the number of connections, and therefore users, the platform will concurrently support.
Edit the limits configuration file in order to increase the open file handles limit:
sudo vi /etc/security/limits.conf
Edit the following lines to suite your needs and enter them into the file in order to increase both the soft and hard limits:
* soft nofile 1000000
* hard nofile 1000000
Reboot your server afterwards to ensure these changes take effect with all appropriate processes.
sudo reboot
Ensure the value set here matches that within the sysctl.conf file set previously.
Production servers intended for large events must be tuned properly to perform up to expectations and avoid system limits. Add the HHVM and NGINX ulimit values to ensure these processes have a sufficient number of open file handles:
sudo vi /etc/default/nginx
sudo vi /etc/default/hhvm
Edit the value to match what you previous set in the limits.conf file and add the following line to both files and save.
ULIMIT="-n 1000000"
Ensure you restart both HHVM and NGINX after making this file change.
sudo service hhvm restart
sudo service nginx restart
Also ensure you are making the ulimit changes in /etc/security/limits.conf and the sysctl.conf file.
Production servers intended for large events must be tuned properly to perform up to expectations and avoid system limits.
Documentation for the supported configuration values of NGINX can be found here.
Modify the values to suite your needs and make the following changes to the NGINX configuration file in order to ensure optimal performance:
sudo vi /etc/nginx/sites-enabled/fbctf.conf
Update the top level section:
worker_processes auto;
worker_rlimit_nofile 1000000;
Update the events section:
worker_connections 500000;
multi_accept on;
use epoll;
Update the http section:
keepalive_timeout 90;
send_timeout 120;
reset_timedout_connection on;
fastcgi_keep_conn on;
fastcgi_read_timeout 90s;
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
Save the NGINX configuration file and ensure you restart the NGINX service:
sudo service nginx restart
When running large events, you may wish to improve performance by increasing the Scoreboard Refresh value in the admin configuration screen.
By default, the value of Scoreboard Refresh is 5 seconds. Increasing this value will delay the background (AJAX) refresh rate of clients to the number of seconds specified. The background refresh is utilized for actions such as countries being captured, and scores updating.
Performance tests have shown increasing this value improves scalability significantly. If you are experiencing performance issues, it is likely users will accept a longer background scoreboard refresh over delays, or even failures, while performing actions such as capturing levels and clicking countries.