roachtest: set up machines and env to collect core dumps

[tbg]

Relevant for random crashes as well as #34241. This rando script might be helpful. I remember spending way too much time to actually get this to work.
Additionally we want GOTRACEBACK=crash for all roachprod run/start invocations.

#!/usr/bin/env bash
CMD_USAGE=""
source "$(dirname "$0")/config.sh"

CORE_PATTERN="/tmp/core.%e.%p.%h.%t"

ssh_task "
  set -euo pipefail
  echo -e '
  * soft core unlimited
  * hard core unlimited
  root soft core unlimited
  root hard core unlimited
  ' | sudo tee /etc/security/limits.d/core_unlimited.conf > /dev/null
  echo '$CORE_PATTERN' | sudo tee /proc/sys/kernel/core_pattern > /dev/null
  sudo sed -i'~' 's/enabled=1/enabled=0/' /etc/default/apport
  sudo sed -i'~' '/.*kernel\\.core_pattern.*/c\\' /etc/sysctl.conf
  echo 'kernel.core_pattern=$CORE_PATTERN' | sudo tee -a /etc/sysctl.conf > /dev/null
  echo 'Done. Make sure cockroach is started through ulimit -c unlimited, or it will run with a soft limit of zero and still not create core dumps.'
"

[tbg]

I just checked whether this maybe "just works" and it doesn't. By default, core dumps are handled by apport which does the following with it:

ERROR: apport (pid 3149) Tue Feb 12 11:32:17 2019: called for pid 3073, signal 6, core limit 0, dump mode 1
ERROR: apport (pid 3149) Tue Feb 12 11:32:17 2019: executable: /home/tschottdorf/cockroach-v2.1.4.linux-amd64/cockroach (command line "./cockroach start --insecure")
ERROR: apport (pid 3149) Tue Feb 12 11:32:17 2019: executable does not belong to a package, ignoring

So I assume that we'll end up reusing most of the above script (which came into existence precisely because of the above)