docs/ec2-debug-and-manual-cleanup

content/docs/self-hosted-runners.md

@@ -391,6 +391,51 @@ provisioned through environment variables instead of files.
 </tab>
 </toggle>
 
+#### Cloud Compute Resource Manual Cleanup
+
+In very rare cases, you may need to cleanup CML cloud resources manually.
+An example of such a problem can be seen
+[when an EC2 instance ran out of storage space](https://github.com/iterative/cml/issues/1006).
+
+The following sections contain lists of all the resources you may need to
+manually cleanup in the case of a failure.
+
+<toggle>
+<tab title="AWS">
+
+- The running EC2 instance (named with pattern `cml-{random-id}`)
+- The volume attached to the running EC2 instance
+  (this should delete itself after terminating the EC2 instance)
+- The generated key-pair (named with pattern `cml-{random-id}`)
+
+If you keep encountering issues, it is appreciated to attempt pulling the logs
+from the running instance before terminating and opening a GitHub Issue.
+
+To do so add a startup command to the runner:
+
+> `--cloud-startup-script=$(echo 'echo "$(curl https://github.com/'"$GITHUB_ACTOR"'.keys)" >> /home/ubuntu/.ssh/authorized_keys' | base64 -w 0)`
+
+Once the instance fails you can attempt to connect to it and dump logs with:
+
+```bash
+ssh ubuntu@instance_public_ip
+sudo journalctl -n all -u cml.service --no-pager > cml.log
+sudo dmesg --ctime > system.log
+sudo dmesg --ctime --userspace > userspace.log
+```
+
+You can then copy those logs to your local machine with:
+
+```bash
+scp ubuntu@instance_public_ip:~/cml.log .
+scp ubuntu@instance_public_ip:~/system.log .
+scp ubuntu@instance_public_ip:~/userspace.log .
+```
+
+There is a chance that the instance could be severely broken if the SSH command
+hangs -- if that happens reboot it from the web console and try the commands
+again.
+
 #### On-premise (Local) Runners
 
 The `cml runner` command can also be used to manually set up a local machine,