Understanding socket connection issues

[cschneid]

What

We'd like to track core agent startup and socket connection issues so we can identify issues and fix them proactively.

Why

The socket connection might fail for multiple reasons, which might include

• Unable to create the core_agent_dir
• Unable to download (e.g outbound firewall prevents it)
• Permission denied to an already existing core_agent_socket path
• noexec is set on the volume where core_agent_dir resides
• Random things like core_agent_dir resides on a bind mount inside a docker container on an OSX host and creating a socket results in a name too long error. Can't run swarm tests in Docker for Mac moby/moby#23545 (comment)

No matter the issue, the end result is that the agent will attempt to connect to the socket, fail, and ultimately stop trying after a few attempts.

How

1. When the core agent fails to download, or socket fails to connect
2. Report the following data in a POST to https://checkin.scoutapp.com/apps/diagnostics.scout
3. With URL parameters: ?key=#{org_key}&name=#{app_name}
4. With Headers:

Agent-Hostname: #{socket.gethostbyname()}
Agent-Version: #{agent version}
Content-Type: application/octet-stream

5. With body data:

{
  version: 1,
  type: socket,
  language: php,
  agent_version: ${agent_version},
  environment: #{app_environment},
  node: #{hostname},
  agent_config: #{json object/hash of key/value of the agent's config options},
  error: {json object or string with anything you can provide w/ more details}
}

To prevent flooding ourselves with reports on every request, send this once per hour at most, and this should only run when a cache is available.

See-Also - Python Agent's matching ticket: scoutapp/scout_apm_python#554