feat: Add metrics, gcp logging to tokenserver scripts

[jrconlin]

Description

This adds GCP logging JSON output formatting for logs as well as DataDog style metric reporting.

New Environs:

• METRIC_HOST - Collector Host address
• METRIC_PORT - Collector Host port
• METRIC_PATH - Socket path to collector (if available)

This also adds the --human arg if you want to use the older, human readable logs instead.

Issue(s)

Closes SYNC-4259

[pjenvey]

Why's datadog used (we switched off it a while ago)?

[jrconlin]

I thought that the metric collector was datadog compliant at least?
I used that library because I figured it was standard enough.

[pjenvey]

I'll note google provides a google-cloud-logging package. For some reason instead of a Formatter it provides a couple different handlers such as StructuredLogHandler for outputting to stdout. I gave it a quick try and it doesn't even include the timestamp from the log record (which is optional anyway, the log capturer probably just fills one in for the time it received it in that case). Just as an FYI: I was going to suggest switching to it but it's a bit odd? I'm fine with your version (though I suppose some of the extra metadata it adds could be handy later)

{"message": "Adding node foo to service sync-1.5","severity": "INFO", "logging.googleapis.com/labels": {"python_logger": "tokenserver.scripts.add_node"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 23, "file": "/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/add_node.py", "function": "add_node"}, "httpRequest": {} }

{"message": "Error while adding node\nTraceback (most recent call last):\n File \"/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/add_node.py\", line 25, in add_node\n database = Database()\n File \"/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/database.py\", line 260, in __init__\n engine = create_engine(os.environ['SYNC_TOKENSERVER__DATABASE_URL'])\n File \"/usr/lib/python3.10/os.py\", line 680, in __getitem__\n raise KeyError(key) from None\nKeyError: 'SYNC_TOKENSERVER__DATABASE_URL'","severity": "ERROR", "logging.googleapis.com/labels": {"python_logger": "tokenserver.scripts.add_node"}, "logging.googleapis.com/trace": "", "logging.googleapis.com/spanId": "", "logging.googleapis.com/trace_sampled": false, "logging.googleapis.com/sourceLocation": {"line": 28, "file": "/home/pjenvey/src/moz/syncstorage-rs/tools/tokenserver/add_node.py", "function": "add_node"}, "httpRequest": {} }

[jrconlin]

Yeah, I figure we can subclass or monkey patch that one at some point.

[pjenvey]

Not used?

Suggested change

	class Retry(requests.HTTPError):
	def is_retryable(self):
	self.response.status_code in [502, 503, 504]

[jrconlin]

Yeah, thanks. Forgot to pull that.