feat: Expose log count metric from Runner #684
Conversation
morgsmccauley
changed the title
| @@ -87,7 +94,9 @@ export const startServer = async (): Promise<void> => { | |||
| app.get('/metrics', async (_req, res) => { | |||
| res.set('Content-Type', aggregatorRegistry.contentType); | |||
|
|
|||
| const metrics = await AggregatorRegistry.aggregate(Array.from(workerMetrics.values())).metrics(); | |||
| const mainThreadMetrics = await register.getMetricsAsJSON(); | |||
There was a problem hiding this comment.
What extra information does this provide? It seems we weren't getting this before.
There was a problem hiding this comment.
I outlined this in the description - mainly just the log counts :)
|
|
||
| const { format, transports } = winston; | ||
|
|
||
| class LogCounter extends Transport { | ||
| log (info: { level: string }, callback: () => void): void { |
There was a problem hiding this comment.
Is this going to count only .log calls? Or is this applied to all log calls (.warn, .error, etc.)? Another thing, it would be nice if we could similarly differentiate between error logs too for metrics. For example, it would be nice to distinguish between a single "Encountering error processing stream" error log and many "Error writing..." error calls. The former indicating the worker shut down whereas the latter could be any of these three: Hasura is down, our system queries are malformed, or an indexer is trying to run some faulty query.
There was a problem hiding this comment.
We could definitely add that, but it may be expensive given all the different log messages. My motive behind this was to be able to quickly see if we are erroring out from Grafana, alert from it, and then go in to GCP to see the actual issue/log messages.
Creates a new
winstontransport method to count logs by level, and exposes a new prometheus metric to record this value.Additionally, metrics recorded on the main thread were not captured. This was not an issue as the majority of metrics were done within the worker. But since we also log on main thread, this PR updates metrics aggregation to expose main thread metrics.