Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Expose log count metric from Runner #684

Merged
merged 2 commits into from
Apr 22, 2024
Merged

feat: Expose log count metric from Runner #684

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a6d9051
feat: Count logs by level
morgsmccauley Apr 22, 2024
3d394da
feat: Aggregate metrics from main thread
morgsmccauley Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion runner/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion runner/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@
"redis": "^4.6.7",
"verror": "^1.10.1",
"vm2": "^3.9.19",
"winston": "^3.13.0"
"winston": "^3.13.0",
"winston-transport": "^4.7.0"
}
}
12 changes: 12 additions & 0 deletions runner/src/logger.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,26 @@
import winston from 'winston';
import { LoggingWinston } from '@google-cloud/logging-winston';
import Transport from 'winston-transport';

import { METRICS } from './metrics';

const { format, transports } = winston;

class LogCounter extends Transport {
log (info: { level: string }, callback: () => void): void {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this going to count only .log calls? Or is this applied to all log calls (.warn, .error, etc.)? Another thing, it would be nice if we could similarly differentiate between error logs too for metrics. For example, it would be nice to distinguish between a single "Encountering error processing stream" error log and many "Error writing..." error calls. The former indicating the worker shut down whereas the latter could be any of these three: Hasura is down, our system queries are malformed, or an indexer is trying to run some faulty query.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could definitely add that, but it may be expensive given all the different log messages. My motive behind this was to be able to quickly see if we are erroring out from Grafana, alert from it, and then go in to GCP to see the actual issue/log messages.

METRICS.LOGS_COUNT.labels({ level: info.level }).inc();

callback();
}
}

const logger = winston.createLogger({
level: 'info',
format: format.combine(
format.timestamp(),
format.errors({ stack: true }),
),
transports: [new LogCounter()],
});

if (process.env.GCP_LOGGING_ENABLED) {
Expand Down
13 changes: 11 additions & 2 deletions runner/src/metrics.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import express from 'express';
import { Gauge, Histogram, Counter, AggregatorRegistry } from 'prom-client';
import { Gauge, Histogram, Counter, AggregatorRegistry, register } from 'prom-client';

import logger from './logger';

Expand Down Expand Up @@ -56,6 +56,12 @@ const EXECUTION_DURATION = new Histogram({
labelNames: ['indexer'],
});

const LOGS_COUNT = new Counter({
name: 'queryapi_runner_logs_count',
help: 'Number of messages logged',
labelNames: ['level'],
});

export const METRICS = {
HEAP_TOTAL_ALLOCATION,
HEAP_USED,
Expand All @@ -66,6 +72,7 @@ export const METRICS = {
UNPROCESSED_STREAM_MESSAGES,
LAST_PROCESSED_BLOCK_HEIGHT,
EXECUTION_DURATION,
LOGS_COUNT
};

const aggregatorRegistry = new AggregatorRegistry();
Expand All @@ -87,7 +94,9 @@ export const startServer = async (): Promise<void> => {
app.get('/metrics', async (_req, res) => {
res.set('Content-Type', aggregatorRegistry.contentType);

const metrics = await AggregatorRegistry.aggregate(Array.from(workerMetrics.values())).metrics();
const mainThreadMetrics = await register.getMetricsAsJSON();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What extra information does this provide? It seems we weren't getting this before.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I outlined this in the description - mainly just the log counts :)

const metrics = await AggregatorRegistry.aggregate([...Array.from(workerMetrics.values()), mainThreadMetrics]).metrics();

res.send(metrics);
});

Expand Down
Loading