Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: first metric (verb call counter) #2130

Merged
merged 7 commits into from
Jul 23, 2024
Merged

Conversation

deniseli
Copy link
Contributor

@deniseli deniseli commented Jul 22, 2024

Instrument our first metric. Next: check if this metric pipes through to DD correctly.

Locally, the output via just otel-stream looks like:

ScopeMetrics #1
ScopeMetrics SchemaURL: 
InstrumentationScope ftl.call 

Metric #0
Descriptor:
     -> Name: requests
     -> Description: Count of FTL verb calls via the controller
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative

NumberDataPoints #0
Data point attributes:
     -> ftl.module.name: Str(time)
     -> ftl.verb.name: Str(time)
StartTimestamp: 2024-07-23 00:25:50.782759 +0000 UTC
Timestamp: 2024-07-23 00:25:59.069408 +0000 UTC
Value: 1

NumberDataPoints #1
Data point attributes:
     -> ftl.module.name: Str(echo)
     -> ftl.verb.name: Str(echo)
StartTimestamp: 2024-07-23 00:25:50.782759 +0000 UTC
Timestamp: 2024-07-23 00:25:59.069408 +0000 UTC
Value: 1
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-07-23T00:26:00.745Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 10, "data points": 15}
2024-07-23
T00:26:00.745Z	info	ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.25.0
Resource attributes:
     -> service.name: Str(echo)
     -> service.version: Str(HEAD)
     -> telemetry.sdk.language: Str(go)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.27.0)

Interestingly, it looks like the metric logging races in a slightly inconsistent way. If you call a verb that calls another verb, you would expect just one verb to get bumped to 1 first, and then the other verb. Sometimes, it does behave that way. However, other times, the first time it appears in otel-stream will already be with both verbs. So essentially, there is no guarantee that transient values actually show up in the stream, but the ultimate value will be correct.

This does appear the report the correct count in even error cases. e.g. I tried calling echo before the deploy was done, so that the verb call would fail without calling out to time. It would correctly log a single call in that case. And if you made another successful call afterwards, the metric would correctly increment to include both successful calls.

@deniseli deniseli changed the title Dli/toy metric and log chore: first metric (verb call counter) Jul 22, 2024
@ftl-robot ftl-robot mentioned this pull request Jul 22, 2024
@deniseli deniseli force-pushed the dli/toy-metric-and-log branch from c896c64 to 2ecc680 Compare July 22, 2024 20:12
@deniseli deniseli marked this pull request as ready for review July 22, 2024 20:17
@deniseli deniseli requested a review from alecthomas as a code owner July 22, 2024 20:17
@deniseli deniseli requested review from a team and worstell and removed request for a team July 22, 2024 20:17
backend/controller/controller.go Outdated Show resolved Hide resolved
backend/controller/controller.go Outdated Show resolved Hide resolved
internal/observability/client.go Outdated Show resolved Hide resolved
Comment on lines +937 to +946
logger := log.FromContext(ctx)

requestCounter, err := otel.GetMeterProvider().Meter("ftl.call").Int64Counter(
"requests",
metric.WithDescription("Count of FTL verb calls via the controller"))
if err != nil {
logger.Errorf(err, "Failed to instrument otel metric `ftl.call.requests`")
} else {
requestCounter.Add(ctx, 1, metric.WithAttributes(attribute.String("ftl.module.name", req.Msg.Verb.Module), attribute.String("ftl.verb.name", req.Msg.Verb.Name)))
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! We can start with this for testing, and move into a dedicated file in the future to keep the controller light :)

@wesbillman wesbillman enabled auto-merge (squash) July 23, 2024 00:30
@wesbillman wesbillman merged commit c9b430c into main Jul 23, 2024
59 checks passed
@wesbillman wesbillman deleted the dli/toy-metric-and-log branch July 23, 2024 00:32
@worstell worstell added the approved Marks an already closed PR as approved label Jul 23, 2024
@deniseli
Copy link
Contributor Author

Linking issue: #2194

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Marks an already closed PR as approved
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants