Metrics pass over FTL

[alecthomas]

FTL needs to be instrumented sufficiently for production. This ticket tracks all the tasks, in the order we should do them in.

• otel: implement just otel #2020
• otel: chat with our friendly customer team to figure out what they expect to see/work in DataDog that isn't WAI #2022
• [Depends on 2020, 2021] otel: make sure everything existing actually WAI #2021
• [Depends on 2020] otel: add attributes to all calls: project_name and is_user_service #2023
• [Depends on 2020] otel: check attribute for all calls: trace_id #2024
• otel: plan out new signals and their attributes #2025
  • Once done, we'll need to create issues for all the signals enumerated for this issue. The new issues for implementing those signals should be added to this list.
• otel: actually instrument core signals #2194
• otel: add a dashboard with the standard connect metrics #2205
• otel: figure out how to get all metric descriptions propagated to datadog #2203
• otel: ensure traces include span parenting for publish->subscription calls #2258
• otel: code cleanup #2286
• otel: instrument observability (metrics + traces) on the database #2285
• otel: add new charts to DD dashboard(s) #2303
• [LOW PRI] otel: change the go-runtime SDK to use a constant string (or configurable?) #2026
• [LOW PRI] otel: support sending to two distinct destinations #2027

[bradleydwyer]

Consider how to test this locally without deployment / deployment of FTL.

[deniseli]

Initial breakdown of tasks that we'll sync with the team on:

1. Implement just otel - should run otel collector in a docker container to collect local logs/metrics/traces/spans and show the stream as they come in in the terminal tab. This way, we can run ftl dev, trigger the events we plan to log, and see them come through the terminal window in real(ish) time.
2. Chat with PFI about whether there's anything they expect to work that isn't. (relates to the next task)
3. Make sure everything existing actually WAI
4. Add these attributes to all signals:
   1. project_name: from ftl-project.toml
   2. is_user_service: We have the serviceName set here, which currently will be one of ftl-controller, ftl-runner, or <module name>. Technically, we could use ftl-controller || ftl-runner but it would be good to be a bit more explicit.
      1. Add bool flag isInternal to observability.Config so that Init’s callers can configure whether it’s user code or not.
   3. trace_id: do we know if the trace_id that otel provides is already instrumented correctly such that it can be used to trace verb-to-verb calls like the trace_id in our request headers, or better yet if they’re actually the same trace_id? If not, then we need to make the one in the request headers available.
      1. Q: should we persist all the header values to attrs? A: we should check if any is done by default. Also, check if there’s any sensitive data (e.g. auth tokens)
5. Plan out new signals + their attributes (incl. pubsub as described in the ticket desc) (reminder: queue depth)
6. Change the go-runtime SDK to use a constant string (or configurable?). It's opt-in, so not a huge deal.

[deniseli]

Just finished updating the ticket description. Everyone, feel free to assign tasks to yourself. I'll take the first one. Please do not take the two marked low priority until the others are closed (and maybe not even then!)

The original PubSub notes in the ticket description are now in #2025, along with @bradleydwyer 's reminders (feel free to add more in that ticket ;) )