[Meta] Improve APM agent instrumentation

[joshdover]

This is an issue to track possible improvements we could make to the APM agent instrumentation by adding custom transactions & spans.

Second iteration (7.11)

Backend

• Number of concurrent users or connections
• Alerting/Task Manager
  • some existing instrumentation in TM already
  • how this is affecting other operations in the server

Frontend

• Frontend route changes at the application level (Seems related)
  • Possibly use ApmRoute component, but has some caveats
• Session length & SPA memory (leaks)

Metadata

• Kibana CPU & memory size
• ES Cluster stats / size
• OS
• Enabled plugins
• HTTPS / TLS termination

Future

Backend

• SavedObjectClient
• @kbn/optimizer (dev only)
• HTTP authorization hooks

These are easy to benchmark in dev

• HTTP request validation
  • how much time do we spend on validation
• RequestHandlerContext providers
  • how much time do we spend creating the various clients, etc.

Frontend

• Chrome rendering performance
• Global search
• Visualization rendering
• SearchService

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[graphaelli]

In the frontend service, I was expecting the transaction name to be /apm/services (or /app/apm/services) , not /services

More generally, some alignment on transaction names is in order:

/app/apm/*?{query}
/app/apm/services
/services
/services/:serviceName/transactions
/services/:serviceName/transactions/view
GET /api/apm/settings/custom_links
GET /api/apm/traces/71099b2bf2ad4875aee562fcd14237f2
POST /../api/console/proxy
POST /api/ui_metric/report
POST /internal/licensing/feature_usage/register

[TinaHeiligers]

@joshdover I'd love to connect and chat about how we can use APM for Kibana Telemetry since our efforts seem to be somewhat aligned.

[TinaHeiligers]

@joshdover are we still going ahead with this issue during the 7.12 cycle? It's somewhat related to #78792, where we're working with the Node APM agent team.

[joshdover]

I'm going to push this one out. I'll pick up the remaining work in #78792 on tuning the agent to minimize overhead. Until we have that ironed out, I don't think it makes sense to spend time adding any additional instrumentation.

[dgieselaar]

One other thing that might be useful is tagging transactions with a plugin label, e.g. plugin: apm or plugin: ml, so teams can filter on that in the UI.

[mshustov]

APM team added API that allows instrumenting a span with a custom context. #90403
It can be leveraged to add domain-specific context for low-level API. For example, we can use it to create a span UiSettings.read instead of showing an HTTP request to ES endpoint /.kibana/_doc. Which makes the APM results much easier to understand for people without knowledge of the low-level Core API.

[dgieselaar]

@restrry the utility functions adds a span as a parent of the spans created in its context. It does not currently replace an automatically generated span. That would require some work in the Node.js agent I think, something that would allow us to intercept spans and rename them. I do think something like that would be better in the long run. @elastic/apm-agent-node-js

[mshustov]

SPA memory (leaks)

see newly added
https://github.com/WICG/performance-measure-memory
https://www.chromestatus.com/feature/5685965186138112

[lizozom]

@mshustov fyi we're already reporting the js heap size\limit\used to fullstory.

If we want to do some quick analytics there, it's available on page load events only since 7.16.

[lizozom]

A lot of these things are tracked on #78869

For example #129050 and #114843

Closing this issue in favor of the maintained one.