feat: Allow starting a span on a transaction that has already ended

[trentm]

This was a restriction that dated back to the v1 APM Server intake
API that required all a transaction's spans to be sent along with the
transaction.

The Java APM agent, FWIW, allows this.

This came up in discussion about some edge cases with the OTel Bridge. As we move closer to OTel semantics, the distinction between transactions and spans gets to be less. Currently can start a child span of a span that has already ended, so it is nice to be able to do the same for a transaction.

Checklist

• Implement code
• Add tests
• Add CHANGELOG.asciidoc entry

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-04-21T18:47:10.399+0000

• Duration: 21 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	247149
Skipped	0
Total	247149

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run module tests for <modules> : Run TAV tests for one or more modules, where <modules> can be either a comma separated list of modules (e.g. memcached,redis) or the string literal ALL to test all modules

• run benchmark tests : Run the benchmark test only.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[trentm]

Here is an example of a trace (in APM UI v7.15.0) that shows a span ("s6") that started after its transaction ended:

This is from running script https://github.com/elastic/apm-agent-nodejs/blob/trentm/otel-bridge/test/opentelemetry-sdk/fixtures/start-span-with-context.js from the ongoing OTel Bridge work.

[astorm]

Seems reasonable -- especially if java does it. Also, the implementation is reasonable.

Just to say it out loud (not a blocker) I am slightly curious what, if any, effect this will have on features that rely on the transaction's span_count -- it seems like it's possible for these late spans to not be counted (depending on whether an ended transaction's been reported).

A question for future us, I suppose :). Approving, presuming the current CI failures as a :jenkins: hiccup.

[trentm]

it seems like it's possible for these late spans to not be counted

Thanks for mentioning it. Yes. We'd discussed this in chat briefly. Felix's opinion was:

[Trent]

Do you think that transaction.span_count.* values being possibly less than the number of span documents will surprise/break any APM UI code?

[Felix]

The Java agent already supports spans that end after their transaction. I don’t think it will break the UI. It shows a warning if it can find fewer spans than expected but not if it finds more than expected.

[trentm]

/test