Rewrite span status section and standardize preamble in manual instrumentation

[cartermp]

fixes #3685

[yurishkuro]

even if an error may have actually occurred

I don't like this emphasis on overriding error, this is not the primary use case for OK status. The primary use case is the application owner with domain knowledge explicitly sets this status to OK. It does not mean that they are changing it from Error.

For example, an instrumentation library may decide to set a span status to Error

Personally, I don't feel like this answers the original question. To me the question is: is instrumentation ever allowed to set anything but Unset? Can they only set Error but not OK? The OTEP says they should not set OK. The reasoning doesn't make sense to me. I would argue that when a span is Ended, if its stuatus is still Unset it should be change to OK, otherwise we get the justifiable confusion from end users who cannot find any OK spans.

[cartermp]

So in general I agree that the spec made the wrong design here. And moreover, I think there might need to be better guidance for users of Observability backends around how you query this kind of data. In generally, "show me what wasn't an error" is what you want, but some UIs (like Tempo) let you filter specifically by a span status and that's where some confusion arises. If the spec detailed things in a more user-friendly way then this wouldn't be an issue, but alas I don't think it can be changed now.

The primary use case is the application owner with domain knowledge explicitly sets this status to OK

I'd agree, but I'd love to have a more specific example though. When there's no real difference between an actually-complete-and-no-error span set to OK and left Unset, then I'm not sure what the benefit to explicitly setting it to OK is.

[yurishkuro]

There was an odd critique in the OTEP about the approach that OpenTracing had, where we had error=true attribute. I actually think it was pretty good, much better than this confusion between Unset and OK. Since that status was elevated to the data model (instead of semconv in OT), I would prefer the status to be OK or Error and not have Unset as an option. Unset is really an intermediate state that could be managed in the SDK until the span is finished. Once it's finished, if the error status was not set then the span is automatically OK. Since OTEL already has Unset in a stable spec, I think the path forward is to tweak the spec to say that Tracer should change Unset->OK when the span is finished. The other train of thought in the OTEP that some backend components might change Unset to OK is completely alien to me - why would a backend have any better knowledge about what happened in the application than the application & instrumentation that produced the data?

[svrnm]

I think this is a valuable discussion, can we turn this into a spec issue? For the docs I think we need to document the as is state and then later update it accordingly.

[cartermp]

I think in the interim we should tweak the example(s) in prose a bit, more to this point:

I don't like this emphasis or overriding error, this is not the primary use case for OK status.

[cartermp]

@yurishkuro I adjusted the wording here and in the preamble. PTAL?

[svrnm]

{{% docs/instrumentation/span-status-preamble %}}

I love that 😍

[yurishkuro]

+1

[cartermp]

@kaylareopelle I applied suggestions -- any further feedback?

Feedback addressed

[cartermp]

Feedback all addressed and multiple approvals, so I'll merge