Setting a span's parent after creation (or, tracing aggregation operations)

[tomasgareau]

I'm looking for a little insight into how I could trace the following scenario -- please let me know if this kind of question is better asked elsewhere (e.g., gitter).

1. Some event occurs
2. N sources will report on that event occurring
3. The N reports are aggregated into one payload
4. Some processing is performed on this payload

The trace diagram I'm looking to build would be something like this:

I'm unclear on how to correctly set up spans for the aggregation portion... Aggregation works roughly like this:

1. A report arrives for some event as a binary payload
2. The payload is deserialized
3. The aggregator now waits X seconds for other reports to arrive for the same event
4. After X seconds, the received reports are aggregated, serialized, and sent down the pipe to the next service

async def aggregate(qin: asyncio.Queue[bytes], aggregator: Aggregator, qout: asyncio.Queue[bytes]):
    while True:
        payload = await qin.get()
        report = deserialize(payload)

        # schedule the handling of this report so we don't block this coroutine: the aggregator will fire
        # after X seconds with whatever reports have arrived, we just need to keep shoving reports
        # into the aggregator
        asyncio.create_task(aggregator.handle_report(report), qout)

The tricky bit is that I won't know what event a report corresponds to until I deserialize the message. What I'd need is some way to set the parent after deserialization. I'd imagine that might look something like:

async def aggregate(qin: asyncio.Queue[bytes], aggregator: Aggregator, qout: asyncio.Queue[bytes]):
    while True:
        payload = await qin.get()
        
        # start_unbound_span is an imaginary function that creates a span to which we can attach a
        # parent later
        with tracer.start_unbound_span("deserialize") as deserialize_span:
            report = deserialize(payload)
        
        # some custom function where we create a root-level `event` span -- this will be propagated to
        # e.g., the `process` service
        event_span = get_or_create_event_span_for(report.event_id)
        
        # some custom function where we keep a list of active aggregation spans for explicit propagation
        # between asyncio tasks -- we'll end this span after aggregation
        aggregate_span = get_or_create_aggregate_span_for(report.event_id, parent=event_span)
        deserialize_span.set_parent(aggregate_span)
        
        # etc...

but AFAIK there's no way to set a parent span after the fact.

I'd love to hear if:
a) it would be possible to get a trace diagram like the one above, or, if that's not possible then:
b) what would be the OpenTelemetry "best practices"/typical approach for a scenario like this

[ocelotl]

Hello, sorry for the late reply

If I am understanding your issue correctly, then this may be a solution:

You can create a span, then pass it as an argument to this function, the returned context object can be used then as an argument here.

[tomasgareau]

Hi -- no problem! I understand you are all busy and appreciate you making the time for these Q&A.

Manually setting a trace's parent context isn't quite what I'm looking to do -- I was trying to manually set the span's parent after creating & using the span.

binary_payload = recv()

with tracer.start_as_current_span("deserialize") as dspan:
    msg = deserialize(binary_payload)

# now that we have deserialized the message, we can see if it corresponds
# to any currently running traces!
parent_ctx = get_context_for_event_id(msg.event_id)

# somehow associate dspan to this parent ctx??
dspan.set_parent_context(parent_ctx)  # not a real function

---

However, from what I can tell, this is not something that would work. I've started using Event rather than a Span for the deserialization operation: this won't capture the time spent deserializing but the main thing I want to know is how far apart messages for a given event are arriving so Events work nicely for my use-case.

This would look something like:

binary_payload = recv()

msg = deserialize(binary_payload)
parent_span = get_span_for_event_id(msg.event_id)
parent_span.add_event("received raw input", {"src": msg.src, "event_id": msg.id})

[Oberon00]

I was trying to manually set the span's parent after creating & using the span.

This is not allowed by the OpenTelemetry specification. If you had that ability, we could no longer guarantee that a trace is a tree. You could make a span a parent of itself, etc.
There is a spec issue open (untouched since a long time) to allow adding links to other spans after creation: open-telemetry/opentelemetry-specification#454

Currently, to model that scenario, you could create a new child span with a link to the additional parent instead of modifying the parent.

[tomasgareau]

Ahhh, that's what I was starting to suspect. That spec issue is along the lines of what I was trying to do.

That said, I think that in my particular case the Event is a better fit for timestamping the incoming messages.

Thanks for clarifying!