Where and how should segmentation work

[slifty]

Discussion

What do you want to talk about?

Let's talk about segmentation.

A video source might have distinct content segments that should be treated as contextually coherent "sets" of data, potentially with its own metadata that applies to all payloads within it.

For example: TV stations have programs that run for a specific duration of time and have explicit metadata associated (e.g. program name, hosts, etc.).

what is a "segment" at a technical level? How does a "segment" manifest in a given payload (if it does at all)?

Once answer these things, we need to decide where in our architecture segmentation should actually occur. What can trigger the creation of a new segment? What should happen when a new segment is created?

Other Thoughts

To complicate things, within a segment there may be sub-segments -- for instance, individual commercials could be segments. Would the ideal segmentation architecture allow for a TVK implementation to dynamically initiate new segments?

Relevant Resources / Research

We started to explore some of these things over in the appliances repository -- inspired by the SRT generation appliance.

tvkitchen/appliances#70

[slifty]

What is a segment?

A segment is a time-constrained subsection of a source. For instance, a source might have 10 hours of content and might be broken into twenty separate 30 minute segments.

How does a segment manifest in a given payload (if it does at all)?

Right now the position attribute of a payloads is based on how many milliseconds into the video source a payload is.

With segments, we might have position instead represent how many milliseconds into the segment a payload is. This means a given source could produce multiple STREAM.CONTAINER payloads with position 0 (one per segment). It essentially breaks a single source into multiple "videos."

Alternatively, we might create a new type of SEGMENT payload (which, as a payload, would have a position as well). This would allow a given appliance / application that cares about segments to calculate "segment position" -- keeping track of new segment events and doing simple math (e.g. position - segment start position = segment position).

A segment might also have metadata associated with it (e.g. program name, host name, program description, etc). We have a vague concept of "metadata" associated with payloads, but I am worried that adding these to the payloads themselves would quickly make things very heavy (e.g. if the program name is included in every single payload that would dramatically increase the size of a payload).

As an aside: the timestamp of a given payload would still exist outside of the context of the segment -- timestamp is defined as being absolute time, so it has nothing to do with segments.

I'm getting convinced that segmentation (A) should be explicit and (B) should NOT modify the positions of payloads directly (or any aspect of a payload). Rather, a segment should be conveyed via its own payload.

This would be more robust and also potentially allows multiple types of segmentation to exist in parallel on a single stream. The tradeoff is that segment context must be "remembered" by applications and appliances that care about segmentation.

What can trigger the creation of a new segment?

For my immediate use case, time is the driver of segments: e.g. I have a schedule of programs for a channel in a given day, and I know that there is a new program at the top of an hour / every 30 minutes / etc. (I may know the names of those programs as well)

For pre-recorded video, there may be a list of specific segment times (e.g. we know when each question was read in a debate and want to segment the debate according to those times).

Where should segmentation occur?

• Should we allow appliances to initiate new segments based on whatever feature it wants?
• Should we have a segmentor appliance that could initiate new segments based on stream-processing logic (e.g. in response to payload data in addition to payload timestamp).

It seems like the answer to all of these things is... yes!

Should we have a SEGMENT.START payload (should we have a SEGMENT.END payload) <-- It would be more apt to find a term that captures a generic delimiter (e.g. SEGMENT.MARK or SEGMENT.DELIMITER). TBD.

---

Conclusions

1. Let's have segmentation happen through a new payload type!

Appliances that care about segmentation can consume that payload type. Appliances that don't care about segmentation can just go about their business.

2. It seems like segmentation won't need countertop support after all.

We might eventually want the countertop to be able to dynamically turn appliances on and off -- but that's deeper and doesn't have to be coupled to segmentation directly (indeed, the logic there could be based on payload analysis, and if segments are payloads... ALL THE BETTER)

3. We may want to add some kind of common / universal segmentation support (e.g. keeping track of the current) to the appliance core; I'm going to reflect on that, but there is no need to start there and it will probably become clear how useful that is after improving the SRT generator appliance to handle segments.

Good talk everybody.