Initial multi channel trace file format specification

doc/architecture/trace_file_formats.adoc

@@ -5,20 +5,200 @@ endif::[]
 [#top-osi_trace_file_formats]
 = OSI trace file formats
 
-There are two formats for storing multiple serialized OSI messages in one trace file.
+== Single channel trace file formats
+There are two formats for storing and reading multiple serialized OSI messages of the same type in one trace file.
+These formats are very bare bones and do not contain any additional information, like meta-data, schemas, nor do they allow for random access.
+For more advanced use-cases, consider using the multi channel trace file format.
 
 *.osi::
-Binary trace file.
+Single channel binary trace file.
 Messages are separated by a length specification before each message.
 The length is represented by a four-byte, little-endian, unsigned integer.
 The length does not include the integer itself.
 
 *.txth::
-Human-readable plain-text trace file.
+Single channel human-readable plain-text trace file.
 Messages are separated by newlines.
+Each message is a serialized OSI message in protocol buffer text format.
+
+NOTE: The `.txth` format is intended for human consumption (e.g. for debugging and manual checks).
+It is currently not supported for reading by the OSI API, as it is not unambiguously deserializable.
 
 NOTE: Previous releases of OSI also supported a so-called plain-text trace file format, with file extension `.txt`.
 This legacy format did not contain plain-text, but rather binary protobuf messages separated by a special separator.
 For obvious reasons the format was deprecated and fully replaced with the `.osi` binary file format.
 This release no longer contains any support for the legacy `.txt` file format.
-These files may be used for manual checks.
+
+== Multi channel trace file format
+
+=== Overview
+
+The OSI multi channel trace file format is a binary file format that allows for storing multiple serialized OSI message streams of the same or different types in one trace file, along with additional meta-data, and other related data streams.
+Due to the nature of the format, it allows for random access to the data streams, and is suitable for more advanced use-cases.
+
+The OSI multi channel trace file format is based on the MCAP file format, which is a generic multi channel trace file format.
+The OSI multi channel trace file format is a specialization of the MCAP file format, with additional constraints and requirements specific to OSI.
+Hence, any valid OSI multi channel trace file is also a valid MCAP file, but not the other way around.
+
+The following rules apply to OSI multi channel trace files:
+
+- The file extension to be used is `.mcap`.
+- The file must be a valid MCAP file according to the https://mcap.dev/spec[MCAP format specification] version `0x30`.
+- The file must be an indexed MCAP file, including chunk index records in the summary section, with all message records written into chunk records.
+- Chunk records must either be uncompressed or compressed using either the `zstd` or `lz4` compression algorithms.
+- The file must contain a meta-data record with the OSI trace file meta-data defined in section <<sec-osi-trace-file-multi-global-meta-data>>.
+  This meta-data record identifies the file as an OSI multi channel trace file.
+- The file must contain a schema record in the summary section for each top-level message type that is used in one or more OSI channels with the OSI message schema as defined in section <<sec-osi-trace-file-multi-schema-record>>.
+- The file must contain at least one OSI message stream in a channel as defined in section <<sec-osi-trace-file-multi-channel>>.
+- The file may contain additional non-OSI message streams in other channels.
+
+NOTE: All OSI and non-OSI message streams stored in the same file share a common time base.
+Storing of unrelated message streams in one trace file is therefore not generally useful and should be avoided.
+
+[#sec-osi-trace-file-multi-global-meta-data]
+=== Multi channel trace file global meta-data
+
+The file must contain a meta-data record with the `name` field being `net.asam.osi.trace` and the following mandatory `metadata` entries:
+
+`version`::
+The semantic version number of the OSI release that this OSI trace file conforms to.
+This is a string in the format `major.minor.patch`, for example `3.0.0`.
+Note that this version number is not necessarily the same as the OSI schema version(s) used in the trace file.
+Rather, it indicates the version of the OSI trace file format itself.
+
+`min_osi_version`::
+The minimum version of the OSI schema used in the trace file OSI channels.
+This is a string in the format `major.minor.patch`, for example `3.0.0`.
+
+`max_osi_version`::
+The maximum version of the OSI schema used in the trace file OSI channels.
+This is a string in the format `major.minor.patch`, for example `3.0.0`.
+
+`min_protobuf_version`::
+The version of the protobuf implementation used in the trace file.
+This is a string in the format `major.minor.patch`, for example `3.17.3`.
+
+`max_protobuf_version`::
+The version of the protobuf implementation used in the trace file.
+This is a string in the format `major.minor.patch`, for example `3.17.3`.
+
+`zero_time`::
+The point in time corresponding to time 0 in all timestamps in the trace file.
+Must be provided as a full ISO8601 formatted date time string, including timezone data, conforming to the https://www.w3.org/TR/xmlschema11-2/#dateTimeStamp[XML Schema dateTimeStamp] lexical space.
+Values must match the following regular expression:
+`-?([1-9][0-9]{3,}|0[0-9]{3})-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T(([01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9](\.[0-9]+)?|(24:00:00(\.0+)?))(Z|(\+|-)((0[0-9]|1[0-3]):[0-5][0-9]|14:00))`
++
+NOTE: that even in pure simulation use cases there is usually a relationship to real time, as on-board components and environment simulation have necessary relationships to real time (for example, the embedded `HostVehicleData` will carry relevant real time information, for localization and other purposes).
+
+`creation_time`::
+The point in time when the trace file was created.
+Must be provided as a full ISO8601 formatted date time string, including timezone data, conforming to the https://www.w3.org/TR/xmlschema11-2/#dateTimeStamp[XML Schema dateTimeStamp] lexical space.
+
+The `net.asam.osi.trace` meta-data record may also contain the following recommended `metadata` entries:
+
+`description`::
+A human-readable description of the data contained in the multi channel trace file.
+
+`creator`::
+A list separated by commas of entities (not tools) involved in the creation of the data contained in the file.
+
+`license`::
+If the contents of the file is licensed under any SPDX registered license, this entry should contain the SPDX identifier for the license.
+
+`data_sources`::
+A list separated by commas of data sources used in the creation of the data contained in the file.
+
+The file may contain arbitrary additional meta-data records, however meta-data records with names starting with `net.asam.osi` are reserved for future use by {THIS_STANDARD}.
+It is strongly recommended to follow reverse domain name notation for custom meta-data record names to avoid conflicts.
+
+[#sec-osi-trace-file-multi-schema-record]
+=== OSI message schema
+
+For each OSI top-level message type that is used in one or more OSI channels, the OSI multi channel trace file must contain a corresponding schema record in the summary section.
+Note that if multiple versions of the OSI schema are used in the same trace file, a schema record must be included for each version, with different schema IDs.
+
+The schema record must contain the following fields:
+
+`id`::
+A file-wide unique non-zero identifier for the schema record.
+
+`name`::
+The fully qualified name to the message within the OSI descriptor set. For example, for the `SensorView` message type, this would be `osi3.SensorView`.
+
+`encoding`::
+The value `protobuf`.
+
+`data`::
+A binary FileDescriptorSet as produced by `protoc --include_imports --descriptor_set_out`.
+
+The schema record must be stored in the summary section of the trace file, and must be referenced by the OSI channels that use the schema.
+
+[#sec-osi-trace-file-multi-channel]
+=== OSI channel
+
+An OSI channel is a data stream within the OSI multi channel trace file that contains serialized OSI top-level messages of the same type.
+Note that non-top-level messages must not be stored directly in OSI channels.
+
+Each OSI channel must be described by a channel record in the summary section of the trace file with the following fields:
+
+`id`::
+A file-wide unique identifier for the channel.
+
+`schema_id`::
+The ID of the schema record that describes the message type of the channel.
+
+`topic`::
+A unique name for the channel within the trace file.
+When recording OSI traces for a model packaged according to the OSI Sensor Model Packaging (OSMP) layer, using the naming conventions defined in the OSMP specification for variables as topics is recommended.
+For example, for a sensor model with two SensorView inputs and one SensorData output, the topic names would be `OSMPSensorViewIn[1]`, `OSMPSensorViewIn[2]`, and `OSMPSensorDataOut`, accordingly.
+In other cases, the topic name should be chosen to reflect the purpose of the channel, and should include some indication of the message type.
+
+`message_encoding`::
+The value `protobuf`.
+
+`metadata`::
+A map of additional meta-data for the channel.
+This map may contain arbitrary key-value pairs, however keys starting with `net.asam.osi` are reserved for use by {THIS_STANDARD}.
+It is strongly recommended to follow reverse domain name notation for custom meta-data keys to avoid conflicts.
+The following mandatory entries are defined by {THIS_STANDARD}:
+
+`net.asam.osi.trace.channel.osi_version`:::
+The version of the OSI schema used in creating the data of this OSI channel.
+This is a string in the format `major.minor.patch`, for example `3.0.0`.
+
+`net.asam.osi.trace.channel.protobuf_version`:::
+The version of the protobuf implementation used in creating the data of this OSI channel.
+This is a string in the format `major.minor.patch`, for example `3.17.3`.
+
+The following recommended entries are defined by {THIS_STANDARD}:
+
+`net.asam.osi.trace.channel.description`:::
+A human-readable description of the channel.
+
+The channel record must be stored in the summary section of the trace file, and must be referenced by the OSI message records that are part of the channel.
+
+All messages in an OSI channel must be stored in chunk records in the data section of the trace file.
+
+Each message record in a chunk record must contain the following fields:
+
+`channel_id`::
+The ID of the channel that the message belongs to.
+
+`sequence`::
+Optional message counter to detect message gaps.
+If the relevant packaging layer or other source of messages provides a sequence number this can be used.
+Otherwise this should be set to zero to indicate that no reliable sequence number is available.
+
+`log_time`::
+This field is in nanoseconds and uses the same epoch as the `publish_time` field.
+It is used to determine the order of messages in the trace file, and provides for time-based random access to the data streams.
+Unless there is a specific reason to set this field to a different value, it should be set to the same value as `publish_time`, as this reflects the time flow as reflected in the OSI message stream.
+Only if recreation of the message stream with actual message transmission times is required - for example for asynchronous packaging layers, this field should be set to the time when the message was enqueued for addition to the trace file.
+
+`publish_time`::
+The timestamp taken from the timestamp field of the stored OSI top-level message.
+The field is in nanoseconds, with the epoch being the epoch of the OSI Timestamp data type.
+If top-level messages that do not contain a timestamp field are stored in the trace file, the `publish_time` field must be set to the time when the message was enqueued for addition to the trace file.
+
+`data`::
+The serialized OSI message data.

doc/architecture/trace_file_naming.adoc

@@ -9,7 +9,7 @@ endif::[]
 The names of OSI trace files should have the following format:
 
 ----
-<timestamp>_<type>_<osi-version>_<protobuf-version>_<number-of-frames>_<custom-trace-name>.osi
+<timestamp>_<type>_<osi-version>_<protobuf-version>_<number-of-frames>_<custom-trace-name>.[osi|txth|mcap]
 ----
 
 **Types**
@@ -44,6 +44,12 @@ Trace file contains `MotionRequest` messages.
 `su`::
 Trace file contains `StreamingUpdate` messages.
 
+`multi`::
+Trace file contains multiple types of messages for use with multi channel trace file format.
+In this case the number-of-frames field should be the largest number of frames across all channels.
+The OSI version field should be based on the `version` field in the file meta-data of the multi-channel trace file.
+The protobuf version field should be based on the `min_protobuf_version` field in the file meta-data of the multi-channel trace file.
+
 **Example**
 
 Given an OSI trace file with the following information:
@@ -76,3 +82,9 @@ The recommended file name is:
 ----
 20210818T150542Z_sv_312_300_1523_highway.osi
 ----
+
+For a corresponding multi channel trace file containing `SensorView` and `GroundTruth` messages, the recommended file name is:
+
+----
+20210818T150542Z_multi_312_300_1523_highway.mcap
+----