Initial proposal

index.html

@@ -16,16 +16,10 @@
           w3cid: "103966"
         },
         {
-          name: "Morgan McLean",
-          company: "Google",
-          companyURL: "https://google.com",
-          w3cid: "104128"
-        },
-        {
-          name: "Alois Reitbauer",
-          company: "Dynatrace",
-          companyURL: "https://dynatrace.com",
-          w3cid: "48276"
+          name: "Clemens Vasters",
+          company: "Microsoft",
+          companyURL: "https://microsoft.com",
+          w3cid: "103966"
         }],
         github: {
           repoURL: "https://github.com/w3c/trace-context-amqp/",

spec/20-AMQP_FORMAT.md

@@ -1,3 +1,132 @@
 # AMQP format
 
-TBD
+The [Advanced Message Queuing Protocol (AMQP)](https://www.amqp.org/) is an open internet protocol for
+business messaging. It defines a binary wire-level protocol that allows for the
+reliable exchange of business messages between two parties. AMQP can be used as
+a protocol for asynchronous communication between components of an application.
+From the distributed tracing and telemetry correlation perspective it is a
+known problem to be able to correlate a component that placed message and
+component that processed it later. This specification describes how trace
+context MUST be encoded into AMQP messages.
+
+## Trace context fields placement in a message
+
+AMQP defines message as a payload with the additional annotations sections.
+There are two annotation sections this specification refers -
+"application-properties" and "message-annotations". See
+[3.2](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-messaging-v1.0-os.html#section-message-format)
+of *OASIS Advanced Message Queuing Protocol (AMQP) Version 1.0, Part 3:
+Messaging* (AMQP specification).
+
+AMQP message section "application-properties" is immutable collection of
+properties that is defined by message publisher and can be read by the message
+consumer. Message brokers cannot mutate those properties. Section
+"message-annotations" is designed for message brokers to use and can be mutated
+during the message processing.
+
+Fields `traceparent` and `tracestate` SHOULD be added to the message in the
+application-properties section by message publisher. Once the message has been
+created, it’s no longer permissible to edit the bare message. So if it were
+necessary to annotate the message inside the middleware as it transits, that
+MUST happen in the “message-annotations” section, using the
+“application-properties” as a base. Message reader SHOULD construct the full
+trace context by reading `traceparent` and `tracestate` fields from the
+“message-annotations” first and if not exist - from “application-properties”.
+
+## Trace context and failed read/write operations
+
+This specification defines how to propagate context from publisher thru broker
+to ultimate reader. It does not define any additional transport level correlation
+constructs that can be used to investigate failed publish or read operations.
+
+It is recommended, however, for AMQP implementations to make the best effort
+attempt to read the trace context from the message and use it while reporting
+such problems.
+
+TODO: do we need to define a way to specify trace context of read operation? Is
+there anything in AMQP protocol that can be used to carry this context?
+
+## `traceparent` AMQP format
+
+The field `traceparent` MUST be encoded and decoded using [binary
+protocol](..\extension-binary.html) and stored as a binary type defined in
+section
+[1.6.19](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#type-binary)
+of AMQP specification.
+
+Property name MUST be `traceparent` - all lowercase without delimiters.
+
+## `tracestate` AMQP format
+
+The field `tracestate` MUST be encoded and decoded as a string to string map.
+See definition of type map in section
+[1.6.23](http://docs.oasis-open.org/amqp/core/v1.0/os/amqp-core-types-v1.0-os.html#type-map)
+of AMQP specification.
+
+---
+
+*NOTE about the strings encoding*
+
+Please note that strings are defined as UTF-8 in AMQP. Should anything be
+different from the HTTP encoded opaque strings? Should we allow to benefit from
+wider character set for better encoding of opaque values? Or this will be too
+error prone?
+
+---
+
+Property name MUST be `tracestate` - all lowercase without delimiters.
+
+## Relations with message id field
+
+The field `message-id` is defined in section
+[3.2.4](http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-messaging-v1.0.html#type-properties)
+of AMQP specification.
+
+Message-id, if set, uniquely identifies a message within the message system. The
+message producer is usually responsible for setting the message-id in such a way
+that it is assured to be globally unique. A broker MAY discard a message as a
+duplicate if the value of the message-id matches that of a previously received
+message sent to the same node.
+
+Trace context identifies the context of a worker who either publish or read the
+message while message-id identifies the content of the message. So the message
+with the same message-id can be send twice with the different values of trace
+context fields in case of retries. Also trace context can be changed from broker
+to broker to identify the broker while message-id will not be changed.
+
+It is recommended to use a unique `parent-id` for every publish of a message. So
+it is not possible that `traceparent` will be identical for the messages with
+the different `message-id`.
+
+## AMQP specific security and privacy considerations
+
+AMQP defines a protocol for a potentially long living messages. Long-term
+storing of `traceparent` and `tracestate` fields may require additional handling
+of security and privacy as that may not be covered by "in-flight" data
+exemptions.
+
+So all the same privacy and security techniques should be applied with the
+potentially more strict requirements.
+
+## Size sensitive environments
+
+TODO: There are many good reason to keep the format and sizes of fields
+unchanged. For the size-sensitive environments, if you ONLY INITIATE the message
+outside of existing context – there may be a possibility to save on size by
+truncating or reusing some fields.
+
+## Batching
+
+The specification defines a binding to the AMQP "message format 0", i.e. the
+layout that is defined in the Messaging section of the the AMQP spec, but does
+not provide any further detail on HOW those messages are being transferred.
+
+AMQP is enormously efficient when filling link credit with sending messages in
+sequence without any special overlaid batching. And is often used without
+batching.
+
+Message brokers like Azure Service Bus, Azure Event Hub and others have a
+special message format that can batch multiple AMQP messages into one. However
+the way to bind to that format is by specifying a binding to the individual
+messages and then have the proprietary batching model pick that up via the
+binding to the standard message.

spec/21-AMQP_FORMAT_RATIONALE.md

@@ -0,0 +1,86 @@
+# Rationale for AMQP protocol decisions
+
+## Strings vs. complex types
+
+Complex types in most cases cheaper than string representation of `traceparent`
+and `tracestate`. It is important in size restricted environments like IoT and
+price-sensitive infrastructures where one pays for message metadata stored in a
+queue server for a long time.
+
+Complex types are well defined and widely used in AMQP and they are part of a
+protocol. So using them is fully supported by all existing clients.
+
+## Binary `traceparent` vs. list of binary values
+
+There are multiple ways to implement a `traceparent`. It can either be
+implemented as string (http-like), binary protocol (like for grpc) or list of
+separate binary values (`trace-id`, `parent-id`, `trace-flags`).
+
+Strings duplicating the size of a field, using list of binaries will require to
+redefine the way the field serialized, parsed, and versioned. So re-using binary
+protocol looks like a logical solution.
+
+## AMQP map for `tracestate`
+
+The benefit of using a built-in map type for AMQP is that serialization and
+de-serialization of the field is built in and doesn't require any custom
+implementation for parsing it.
+
+Maps in AMQP preserving the order so there is no conflict with the semantics of
+the field.
+
+## Why use both - application and message properties
+
+Trace context is defined in an app in a context that mostly operates with the
+application-properties collection. So ideally trace context should be set and
+carried as part of application-properties collection. However,
+application-properties are immutable so there should be a fallback mechanism to
+use message-annotations if tracing must be implemented by one of message
+brokers.
+
+## Why not delivery-annotations
+
+There are three grand scenarios:
+
+*Application trace context* through a simple AMQP brokered entity. If you send a
+message into a queue from an app and then pull that message from the queue with
+the same or another app and nothing else interesting happens,
+application-properties is the right place, because you're not doing any tracing
+of the broker, you're just passing the context alongside the message.
+
+*Application context plus routing context*. If you have a chain of AMQP entities
+as you might have in a complex routing scenario, you might want to enrich the
+context as the message is being passed on. In that case, the prior scenario
+still exists, i.e. you will still want the option to have the context
+information as set by the message producer and use that for application level
+tracing unencumbered by whatever the routing layer might be tracking. As the
+routing layer gets it hand on the message for tracing, it basically forks the
+context off into the message-annotations and that's being manipulated on the
+route. As the message reaches the destination, you now have the app-level
+context from the producer in the application-properties AND the routing context
+that was layered on top of that in the message-annotations.
+
+Hop to hop tracing: A delivery-annotations usage scenario would be purely
+additive to either of the prior two, allowing any intermediary to spawn a new
+context for itself or to propagate a context only via subset of hops. I think
+that is fairly esoteric at this point and we should NOT take that into a spec
+unless someone shows a hard use-case.
+
+## Prefix of the field names
+
+The properties are typically prefixed because they’re generic and might
+interfere with an app’s use of the same names. So, for instance, instead of
+`traceparent` prefixed name like `w3c:traceparent` can be used. In general AMQP
+apps tend to be more metadata heavy than HTTP. For instance, `http:` is used in
+the HTTP-over-AMQP spec.
+
+The question is whether prefix is required for trace context and if so - what
+may be the name of the prefix. Options may be `w3c` for the origin of the spec,
+`tcx` for trace context, `dt` for distributed tracing.
+
+However in the current spec, even if customer decided to use the same name, the
+chance that customer didn’t mean to override those properties intentionally is
+very small. Those are rare names. So prefix doesn’t add much, but increases a
+chance for an error and interoperability.
+
+So suggestion is to keep the name un-prefixed.