From 71228b93ebf1f39ba2e026b35da5a4836458d311 Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Thu, 18 Feb 2021 16:09:47 +0100 Subject: [PATCH 01/11] Draft AWS services spec --- specs/agents/tracing-instrumentation-aws.md | 53 +++++++++++++++++++++ specs/agents/tracing-instrumentation-db.md | 2 +- 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 specs/agents/tracing-instrumentation-aws.md diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md new file mode 100644 index 00000000..a3f201b6 --- /dev/null +++ b/specs/agents/tracing-instrumentation-aws.md @@ -0,0 +1,53 @@ +## AWS services spans + +We describe how to instrument some of AWS' services in this document. +Many of the services have a fairly small API beyond the basic administrative commands. We therefore +only instrument the most interesting and often-repeated operations. + +### S3 (Simple Storage Service) + +AWS Simple Storage Service offers object storage via a REST API. The objects are organized into buckets, which are +themselves organized into regions. + +- `span.name`: The span name should follow this pattern: `S3 `. For example, +`S3 GET_OBJECT myBucket`. Note that the operation name is in all caps with underscores (snake case in caps). +- `span.type`: `storage` +- `span.subtype`: `s3` +- `span.action`: The operation name in snake case in lower case. For example ‘get_object’. + +#### Span context fields + +- **`context.destination.address`**: optional. Not available in some cases. Only set if the actual connection is available. +- **`context.destination.port`**: optional. Not available in some cases. Only set if the actual connection is available. +- **`context.destination.region`**: mandatory. The AWS region where the bucket is. +- **`context.destination.service.name`**: mandatory. Use `s3` +- **`context.destination.service.resource`**: mandatory. The bucket ARN, if available, otherwise simply `s3/`. For example, `sqs/MyQueue` +- **`context.destination.region`**: mandatory. The AWS region where the queue is. +- **`context.destination.service.resource`**: mandatory. The queue url, which is required for all queue operations. + +### SNS (AWS Simple Notification Service) + +The AWS Simple Notification Service can be instrumented using the [messaging spec](tracing-instrumentation-messaging.md), +but the only action that is instrumented is `PUBLISH`. These specifications supersede those of the messaging spec: + +- `span.name`: The span name should follow this pattern: `SNS PUBLISH `. For example, +`SNS PUBISH MyTopic`. +- **`context.message.queue.name`**: mandatory. `sns/`. For example, `sns/MyTopic` +- **`context.destination.region`**: mandatory. The AWS region where the topic is. +- **`context.destination.service.resource`**: mandatory. The topic ARN. \ No newline at end of file diff --git a/specs/agents/tracing-instrumentation-db.md b/specs/agents/tracing-instrumentation-db.md index a934faee..0d19a4c3 100644 --- a/specs/agents/tracing-instrumentation-db.md +++ b/specs/agents/tracing-instrumentation-db.md @@ -2,7 +2,7 @@ We capture spans for various types of database/data-stores operations, such as SQL queries, Elasticsearch queries, Redis commands, etc. We follow some of the same conventions defined by OpenTracing for capturing database-specific span context, including: - - `db.instance`: database instance name, e.g. "customers" + - `db.instance`: database instance name, e.g. "customers". For DynamoDB, this is the region. - `db.statement`: statement/query, e.g. "SELECT * FROM foo" - `db.user`: username used for database access, e.g. "readonly_user" - `db.type`: database type/category, which should be "sql" for SQL databases, and the lower-cased database name otherwise. From d8f8a94ee3ee6c2b0ce5daf8e30d0077d60936d8 Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Tue, 23 Feb 2021 11:09:21 +0100 Subject: [PATCH 02/11] Add note about destination.region for DynamoDB --- specs/agents/tracing-instrumentation-aws.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index a3f201b6..4da60b68 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -32,6 +32,9 @@ AWS DynamoDB is a document database so instrumenting it will follow the [db spec Note that the `db.instance` field is the AWS region. +The following field should also be set for DynamoDB: +- **`context.destination.region`**: mandatory. The AWS region where the table is, if available. + ### SQS (Simple Queue Service ) AWS Simple Queue Service is a message queuing service. The [messaging spec](tracing-instrumentation-messaging.md) can From e6fbbff0baf89c8f78fa6a6a7c9b77c9101c691e Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Tue, 23 Feb 2021 11:36:37 +0100 Subject: [PATCH 03/11] Updates from review --- specs/agents/tracing-instrumentation-aws.md | 24 ++++++++------------- 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 4da60b68..06194759 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -1,19 +1,18 @@ ## AWS services spans We describe how to instrument some of AWS' services in this document. -Many of the services have a fairly small API beyond the basic administrative commands. We therefore -only instrument the most interesting and often-repeated operations. +Some of the services can use existing specs. When there are differences or additions, they have been noted below. ### S3 (Simple Storage Service) AWS Simple Storage Service offers object storage via a REST API. The objects are organized into buckets, which are themselves organized into regions. -- `span.name`: The span name should follow this pattern: `S3 `. For example, -`S3 GET_OBJECT myBucket`. Note that the operation name is in all caps with underscores (snake case in caps). +- `span.name`: The span name should follow this pattern: `S3 `. For example, +`S3 GetObject myBucket`. Note that the operation name is in CamelCase. - `span.type`: `storage` - `span.subtype`: `s3` -- `span.action`: The operation name in snake case in lower case. For example ‘get_object’. +- `span.action`: The operation name in CamelCase. For example ‘GetObject’. #### Span context fields @@ -21,16 +20,15 @@ themselves organized into regions. - **`context.destination.port`**: optional. Not available in some cases. Only set if the actual connection is available. - **`context.destination.region`**: mandatory. The AWS region where the bucket is. - **`context.destination.service.name`**: mandatory. Use `s3` -- **`context.destination.service.resource`**: mandatory. The bucket ARN, if available, otherwise simply `s3/`. For example, `sqs/MyQueue` - **`context.destination.region`**: mandatory. The AWS region where the queue is. -- **`context.destination.service.resource`**: mandatory. The queue url, which is required for all queue operations. ### SNS (AWS Simple Notification Service) @@ -51,6 +47,4 @@ but the only action that is instrumented is `PUBLISH`. These specifications supe - `span.name`: The span name should follow this pattern: `SNS PUBLISH `. For example, `SNS PUBISH MyTopic`. -- **`context.message.queue.name`**: mandatory. `sns/`. For example, `sns/MyTopic` -- **`context.destination.region`**: mandatory. The AWS region where the topic is. -- **`context.destination.service.resource`**: mandatory. The topic ARN. \ No newline at end of file +- **`context.destination.region`**: mandatory. The AWS region where the topic is. \ No newline at end of file From e159b482ec7957ff526d9d5130e03dae3dcf0f59 Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Tue, 23 Feb 2021 13:51:29 +0100 Subject: [PATCH 04/11] Add note about using message attributes as headers --- specs/agents/tracing-instrumentation-aws.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 06194759..399f4749 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -38,6 +38,8 @@ The following field should also be set for DynamoDB: AWS Simple Queue Service is a message queuing service. The [messaging spec](tracing-instrumentation-messaging.md) can be used for instrumenting SQS, but the follow specifications supersede those of the messaging spec. +For distributed tracing, the SQS API has "message attributes" that can be used in lieu of headers. + - **`context.destination.region`**: mandatory. The AWS region where the queue is. ### SNS (AWS Simple Notification Service) @@ -47,4 +49,6 @@ but the only action that is instrumented is `PUBLISH`. These specifications supe - `span.name`: The span name should follow this pattern: `SNS PUBLISH `. For example, `SNS PUBISH MyTopic`. -- **`context.destination.region`**: mandatory. The AWS region where the topic is. \ No newline at end of file +- **`context.destination.region`**: mandatory. The AWS region where the topic is. + +For distributed tracing, the SNS API has "message attributes" that can be used in lieu of headers. \ No newline at end of file From 8a3429d17e7cbbe7d4ccedf9e13d5e96b4b8dc62 Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Wed, 24 Feb 2021 10:35:26 +0100 Subject: [PATCH 05/11] Rearrange ordering of statements --- specs/agents/tracing-instrumentation-aws.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 399f4749..44a39644 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -38,10 +38,10 @@ The following field should also be set for DynamoDB: AWS Simple Queue Service is a message queuing service. The [messaging spec](tracing-instrumentation-messaging.md) can be used for instrumenting SQS, but the follow specifications supersede those of the messaging spec. -For distributed tracing, the SQS API has "message attributes" that can be used in lieu of headers. - - **`context.destination.region`**: mandatory. The AWS region where the queue is. +For distributed tracing, the SQS API has "message attributes" that can be used in lieu of headers. + ### SNS (AWS Simple Notification Service) The AWS Simple Notification Service can be instrumented using the [messaging spec](tracing-instrumentation-messaging.md), From 34e3b3ef5ea11c2d0f88c046022f282ad754ea4a Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Wed, 24 Feb 2021 16:26:39 +0100 Subject: [PATCH 06/11] Clarify what to extract from a bucket Access Point --- specs/agents/tracing-instrumentation-aws.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 44a39644..1e55679c 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -8,8 +8,8 @@ Some of the services can use existing specs. When there are differences or addit AWS Simple Storage Service offers object storage via a REST API. The objects are organized into buckets, which are themselves organized into regions. -- `span.name`: The span name should follow this pattern: `S3 `. For example, -`S3 GetObject myBucket`. Note that the operation name is in CamelCase. +- `span.name`: The span name should follow this pattern: `S3 `. For example, +`S3 GetObject my-bucket`. Note that the operation name is in CamelCase. - `span.type`: `storage` - `span.subtype`: `s3` - `span.action`: The operation name in CamelCase. For example ‘GetObject’. @@ -21,8 +21,9 @@ themselves organized into regions. - **`context.destination.region`**: mandatory. The AWS region where the bucket is. - **`context.destination.service.name`**: mandatory. Use `s3` - **`context.destination.service.resource`**: optional. The bucket name, if available. The s3 API allows either the -bucket name or the ARN to be provided when referring to a bucket. When an ARN is provided, the bucket name should -be extracted from it. +bucket name or an Access Point to be provided when referring to a bucket. When an Access Point is provided, the access +point name preceded by `accesspoint/` should be extracted. For example, given an Access Point such as +`arn:aws:s3:us-west-2:123456789012:accesspoint/myendpoint`, the agent extracts `accesspoint/myendpoint`. - **`context.destination.service.type`**: mandatory. Use `storage`. ### DynamoDB From 68cb345df55856456c77fddc48347b8a3a884143 Mon Sep 17 00:00:00 2001 From: Emily S Date: Thu, 25 Feb 2021 10:16:23 +0100 Subject: [PATCH 07/11] Update specs/agents/tracing-instrumentation-aws.md Co-authored-by: Andrew Wilkins --- specs/agents/tracing-instrumentation-aws.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 1e55679c..1e5505e3 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -34,7 +34,7 @@ Note that the `db.instance` field is the AWS region. The following field should also be set for DynamoDB: - **`context.destination.region`**: mandatory. The AWS region where the table is, if available. -### SQS (Simple Queue Service ) +### SQS (Simple Queue Service) AWS Simple Queue Service is a message queuing service. The [messaging spec](tracing-instrumentation-messaging.md) can be used for instrumenting SQS, but the follow specifications supersede those of the messaging spec. @@ -52,4 +52,4 @@ but the only action that is instrumented is `PUBLISH`. These specifications supe `SNS PUBISH MyTopic`. - **`context.destination.region`**: mandatory. The AWS region where the topic is. -For distributed tracing, the SNS API has "message attributes" that can be used in lieu of headers. \ No newline at end of file +For distributed tracing, the SNS API has "message attributes" that can be used in lieu of headers. From d8af79832284d9e823668ea5216d11dbca369049 Mon Sep 17 00:00:00 2001 From: Emily S Date: Thu, 25 Feb 2021 10:16:27 +0100 Subject: [PATCH 08/11] Update specs/agents/tracing-instrumentation-aws.md Co-authored-by: Andrew Wilkins --- specs/agents/tracing-instrumentation-aws.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 1e5505e3..5929a150 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -49,7 +49,7 @@ The AWS Simple Notification Service can be instrumented using the [messaging spe but the only action that is instrumented is `PUBLISH`. These specifications supersede those of the messaging spec: - `span.name`: The span name should follow this pattern: `SNS PUBLISH `. For example, -`SNS PUBISH MyTopic`. +`SNS PUBLISH MyTopic`. - **`context.destination.region`**: mandatory. The AWS region where the topic is. For distributed tracing, the SNS API has "message attributes" that can be used in lieu of headers. From 5845b6b6b48fc2d6c12c8b7013c84644f481ce5d Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Thu, 25 Feb 2021 10:33:44 +0100 Subject: [PATCH 09/11] Specify that Access Points can use slashes or colons --- specs/agents/tracing-instrumentation-aws.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 5929a150..9f8e4653 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -21,9 +21,11 @@ themselves organized into regions. - **`context.destination.region`**: mandatory. The AWS region where the bucket is. - **`context.destination.service.name`**: mandatory. Use `s3` - **`context.destination.service.resource`**: optional. The bucket name, if available. The s3 API allows either the -bucket name or an Access Point to be provided when referring to a bucket. When an Access Point is provided, the access -point name preceded by `accesspoint/` should be extracted. For example, given an Access Point such as -`arn:aws:s3:us-west-2:123456789012:accesspoint/myendpoint`, the agent extracts `accesspoint/myendpoint`. +bucket name or an Access Point to be provided when referring to a bucket. Access Points can use either slashes or colons. +When an Access Point is provided, the access point name preceded by `accesspoint/` or `accesspoint:` should be extracted. +For example, given an Access Point such as `arn:aws:s3:us-west-2:123456789012:accesspoint/myendpointslashes`, the agent +extracts `accesspoint/myendpointslashes`. Given an Access Point such as +`arn:aws:s3:us-west-2:123456789012:accesspoint:myendpointcolons`, the agent extracts `accesspoint:myendpointcolons`. - **`context.destination.service.type`**: mandatory. Use `storage`. ### DynamoDB From e28a905574509f4bf8e9dcd353f328ee9bdfec8c Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Mon, 1 Mar 2021 12:13:54 +0100 Subject: [PATCH 10/11] Add details for DynamoDB --- specs/agents/tracing-instrumentation-aws.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 9f8e4653..34181c4e 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -30,18 +30,23 @@ extracts `accesspoint/myendpointslashes`. Given an Access Point such as ### DynamoDB -AWS DynamoDB is a document database so instrumenting it will follow the [db spec](tracing-instrumentation-db.md). -Note that the `db.instance` field is the AWS region. +AWS DynamoDB is a document database so instrumenting it will follow the [db spec](tracing-instrumentation-db.md). +The follow specifications supersede those of the db spec. -The following field should also be set for DynamoDB: -- **`context.destination.region`**: mandatory. The AWS region where the table is, if available. +- **`span.name`**: The span name should capture the operation name in CamelCase and the table name, if available. +The format should be `DynamoDB `. So for example, `DynamoDB UpdateItem my_table`. + +#### Span context fields +- **`context.db.instance`**: mandatory. The AWS region where the table is. +- **`context.db.statement`**: optional. For a DynamoDB `query` operation, capture the `KeyConditionExpression` in this field. +- **`context.destination.cloud.region`**: mandatory. The AWS region where the table is, if available. ### SQS (Simple Queue Service) AWS Simple Queue Service is a message queuing service. The [messaging spec](tracing-instrumentation-messaging.md) can be used for instrumenting SQS, but the follow specifications supersede those of the messaging spec. -- **`context.destination.region`**: mandatory. The AWS region where the queue is. +- **`context.destination.cloud.region`**: mandatory. The AWS region where the queue is. For distributed tracing, the SQS API has "message attributes" that can be used in lieu of headers. @@ -52,6 +57,7 @@ but the only action that is instrumented is `PUBLISH`. These specifications supe - `span.name`: The span name should follow this pattern: `SNS PUBLISH `. For example, `SNS PUBLISH MyTopic`. -- **`context.destination.region`**: mandatory. The AWS region where the topic is. + +- **`context.destination.cloud.region`**: mandatory. The AWS region where the topic is. For distributed tracing, the SNS API has "message attributes" that can be used in lieu of headers. From d0d930ca3e776d48fc9780cccbe71b453ea530cd Mon Sep 17 00:00:00 2001 From: Emily Stolfo Date: Mon, 1 Mar 2021 12:29:26 +0100 Subject: [PATCH 11/11] Query should be capitalized for DynamoDb --- specs/agents/tracing-instrumentation-aws.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-aws.md b/specs/agents/tracing-instrumentation-aws.md index 34181c4e..c42d6043 100644 --- a/specs/agents/tracing-instrumentation-aws.md +++ b/specs/agents/tracing-instrumentation-aws.md @@ -38,7 +38,7 @@ The format should be `DynamoDB `. So for example, `Dynam #### Span context fields - **`context.db.instance`**: mandatory. The AWS region where the table is. -- **`context.db.statement`**: optional. For a DynamoDB `query` operation, capture the `KeyConditionExpression` in this field. +- **`context.db.statement`**: optional. For a DynamoDB `Query` operation, capture the `KeyConditionExpression` in this field. - **`context.destination.cloud.region`**: mandatory. The AWS region where the table is, if available. ### SQS (Simple Queue Service)