-
-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: proposed set of bindings for AWS - SNS and SQS #84
Changes from 8 commits
b64dcf6
cc91daa
c322a0b
0d2956a
6e6bbb3
af5fa86
325291c
6f939c6
d5493f4
f97af36
94443a0
cada568
471e92a
0c8a141
f82ac68
57d67ed
1139f7a
c7a202a
1690fe5
1d33e72
cfa8115
192c7ce
27829d9
08d2ba2
401ec78
61cc69a
eaa8bec
a7d982f
0f75115
162d72c
6c495ba
a71f798
d7ad6c2
677e095
9fbc3e6
807dc5c
8c3a1fe
f5dbc92
fb4c786
8533ae3
d05406d
d0f3921
d3367a3
414eb68
bfbcdd0
4a6aa9f
d775f18
d2d9da5
a4b440e
504dc1e
08a5bfd
f11a17d
ce32e23
31d0df8
9cc5f83
4e946c5
d78083c
c988dbd
e9d2979
2699386
4bb2c4f
df95b2f
1dedbbe
2912e3b
87c0756
2417fff
bd26fa4
f5b6060
a9fbb73
8c637f4
0f947a6
343fe18
bd462c5
891a4ec
45a5181
c6df5d8
279a08c
5491c71
3e13191
604f8ff
32f6e8c
bf3d591
3f3dc89
49dc0b2
916ce97
9b4d174
3fa08af
7d089e6
25c0134
38a9a3c
edf0563
b780cca
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,36 +2,237 @@ | |
|
||
This document defines how to describe SQS-specific information on AsyncAPI. | ||
|
||
SQS can be used both stand-alone as a point-to-point and paired with SNS and as a publish-subscribe channel (where SQS is the endpoint that SNS delivers messages to). For this reason we define a Queue schema, and reference that schema from both a Channel Binding Object and a **publish** Operation Binding Object. | ||
|
||
For point-to-point scenarios, use the Channel Binding Object, as producers send to the queue and consumers receive from it directly. | ||
|
||
For publish-subscribe scenarios, use as a **publish** Operation Binding Object, as the producer sends to SNS and the consumer receives via SQS. | ||
|
||
<a name="version"></a> | ||
|
||
## Version | ||
|
||
Current version is `0.1.0`. | ||
|
||
|
||
<a name="server"></a> | ||
|
||
## Server Binding Object | ||
|
||
This object MUST NOT contain any properties. Its name is reserved for future use. | ||
|
||
|
||
|
||
<a name="channel"></a> | ||
|
||
## Channel Binding Object | ||
|
||
This object MUST NOT contain any properties. Its name is reserved for future use. | ||
|
||
## Channel Binding Object | ||
|
||
Use the Channel Binding Operation for Point-to-Point SQS channels. | ||
|
||
There are three likely scenarios for use of the Channel Binding Object: | ||
|
||
- One file defines both publish and subscribe operations, for example if we were implementing the work queue pattern to offload work from an HTTP API endpoint to a worker process. In this case the channel would be defined on the Channel Object in that single file. | ||
- The producer and consumer both have an AsyncAPI specification file, and the producer is raising an event, for example interop between microservices, and the producer 'owns' the channel definition and thus has the SQS Binding on its Channel Object. | ||
- The producer and consumer both have an AsyncAPI specification file, and the consumer receives commands, for example interop between microservices, and the consumer 'owns' the channel definition and thus has the SQS Binding on its Channel Object. | ||
|
||
An SQS queue can set up a Dead Letter Queue as part of a Redelivery Policy. To support this requirement, the Channel Binding Object allows you to define both a Queue Object to use as the Channel or target in a *publish* Operation and a Dead Letter Queue. You can then refer to the Dead letter Queue in the Redrive Policy using the Identifier Object and setting the *name* field to match the *name* field of your Dead Letter Queue Object. (If you define the DLQ externally, the Identifier also supports an ARN). | ||
|
||
### Fields | ||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
| <a name="channelBindingObjectQueue"></a>`queue` | [Queue](#queue)| **Required.** A definition of the queue that will be used as the channel. | | ||
| <a name="channelBindingObjectDLQ"></a>`deadLetterQueue` | [Queue](#queue)| **Optional.** A definition of the queue that will be used as the channel. | | ||
|<a name="channelBindingObjectBindingVersion"></a>`bindingVersion` | string | **Optional**, defaults to `latest`. The version of this binding.| | ||
|
||
### Schemas | ||
|
||
#### Queue | ||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
|<a name="queueObjectRef"></a>$ref | `string` | Allows for an external definition of a queue. The referenced structure MUST be in the format of a [Queue](#queue). If there are conflicts between the referenced definition and this Queue's definition, the behavior is *undefined*.| | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Defined where? Is it a link, or is 'external' meaning it should go into There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can remove the queueObjectRef because that looks like a URI Reference explained in RFC3986. This can be either relative or absolute reference. URI References can be used for any object when it's needed. Any software libraries which serialise/deserialise JSON or YAML files must provide the option (or the default) to handle the references. Because the URI Reference property ($ref) is a standard and can be used in any object I think it's unnecessary to specify the $ref property again in the Queue object. |
||
| <a name="queueObjectName"></a>`name` | string | **Required.** The name of the queue. When an [SNS Operation Binding Object]() references an SQS queue by name, the identifier should be the one in this field.| | ||
| <a name="queueObjectType"></a>`type` | string one of: fifo, standard | **Required.** Is this a FIFO queue or a standard queue? | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Any reason to use a string field in here instead of a Boolean for |
||
| <a name="queueObjectDeliveryDelay"></a>`deliveryDelay` | integer | **Optional.** The number of seconds to delay before a message sent to the queue can be received. used to create a *delay queue*. Range is 0 to 15 minutes. Defaults to 0. | | ||
| <a name="queueObjectVisbilityTimeout"></a>`visibilityTimeout` |integer| **Optional.** The length of time, in seconds, that a consumer locks a message - hiding it from reads - before it is unlocked and can be read again. Range from 0 to 12 hours. Defaults to 30 seconds. | | ||
| <a name="queueObjectRecieveMessageWaitTime"></a>`receiveMessageWaitTime` |integer| **Optional.** Determines if the queue uses [short polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) or [long polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html). On a 0 (the default) the queue reads available messages and returns immediately. On a non-zero integer, long polling waits the specified number of seconds for messages to arrive before returning. | | ||
| <a name="queueObjectMessageRetentionPeriod"></a>`messageRetentionPeriod` |integer| **Optional.** How long to retain a message on the queue, unless deleted, in days. The rang is 1 minute to 14 days. The default is 4 days. | | ||
| <a name="queueObjectRedrivePolicy"></a>`reDrivePolicy` | [Redrive Policy](#redrive-policy) | **Optional.** Prevent poison pill messages by moving un-processable messages to an SQS dead letter queue.| | ||
| <a name="queueObjectPolicy"></a>`policy` |[Policy](#policy) | **Optional.** The security policy for the SQS Queue | | ||
| <a name="queueObjectTags"></a>`tags` |Object | **Optional.** Key-value pairs that represent AWS tags on the queue. | | ||
|
||
#### Identifier | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can an |
||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
|<a name="identifierObjectArn"></a>`arn` |string| **Optional.** The target is an [ARN](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html). For example, for SQS, the identifier may be an ARN, which will be of the form: ["arn:aws:sqs:{region}:{account-id}:{queueName}"](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html)| | ||
|<a name="identifierObjectName"></a>`name` |string| **Optional.** The endpoint is identified by a name, which corresponds to an identifying field called 'name' of a binding for that protocol on this **publish** Operation Object. For example, if the protocol is 'sqs' then the name refers to the name field **sqs** binding| | ||
|
||
#### Policy | ||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
| <a name="channelBindingPolicyObjectPolicyStatements"></a>`Statements` | [Statement](#statement) | **Required.** An array of Statement objects, each of which controls a permission for this topic | | ||
|
||
#### Redrive Policy | ||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
| <a name="redrivePolicyObjectDeadLetterQueue"></a>`deadLetterQueue` |[Identifier](#identifier)| **Required.** The SQS queue to use as a dead letter queue (DLQ) | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @iancooper why DeadLetterQueue is required? Don't you think that the Redrive policy should be a decision of the application and the spec shouldn't enforce it? |
||
| <a name="redrivePolicyObjectMaxReceiveCount"></a>`maxReceiveCount` |integer| **Optional.** The SQS queue to use as a dead letter queue (DLQ) | ||
|
||
#### Statement | ||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
| <a name="channelBindingPolicyStatementObjectEffect"></a>`effect` | string |**Required.** Either "Allow" or "Deny"| | ||
| <a name="channelBindingPolicyStatementObjectPrincipal"></a>`principal` | string or array of string |**Required.** The AWS account or IAM user to whom this statement applies| | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same remark as SNS. |
||
| <a name="channelBindingPolicyStatementObjectAction"></a>`action` | string or array of string |**Required.** The SNS permission being allowed or denied e.g. sns:Publish| | ||
|
||
|
||
<a name="operation"></a> | ||
|
||
## Operation Binding Object | ||
|
||
This object MUST NOT contain any properties. Its name is reserved for future use. | ||
### SQS Point-To-Point | ||
|
||
Because we have defined Queue as part of the Channel Binding Binding object, we do not require Binding information for the **publish** Operation Object of the **subscribe** Operation Object. You can use an empty Queue object ({}) to denote the Binding on the Operation Object, if you want to indicate the protocol used to send or receive for generation purposes such as Infrastructure As Code. | ||
|
||
### SNS to SQS Pub-Sub | ||
|
||
Use the Operation Binding Object when SQS is listening to an SNS Topic. In this case we need to define both an SQS Operation Binding Objects on the receiver **publish** Operation Object to represent the queue definition and we need to define an SNS Operation Binding Object to define the Subscription to SNS that makes your queue a receiver of that endpoint. | ||
|
||
Assuming you have separate AsyncAPI specifications for the producer and the consumer, we would assume the following bindings would appear for an SNS producer and an SQS consumer. | ||
|
||
Producer: SNS Channel Binding Object, SNS **subscribe** Operation Binding Object [if required] | ||
Consumer: SNS **publish** Operation Binding Object, SQS **publish** Operation Binding Object | ||
|
||
- We assume that the SNS binding information only needs to be present in the producer file (although defining it in both is allowable) and any infrastructure as code dependencies can recognize this. | ||
|
||
|
||
On an Operation Binding Object we support an array of Queue objects. Members of this array may be Queue Objects that define the *endpoint* field required by an [SNS Operation Object]() delivering by the SQS protocol or Queue Objects that define the Dead Letter Queue used by either the Redrive Policy of the SNS Subscription (see the SNS Binding Object) or the [Redrive Policy of the SQS Queue](#redrive-policy). The name of the Queue Object is used by an Identifier field on either the *endpoint* field of the SNS Operation Object of *deadLetterQueue* on the Redrive Policy to identify the required member of this array. | ||
|
||
|
||
### Fields | ||
|Field Name | Type | Description| | ||
|---|:---:|---| | ||
| <a name="channelBindingObjectQueue"></a>`queues` | [[Queue](#queue)]| **Required.** Queue objects that are either the *endpoint* for an SNS Operation Binding Object, or the *deadLetterQueue* of the SQS Operation Binding Object | | ||
|<a name="channelBindingObjectBindingVersion"></a>`bindingVersion` | string | **Optional**, defaults to `latest`. The version of this binding.| | ||
|
||
### Examples | ||
|
||
#### SQS Point-To-Point | ||
|
||
[<img src="SQS-Point-To-Point.png" height = "400" width="600"/>](SQS-Point-To-Point.png) | ||
|
||
In this example, we are using SQS for a point-to-point channel. For this example, we assume that we are defining two microservices that communicate over a shared SQS channel, with the consumer receiving events over that channel and the producer owning the channel definition. | ||
|
||
The producer file would look like this: | ||
|
||
```yaml | ||
channels: | ||
user-signedup: | ||
bindings: | ||
sqs: | ||
queue: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From comment above, shouldn't this be |
||
name: user-signedup-queue | ||
type: standard | ||
receiveMessageWaitTime: 4 | ||
reDrivePolicy: | ||
deadLetterQueue: | ||
name: user-signedup-dlq | ||
policy: | ||
statements: | ||
- effect : Allow | ||
principal: * | ||
action: Sqs:SendMessage | ||
- effect : Allow | ||
principal: * | ||
action: Sqs:ReceiveMessage | ||
deadLetterQueue: | ||
name: user-signedup-dlq | ||
messageRetentionPeriod: 1209600 | ||
type: standard | ||
subscribe: | ||
operationId: sendMessage | ||
description: sends messages when a user has signed up | ||
bindings: | ||
sqs: {} | ||
|
||
``` | ||
In this case we can minimize duplicated information by omitting the binding in our specification, and assume it is picked up from the producer file. We can use an empty object to indicate the SQS Binding on the **publish** Operation Object, if need a marker for generation, otherwise we could omit the Operation Binding Object. | ||
|
||
```yaml | ||
channels: | ||
user-signedup: | ||
publish: | ||
operationId: receiveMessage | ||
description: receives a messages when a user has signed up | ||
bindings: | ||
sqs: {} | ||
|
||
``` | ||
|
||
#### SNS to SQS Pub-Sub | ||
|
||
[<img src="SNS-SQS-Pub-Sub.png" height = "400" width="600"/>](SNS-SQS-Pub-Sub.png) | ||
|
||
In this example, we are using SNS for the channel, and SQS to receive from SNS. | ||
|
||
|
||
The producer files looks like this (see the [SNS Binding]() for more). | ||
|
||
```yaml | ||
channels: | ||
user-signedup: | ||
description: A user has signed up for our service | ||
binding : | ||
sns: {} # Indicates that the channel is an SNS Topic | ||
subscribe: | ||
operationId: sendMessage | ||
description: send messages to the topic | ||
bindings: | ||
sns: | ||
policy: | ||
statements: | ||
- effect : Allow | ||
principal: * | ||
action: SNS:Publish | ||
``` | ||
|
||
And the consumer file would look like this. Note that for simplicity, we choose not to repeat the SNS Binding on the Consumer as it does not 'own' the channel. | ||
|
||
|
||
```yaml | ||
channels: | ||
user-signedup: | ||
description: A user has signed up for our service | ||
publish: | ||
operationId: receiveMessage | ||
description: receive messages from the topic | ||
bindings: | ||
sns: | ||
consumers: | ||
- protocol: sqs | ||
endpoint: | ||
name: user-signedup-queue | ||
rawMessageDelivery: true | ||
filterPolicy: | ||
attributes: | ||
reason: | ||
anything-but: password-reset | ||
reDrivePolicy: | ||
deadLetterQueue: | ||
name: user-signedup-queue-dlq | ||
sqs: | ||
queues: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I noticed that this item uses There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hope my comment above addressed this. At the |
||
- name: user-signedup-queue | ||
type: standard | ||
receiveMessageWaitTime: 4 | ||
policy: | ||
statements: | ||
- effect : Allow | ||
principal: * | ||
action: Sqs:SendMessage | ||
- effect : Allow | ||
principal: * | ||
action: Sqs:ReceiveMessage | ||
- name: user-signedup-dlq | ||
messageRetentionPeriod: 1209600 | ||
type: standard | ||
``` | ||
|
||
<a name="message"></a> | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should'n the name here be
queues
? This is an array, so why is it singular? Was that a typo?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeff9finger I don't think the type is an array. It's of type
Queue
which I think is consistent with source bindings living at this level of the specification.The
queues
property does exist on the proposed binding but its part of the binding that sits at thepublish
level for facilitating things like ansns
configured at the channel level and then multiple consumer queues receiving messages. Example from the proposal below as you cannot link to specific MD lines.