Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Listing with chunked transfer #145

Merged
merged 3 commits into from
Mar 29, 2021
Merged

Listing with chunked transfer #145

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f4002c2
listing with chunked transfer
farshidtz Mar 23, 2021
f22b432
grammar fix: add article before body
farshidtz Mar 23, 2021
170ddf5
Add assertion for listing using HTTP/2 frames
farshidtz Mar 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions directory.td.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,21 @@
}
]
},
"retrieveTDs": {
"description": "Retrieve all Thing Descriptions",
"forms": [
{
"href": "/td",
"htv:methodName": "GET",
"response": {
"description": "Success response",
"htv:statusCodeValue": 200,
"contentType": "application/ld+json",
},
"scopes": "readAll"
}
]
},
"searchJSONPath": {
"description": "JSONPath syntactic search",
"uriVariables": {
Expand Down
141 changes: 36 additions & 105 deletions index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -985,93 +985,49 @@ <h4>Deletion</h4>
<section id="exploration-directory-api-registration-listing" class="normative">
<h4>Listing</h4>
<p>
<span class="rfc2119-assertion" id="tdd-reg-list">
The listing endpoint provides a way to query the collection of TD objects
from the directory. The Search API may be used to retrieve <a>Partial TD</a>s
or TD fragments; see [[[#exploration-directory-api-search]]].
</p>


<p>
<span class="rfc2119-assertion" id="tdd-reg-list-method">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assertions should ideally be self-contained. I would add "from the retrieveTDs property". BTW, "retrieveTDs" is a verb (phrase). Properties should ideally be nouns, and actions verbs. This is a bikeshed issue, but maybe something like "allTDs" makes more sense. Likewise, it seems query actions like "searchJSONPath" are properties... that doesn't necessarily make sense either.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, we don't have to fix this in this PR, but maybe we should create another issue to clean this up. We should probably create an actual TM also.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this point is being discussed in #133

The list of TDs MUST be retrieved from the directory using an
HTTP `GET` request to the root endpoint of the registration API.
HTTP `GET` request.
</span>

<span class="rfc2119-assertion" id="tdd-reg-list-resp">
A successful response MUST have 200 (OK) status, contain `application/ld+json`
Content-Type header, and an array of TDs in body.
farshidtz marked this conversation as resolved.
Show resolved Hide resolved
</span>
</p>
<p> <!-- paginated listing -->
<span class="rfc2119-assertion" id="tdd-reg-list-resp-pagination">
The default response MUST be paginated to limit the
set of enclosed TDs in a single response.

<p>
Serializing and returning the full list of TDs may be burdensome to servers.
As such, servers should serialize incrementally and utilize protocol-specific
mechanisms to respond in chunks.
<span class="rfc2119-assertion" id="tdd-reg-list-http11">
HTTP/1.1 servers SHOULD perform chunked
<a href="https://tools.ietf.org/html/rfc7230#section-3.3.1">Transfer-Encoding</a> [[RFC7230]]
to respond the data incrementally.
</span>
The pagination must be based on the following rules:
<ul>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-resp">
A successful response MUST have 200 (OK) status, contain `application/ld+json`
Content-Type header, and a set of TDs in a collection object in body.
</span>
</li>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-envelop">
The TDs MUST be complete TD objects inside an array
called `items` enclosed in the collection object.
</span>
</li>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-total">
The number of TDs in each response MUST match the requested number.
</span>
The server should enforce an appropriate or configurable maximum number
of TDs per request to manage the processing load on the server side.
</li>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-total-default">
When a total number is not given in a request, it MUST default to 1.
</span>
This is to avoid unintended traffic and load on memory-constrained clients.
</li>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-id">
The collection object MUST contains an `id` attribute
pointing to the current page (self link).
</span>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-links">
The collection object MUST contains a `nextLink` attribute
for querying the next page, when present.
</span>
The links may be relative to directory API base URL, or be absolute URLs.
These links may include additional arguments for ordering or session keys.
</li>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-total">
When requested, the collection object MUST contain a `total` attribute
equal to total number of TDs within the directory.
</span>
</li>
<li>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-order-default">
By default, the collection MUST be sorted alphanumerically
by the unique identifiers of TDs.
</span>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-order">
The server MAY support sorting by other TD attributes (e.g. `created` field) and
different sorting orders (i.e. ascendingly/descendingly).
</span>
<span class="rfc2119-assertion" id="tdd-reg-list-pagination-order-unsupported">
If the server does not support a given sorting argument,
it MUST reject the request with 501 (Not Implemented) status.
</span>
</li>
</ul>
Most HTTP/1.1 clients automatically process the data received with
chunked transfer encoding.
Memory-constrained applications which require the full list
should consider processing the received data incrementally.

<p class="ednote" title="HTTP/2 chunking">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I see you do include this. Good. Yeah, testing is going to be a pain. However, we probably DO need to include support for HTTP/2.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My implementation supports both HTTP/1.1 and 2 (built-in Go HTTP library). I just need to prepare SSL certificates (generally required for HTTP/2) to and simply test it with a modern browser and curl.

Copy link
Member Author

@farshidtz farshidtz Mar 26, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I finished testing this in the following ways:

  • HTTP/2 proxy in front of the directory. This was as simple as adding an http2 flag to the respective Nginx listener. The proxy was able to translate HTTP/1.1 chunked encoding to an HTTP/2 stream. The HTTP/2 performance (speed) was twice better than HTTP/1.1 when tested over the internet.
  • Adding TLS certificates on the directory. This enabled the serving HTTP/2 clients with the HTTP/2 stream. A benchmark showed slightly better performance compared to HTTP/1.1 over TLS when tested in a local network.

I will go ahead with taking the assertion out of the ednote.

This should be tested before turned into an assertion:<br>
Chunked transfer encoding is not supported in HTTP/2.
HTTP/2 servers SHOULD respond the data incrementally using frames.
</p>
</p>

<!-- <p>
The paginated list operation is specified as `listTD` property in
<p>
The paginated list operation is specified as `retrieveTDs` property in
[[[#directory-thing-description]]].
</p> -->
<p class="ednote" title="Registration listing query arguments">
Need to specify and describe the query arguments to paginate
and get total. The paginated list operation should be specified as `listTD`
property in [[[#directory-thing-description]]].
</p>


<span class="rfc2119-assertion" id="tdd-reg-list-resp-alternative">
A server which supports alternative mechanisms (such as streaming)
MAY rely on server-driven content negotiation and respond accordingly.
</span>

<p>
Error responses:
Expand All @@ -1085,32 +1041,7 @@ <h4>Listing</h4>
</ul>
</p>

The following is an example paginated list response with total:
<aside class="example" id="example-td-registration-info" title="Example response for a paginated list with total">
<pre>
{
"@context": "https://w3c.github.io/wot-discovery/context/discovery-context.jsonld",
"id": "/td?offset=0&limit=10",
"type": "Collection",
"items": [
{
"@context": "https://www.w3.org/2019/wot/td/v1",
"id": "urn:example:simple-td",
"title": "Simple TD",
"security": "basic_sc",
"securityDefinitions": {
"basic_sc": {
"scheme": "basic"
}
}
},
...
],
"total": 350,
"nextLink": "/td?offset=10&limit=10"
}
</pre>
</aside>

</section>

<section id="validation" class="normative">
Expand Down