Scaling - a proposal for a more scalable thingtracker

[edef1c]

The current approach does not scale in the slightest. I think an approach that invalidates a minimum amount of information at a time would be better.
My proposal does not cover inter-tracker stuff. We could specify that as part of the manifest.
This specification is still not optimal. My other idea involves a more CouchDB-y approach, perhaps an optional _changes feed that is tried if the cached version of the document set has been invalidated so that static trackers still work.
I don't think objects necessarily need to be part of a tracker. I think microdata is an excellent idea.

My proposal:

All files that are described by this specification SHOULD follow HTTP caching specifications.
All URLs should be SHOULD be relative, but absolute URLs MUST be accepted too.
A URL template works like this:
"/thing/{}" applied to 9 will yield "/thing/9"
"https://raw.github.com/{user}/{repo}/master/object.json" applied to { "user": "thedoctor", "repo": "fez" } will yield "https://raw.github.com/thedoctor/fez/master/object.json"

A tracker

• MUST have a manifest file
• MAY indicate the location of the manifest through a microformat in its index HTML file, otherwise objtracker.json is assumed (the objtracker.json is to be given priority and thus MUST be checked first)

A manifest

• MUST be a JSON document
• MUST have an objUrl property, a URL template string
• MAY have an objList property, a URL indicating the location of an object list

An object list

• MUST be a JSON document
• MUST consist of a single array
• the URLs formed by applying the objUrl template to a value from the object list MUST point at objects

An object

• MUST be a JSON document or an HTML document
• if the object is an HTML document, it MUST contain an object description in the form of microdata properties (XXX: define a microdata schema for them)
• if the object is a JSON document, it is equivalent to an object description

An object description

is something we've already kind of specified

[garyhodgson]

Hi Nathan,

At first I didn't get what you were trying to say, but now I see it, and your suggestion of dereferencing each level of the tracker is a good one that I think should be adopted.

Your concerns about scaling are well founded - a tracker with even only several hundred objects could result in several MB of data, and even with gzip compression this would be barely usable. My initial thoughts on tacking this was my means of an API - using offset and limit, or lastUpdated, parameters to limit the number of objects returned. Implementing an API is a greater undertaking than simply serving up a document, and your suggestion for dereferencing the data may mitigate the need for one, or at least minimise it.

Microdata is also a great idea, and coincides with my recent reading on Linked Data, which I am realising covers pretty much what we are trying to achieve. I am starting to think we should at least consider directing the efforts of the project towards being a vocabulary and implementation of Linked Data. Your suggestions fit in very well with such an approach.

Are you on Google+? I'd like to write about your suggestions to the TTN community there and would obviously like to have your input too.

Thanks,
Gary