Site routing in DAT?

[aral]

Not sure if this is the right place to hatch this idea (or whether it’s already been suggested elsewhere), but I want to put forward a use case and a possible implementation and get your thoughts on it (including on where I might have the wrong preconceptions and/or incorrect assumptions):

Use case

I don’t want a single archive per DAT site as this will quickly get rather large. I also would like to delete posts, etc., and that’s not currently possible.

For my blog, the sweet spot might be one archive (e.g.) per post.

Currently, as far as I can see in the DAT DNS DEP, I can set up a DNS mapping in ./well-known/dat for the root of the site and any paths off of that route are assumed to be in the same archive that the root maps to.

So, for dat://live.ar.al, for example, I have:

dat://bfb2eeb077826ecee6c1105d419755d5d8e0893d653d3ce39e50aee2c00b7701/
TTL=60

And the site is comprised of a single DAT archive (bfb2eeb…).

One way to implement a more granular mapping would be to extend (bastardise) this to include routing from multiple internal URLs to DAT public keys. However, this is not ideal for a number of reasons, including that the routing information itself would not be exposed over DAT so the site itself would not be a DAT site.

Suggestion

1. Beaker follows a DNS mapping as per the DAT DNS DEP
2. When/if it finds the root/index DAT archive, it looks for a routing file within the root of that archive (e.g., index-dat.json)
3. Beaker uses the routing information from that file, if it exists, to map paths relative to the root to multiple DAT archives.

e.g.,

{
  "routes": [
    {
      "path": "assets",
      "key": "6ef7628650d24ea0dbb8aa20788e9b6750c32a78e2e251a58a88c75c6bbd3876"
    },
    {
      "path": "videos",
      "key": "2968a49e9d87317c51d086bf497c648331efd8650fbc517ee0d9d25a00d104df"
    },
    {
      "path": "post1",
      "key": "87ed2e3b160f261a032af03921a3bd09227d0a4cde73466c17114816cae43336"
    },
  ]
}

So, for example, the above index would lead the following route mappings:

• dat://live.ar.al/assets/images/some-image.jpg → dat://6ef7628…/images/some-image.jpg
• dat://live.ar.al/videos/large-video.mp4 → dat://2968a49…/large-video.mp4_
  *dat://live.ar.al/post1/ → dat://87ed2e3b…/
• dat://live.ar.al/some-other-path/amazing.html → dat://bfb2eeb…/some-other-path/amazing.html (the default archive as presented by the DAT DNS lookup)

Advantages

• Index DAT file contains routing (and the HTML index of the site) and is under DAT itself
• Multiple archives allow for deletion of content
• Multiple archives allow for separation of concerns. Linking of other archives into a site (embedding archives)

Questions

• Apart from the deletion use case, how much of the file size/replication use case is already covered by sparse archives?

Thoughts? :)

[webdesserts]

@aral two things that might be of interest to you:

1. Just in case you weren't aware, you can already do SPA style routing by pointing the fallback_page in your dat.json to the index.html, where you can use the history/location apis to write fully customizable dynamic routes.
2. Paul just opened an issue for "mounting" or "symlinking" a dat which, in combination with the above, would allow you to split your site into multiple archives like you mentioned and route it however you want.

[pfrazee]

Yeah @webdesserts has the right answer. I'm very interested in the mount/symlink idea.

I also would like to delete posts, etc., and that’s not currently possible.

Quick clarification: because you can't remove them from the history?

[aral]

@pfrazee Ah, I was confused about deletion (still a lot to learn about DAT, making steady progress). Hadn’t realised DAT uses tombstones and garbage collection. Right, so the deletion/garbage collection use case then becomes for deleting history as per your comment.

[pfrazee]

Yeah. We haven't implemented history GC in Beaker yet but we plan to. I also think it'd be interesting to explore a "purge" tombstone that leads to automatic removal from the history cache.

[pfrazee]

We've decided 100% to move forward with mounts. Closing.