Dash integration

[hobofan]

I contacted @Kapeli regarding a docs.rs integration for Dash, similar to ones of rubygems.org, etc..

He said that while it is on the roadmap, he can't say when he will get to it. However, to make the process as smooth as possible, docs.rs should be ready for when the time comes. 😉

Quote from the email:

What I would need from docs.rs is:

1. A way to search for packages
2. A way to get the available versions for a given package
3. A way to download the docs in HTML format for a given package and version

Current status:

1. should be pretty straightforward, since a similar route already exists, so we just need a JSON version of it
2. AFAIK there is no specfic route for that yet, but since the same info is already shown at other places, so I guess the groundwork for that is there, and it just needs to be exposed on its own route
3. Route also doesn't exist yet (related to API for programmatic documentation fetching #88). I guess the best way to achieve it would be to generate a .tar.gz archive on the fly when it is first requested, and then cache that for subsequent requests. From what I can see, the tar crate makes it pretty easy to compress directories into a file.

---

@onur If the proposed approaches sound okay to you, I'll prepare a PR for each of routes.

[emoon]

Any update on this? I would love to have docs.rs support in Dash.

[hobofan]

@Kapeli also created an issue regarding a Dash integration: #174

But other than that, no progress really. My first pull request is still open, and I'd still like to do further work towards it, if that one is accepted.

Maybe now that the repo has been moved under the rust-lang-nursery umbrella, things can move forward a bit...

[zargony]

I'd like to see this happen too. I'm voting for crate docs support in Dash for years now (almost thought I'm the only one who'd like it).

Kapeli stated that he just needs docs to be downloadable in some way. But since docsets seem to be pretty common (e.g. Zeal on Linux also uses them), wouldn't it be more versatile for docs.rs to provide docsets directly? (or maybe some other filetype if there's a more popular one).
Additional processing for creating docs upon a crate release isn't probably much. However, additional disk storage size would be significant.

[hobofan]

I'm not sure if doc.rs should have native support for docsets. I think the current way of having one canonical doc format, that can then be transformed into others by the user is a more proper way, and also places less maintenance burden on the docs.rs team.

My rough plan was that once the features outlined in this issues have landed, I would implement support for using them from docs.rs in https://github.com/hobofan/rsdocs-dashing, and also expose rsdocs-dashing as a library. It should then be trivial to generate docsets for existing docs.rs documentation on-the-fly for other docset consumers, like Zeal users.

[fanzeyi]

I added a route for retrieving basic information of a given crate that includes available versions of the crate: fanzeyi@99eb7ca This should address the need 2 of this issue.

[fanzeyi]

On building archives of Rustdoc, I wonder if it is possible to simply run some sort of tar command at the end of the documentation building process, and we can put this archive file into the database along with other files. Dash and other documentation viewers can simply download that archive as accessing other files.

The downside of this method is that we would need to rebuild all the existing documents which is going to be an exhausting process :/ We probably can have some sort of script fetching the file contents from database and compile them into a tar archive then store it in the database.

[saskenuba]

Any updates on this?

[jyn514]

wonder if it is possible to simply run some sort of tar command at the end of the documentation building process, and we can put this archive file into the database along with other files.

The problem with this is it will greatly increase our storage requirements. I don't have hard numbers, but we currently store about 3 TB of docs on S3. If we stored a compressed archive as well, I expect that to easily go up to 3.5. These storage costs are in perpetuity; we can't get rid of old versions because people are still using them.

Actually creating the archive is relativity cheap compared to the cost of the build, it's just storage costs that are an issue.

The downside of this method is that we would need to rebuild all the existing documents

This is unlikely to happen, but we have an issue open for a while: #464

[jyn514]

Also note that it's not really feasible to store only the compressed archive because we would have to download the whole thing from S3 anytime someone visited a page for the crate+version. For large crates like winapi, that would greatly increase our egress costs.

[jyn514]

Whoops, it looks like all this has been said before. Closing as duplicate of #174

[jyn514]

Also note that it's not really feasible to store only the compressed archive because we would have to download the whole thing from S3 anytime someone visited a page for the crate+version. For large crates like winapi, that would greatly increase our egress costs.

Correction: egress is free, but downloading the entire archive would kill our response time for single pages.