rustdoc: Implement cross-crate searching #12946
Conversation
|
The crates column should have a consistent order (alphabetical?). I really like this, but I'm slightly nervous about it: it makes the documentation index a huge(r) blob (especially with syntax and rustc, like the online docs have) that is loaded unconditionally: #12597 is a bigger priority with this change IMO. |
|
The search index is loaded asynchronously, so the page is usable before its loaded. Additionally, it's super-cacheable because all crates and pages share the exact same index. I agree that it's a little large. With rustc/syntax, it's 1.7MB, 264K compressed. This is what we're buying into though if we want to be able to search across crates. For comparison, google's homepage downloaded over a megabyte of JS. |
|
There should probably be some indication that you're going to a different crate. I'd be confused if I found |
The search isn't usable immediately; and, on even just mildly slow internet connections, the delay between page loading and search working is annoying.
1 MB after decompressing it, and I'm sure their mobile site is significantly smaller.
Do we actually set the appropriate caching headers so that it can be cached? |
Each page has an accurate "breadcrumb trail" at the top, you think it should be more obvious than that? Everything is prefixed with what crate it's from.
Our options here are abandon this completely or remove search altogether, otherwise I think this is basically a fact of how it works?
It's all hosted on S3, which apparently does the right thing. I'm getting lots of 304's on the homepage. |
With compression, the transfer time would be much smaller; or if we had the option of a dynamic backend the main website wouldn't need to transfer anything. (The former is the solution I'm suggesting we should try to adopt as soon as possible.)
Ok, awesome. |
| }; | ||
| if ret == -1 { | ||
| unsafe { libc::close(fd); } | ||
| fail!() |
There was a problem hiding this comment.
fail!("could not lock {}", p.display())? (And similarly below.)
There was a problem hiding this comment.
Also, won't this cause rustdoc to just fail when another rustdoc process is running? Should it be sleeping and then retrying a few times? (My reading of the man-page is that fcntl doesn't block and just returns immediately?)
There was a problem hiding this comment.
Apparently F_SETLKW is special (this code is all from online examples, I'm not intimately familiar with file locking).
From the manpage:
F_SETLKW
This command is the same as F_SETLK except that if a shared
or exclusive lock is blocked by other locks, the process
waits until the request can be satisfied. If a signal that
is to be caught is received while fcntl is waiting for a
region, the fcntl will be interrupted if the signal handler
has not specified the SA_RESTART (see sigaction(2)).
Which means that this should probably handle EINTR, but rustdoc usage does not currently generate any signals.
There was a problem hiding this comment.
Ah, of course, I was reading the section for F_SETLK; my mistake.
|
r=me after comments (especially the file locking stuff). |
|
FYI: it compresses from 838,669 bytes down to 135,868. |
|
It shouldn't be too bad getting |
A major discoverability issue with rustdoc is that all crates have their documentation built in isolation, so it's difficult when looking at the documentation for libstd to learn that there's a libcollections crate with a HashMap in it. This commit moves rustdoc a little closer to improving the multiple crate experience. This unifies all search indexes for all crates into one file so all pages share the same search index. This allows searching to work across crates in the same documentation directory (as the standard distribution is currently built). This strategy involves updating a shared file amongst many rustdoc processes, so I implemented a simple file locking API for handling synchronization for updates to the shared files. cc rust-lang#12554
Update bug_report.md Closes rust-lang/rust-analyzer#12944
A major discoverability issue with rustdoc is that all crates have their
documentation built in isolation, so it's difficult when looking at the
documentation for libstd to learn that there's a libcollections crate with a
HashMap in it.
This commit moves rustdoc a little closer to improving the multiple crate
experience. This unifies all search indexes for all crates into one file so all
pages share the same search index. This allows searching to work across crates
in the same documentation directory (as the standard distribution is currently
built).
This strategy involves updating a shared file amongst many rustdoc processes, so
I implemented a simple file locking API for handling synchronization for updates
to the shared files.
cc #12554