Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rust_doc: add web server for docs #475

Closed
wants to merge 16 commits into from

Conversation

wchargin
Copy link
Contributor

@wchargin wchargin commented Nov 4, 2020

A rust_doc rule emits its output into a zip archive so that the build
graph can be statically known, but this makes it hard to actually view
the documentation. This patch adds a rust_doc_server macro, a sibling
to rust_doc that spins up a tiny web server that reads from the zip
file. The server can run under ibazel and automatically restart when
the documentation sources are updated.

This is written in Python for a simple cross-platform solution that
doesn’t require compiling a Rust web stack, which takes about 20 seconds
on my workstation (Xeon W-2135).

Fixes #472.

Test Plan:
From examples/, run bazel run //hello_world:hello_world_doc_server
and note that the server responds properly. Run with -- --port 0
(extra -- for bazel run) and note that the server properly prints
its actual port.

wchargin-branch: rustdoc-server

The Zip archives emitted by `rust_doc` rules used to contain a long
`/bazel-bin/k8-fastbuild/path/to/target` prefix before each archive
entry name. This made it hard to work with the generated archive. This
patch adds a transformation to

This could be written in Starlark using the `DirectoryExapnder` API, but
that was only introduced in Bazel 3.4.0. Since we have a minimum Bazel
version of 0.17.1, we instead write the helper as a small Python script.

Fixes bazelbuild#471.

Test Plan:
Unit tests included. As an end-to-end test, run

```
cd examples/ &&
bazel build //hello_world:hello_world_doc &&
unzip -l bazel-bin/hello_world/hello_world_doc.zip
```

to list the contents of the built `hello_world_doc.zip` archive. Before
this patch, the result was like:

```
Archive:  bazel-bin/hello_world/hello_world_doc.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2010-01-01 00:00   bazel-out/k8-fastbuild/bin/hello_world/hello_world_doc/.lock
     1792  2010-01-01 00:00   bazel-out/k8-fastbuild/bin/hello_world/hello_world_doc/COPYRIGHT.txt
     4421  2010-01-01 00:00   bazel-out/k8-fastbuild/bin/hello_world/hello_world_doc/FiraSans-LICENSE.txt
...
     2132  2010-01-01 00:00   bazel-out/k8-fastbuild/bin/hello_world/hello_world_doc/storage.js
     1180  2010-01-01 00:00   bazel-out/k8-fastbuild/bin/hello_world/hello_world_doc/theme.js
     3764  2010-01-01 00:00   bazel-out/k8-fastbuild/bin/hello_world/hello_world_doc/wheel.svg
---------                     -------
   890555                     38 files
```

After this patch:

```
Archive:  bazel-bin/hello_world/hello_world_doc.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2010-01-01 00:00   .lock
     1792  2010-01-01 00:00   COPYRIGHT.txt
     4421  2010-01-01 00:00   FiraSans-LICENSE.txt
...
     2132  2010-01-01 00:00   storage.js
     1180  2010-01-01 00:00   theme.js
     3764  2010-01-01 00:00   wheel.svg
---------                     -------
   890555                     38 files
```

wchargin-branch: rustdoc-strip-prefix
wchargin-source: 29978cf3bdb7de6cee154f8c11d0251574c2b104
wchargin-branch: rustdoc-strip-prefix
wchargin-source: 69b2e78f1f52c88609aff631c908471c7521c5f7
A `rust_doc` rule emits its output into a zip archive so that the build
graph can be statically known, but this makes it hard to actually view
the documentation. This patch adds a `rust_doc_server` macro, a sibling
to `rust_doc` that spins up a tiny web server that reads from the zip
file. The server can run under `ibazel` and automatically restart when
the documentation sources are updated.

This is written in Python for a simple cross-platform solution that
doesn’t require compiling a Rust web stack, which takes about 20 seconds
on my workstation (Xeon W-2135).

Fixes bazelbuild#472.

Test Plan:
From `examples/`, run `bazel run //hello_world:hello_world_doc_server`
and note that the server responds properly. Run with `-- --port 0`
(extra `--` for `bazel run`) and note that the server properly prints
its actual port.

wchargin-branch: rustdoc-server
wchargin-source: 307dcd532d0f7880b1cceaf0421a51416607b44f
@google-cla google-cla bot added the cla: yes label Nov 4, 2020
@wchargin
Copy link
Contributor Author

wchargin commented Nov 4, 2020

This PR’s diffbase is the #474 branch, so please ignore the extraneous
changes (you can look at just a subset of the commits).

Opening this to get @dfreese’s opinion re: Python usage, per discussion
on #474. If the direction looks good to you, I’ll flesh out docs/polish.
(edit: added minimal docs to make CI happy).

Thanks! Happy to incorporate any feedback.

wchargin-branch: rustdoc-server
wchargin-source: bdb753c552346c721fdf558a15e95777f8214827
@dfreese
Copy link
Collaborator

dfreese commented Nov 5, 2020

From a high level design perspective, do we want to be generating a server target for every rustdoc target, or just a single binary target that serves as a tool that can handle any rustdoc output?

wchargin-branch: rustdoc-strip-prefix
wchargin-source: 40227180c7f910d9f76716cf42621e08e5f808a1
wchargin-branch: rustdoc-server
wchargin-source: 1f90836672a1cc8467b6b11e3b045497b8b05903
wchargin-branch: rustdoc-server
wchargin-source: 1f90836672a1cc8467b6b11e3b045497b8b05903
wchargin-branch: rustdoc-strip-prefix
wchargin-source: 8c46290195fc0668f3126589fd07ee863df8749f
wchargin-branch: rustdoc-server
wchargin-source: 62146a7c4b075d8567d2e92309c07b3c36bd5ea0
wchargin-branch: rustdoc-strip-prefix
wchargin-source: 838287ca34eb69c27f62e4a0900dbb7ac6a3ebd0
wchargin-branch: rustdoc-server
wchargin-source: 442c03cd289428a9f7f461159599daf6876686fd
wchargin added a commit to tensorflow/tensorboard that referenced this pull request Nov 9, 2020
Summary:
The Rust Bazel rules provide a `rust_doc` rule for documentation, but it
generates a zipfile with long file paths, which is inconvenient to
consume. This patch adds a simple Werkzeug server to serve from the
generated documentation. It’s compatible with `ibazel run`: just change
a Rust source file and refresh your browser tab to see your changes.

I have a [pending upstream patch for this][upstream], but I’m still
working on it, and it’s on my back burner. In the meantime, this is
helpful for both Rust authors and Rust reviewers on TensorBoard.

[upstream]: bazelbuild/rules_rust#475

Test Plan:
Run `bazel run //tensorboard/data/server:rustboard_core_doc` and
navigate to port 6005:

![Screenshot of the docs for `rustboard_core::masked_crc::MaskedCrc` in
a standard rustdoc output format][ss]

[ss]: https://user-images.githubusercontent.com/4317806/98600750-e788bd80-2292-11eb-9f09-fd51ed5e5996.png

wchargin-branch: rust-doc-server
wchargin-source: ddc3596e08606fb52ab63157900df1b3741fbfaf
wchargin added a commit to tensorflow/tensorboard that referenced this pull request Nov 11, 2020
Summary:
The Rust Bazel rules provide a `rust_doc` rule for documentation, but it
generates a zipfile with long file paths, which is inconvenient to
consume. This patch adds a simple Werkzeug server to serve from the
generated documentation. It’s compatible with `ibazel run`: just change
a Rust source file and refresh your browser tab to see your changes.

I have a [pending upstream patch for this][upstream], but I’m still
working on it, and it’s on my back burner. In the meantime, this is
helpful for both Rust authors and Rust reviewers on TensorBoard.

[upstream]: bazelbuild/rules_rust#475

Test Plan:
Run `bazel run //tensorboard/data/server:rustboard_core_doc_server` and
navigate to port 6005:

![Screenshot of the docs for `rustboard_core::masked_crc::MaskedCrc` in
a standard rustdoc output format][ss]

[ss]: https://user-images.githubusercontent.com/4317806/98600750-e788bd80-2292-11eb-9f09-fd51ed5e5996.png

wchargin-branch: rust-doc-server
wchargin-branch: rustdoc-strip-prefix
wchargin-source: ea88a0484df0608cb08521fb4cde2d4dc7904553
wchargin-branch: rustdoc-strip-prefix
wchargin-source: ea88a0484df0608cb08521fb4cde2d4dc7904553
wchargin-branch: rustdoc-server
wchargin-source: f1172908d26a8f6621f0ed0a894ec8541db6c345
wchargin-branch: rustdoc-server
wchargin-source: f1172908d26a8f6621f0ed0a894ec8541db6c345
wchargin-branch: rustdoc-server
wchargin-source: 8817e8fef2309159717320f672701c2fd8999517
@wchargin
Copy link
Contributor Author

wchargin commented Dec 3, 2020

From a high level design perspective, do we want to be generating a server target for every rustdoc target, or just a single binary target that serves as a tool that can handle any rustdoc output?

If you want to be able to ibazel run it and have the server
automatically restart when the Rust code is updated, there needs to be a
build path from the target that you run to your code, so I think that
there needs to be a server target for each rustdoc target. Now, these
could be trivial shims that just delegate to a single core server, but
given that they’re just Python files it seems easier to just let each
one be its own self-contained server. I don’t feel strongly about this.

On the build side, I’m having trouble getting the docs sub-repo to work
with rules_python without crashing the build. So I’m feeling inclined
to abandon this. On TensorBoard, we just have our own version of this
server as a local py_binary, so I’m not personally blocked by this.
If anyone happens to know an easy fix for the build issue, I’d be happy
to incorporate it, or if you want to poke at it yourself, please feel
free to yoink my server code.

@dfreese, does this sound reasonable to you?

@wchargin wchargin closed this Dec 13, 2020
@wchargin wchargin deleted the wchargin-rustdoc-server branch December 13, 2020 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rustdoc: offer web server as a Bazel binary
2 participants