Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README #942

Merged
merged 2 commits into from
Aug 7, 2020
Merged

Update README #942

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
abf1e9d
Update README
jyn514 Aug 6, 2020
b4faa1e
Improve wording
jyn514 Aug 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 56 additions & 69 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,46 +9,33 @@ of crates for the Rust Programming Language.
Docs.rs automatically builds crates' documentation released on crates.io using
the nightly release of the Rust compiler.

The README of a crate is taken from the readme field defined in Cargo.toml.
If a crate doesn't have this field, no README will be displayed.
This readme is for developing docs.rs. See [the about page](https://docs.rs/about) for user-facing documentation.

### Redirections
## Changing the build environment

Docs.rs is using semver to parse URLs. You can use this feature to access
crates' documentation easily. Example of URL redirections for `clap` crate:

| URL | Redirects to documentation of |
|-----------------------------------------------|--------------------------------------------------|
| <https://docs.rs/clap> | Latest version of clap |
| <https://docs.rs/clap/~2> | 2.* version |
| <https://docs.rs/clap/~2.9> | 2.9.* version |
| <https://docs.rs/clap/2.9.3> | 2.9.3 version (you don't need = unlike semver) |
| <https://docs.rs/clap/*/clap/struct.App.html> | Latest version of this page (if it still exists).|

### Badges
To make a change to [the build environment](https://github.com/rust-lang/crates-build-env)
and test that it works on docs.rs, see [the wiki](https://forge.rust-lang.org/docs-rs/add-dependencies.html).

You can use badges to show state of your documentation to your users.
The default badge will be pointed at the latest version of a crate.
You can use `version` parameter to show status of documentation for
any version you want.
## Development

Badge will display in blue if docs.rs is successfully hosting your crate
documentation, and red if building documentation failing.
The recommended way to develop docs.rs is a combination of `cargo run` for
the main binary and [docker-compose](https://docs.docker.com/compose/) for the external services.
This gives you reasonable incremental build times without having to add new users and packages to your host machine.

Example badges for `mio` crate:
### Dependencies

| URL | Badge |
|-------|-------|
| Latest version: <https://docs.rs/mio/badge.svg> | ![mio](https://docs.rs/mio/badge.svg) |
| Version 0.4.4: <https://docs.rs/mio/badge.svg?version=0.4.4> | ![mio](https://docs.rs/mio/badge.svg?version=0.4.4) |
| Version 0.1.0: <https://docs.rs/mio/badge.svg?version=0.1.0> | ![mio](https://docs.rs/mio/badge.svg?version=0.1.0) |
Docs.rs requires at least the following native C dependencies.

- gcc
- g++
- pkg-config
- git
- make
- cmake
- zlib
- openssl

## Development

We strongly recommend using [docker-compose](https://docs.docker.com/compose/),
which will make it easier to get started without adding new users and packages
to your host machine.
There may be other dependencies that have not been documented.

### Getting started

Expand All @@ -57,35 +44,42 @@ Make sure you have docker-compose and are able to download ~10GB data on the fir
```sh
git clone https://github.com/rust-lang/docs.rs.git docs.rs
cd docs.rs
# Configure the default settings for external services
cp .env.sample .env

docker-compose build # This builds the docs.rs binary

# Builds the docs.rs binary
cargo build
# Start the extenal services
docker-compose up -d db s3
# Build a sample crate to make sure it works
# This sets up the docs.rs build environment, including installing the nightly
# Rust toolchain. This will take a while the first time but will be cached afterwards.
docker-compose run web build crate regex 1.3.1

cargo run web build crate regex 1.3.1
# This starts the web server but does not build any crates.
# If you want to build crates, see below under `build` subcommand.
# It should print a link to the website once it finishes initializing.
docker-compose up

cargo run start-web-server
```

If you need to store big files in the repository's directory it's recommended to
put them in the `ignored/` subdirectory, which is ignored both by git and
Docker.

### Pure docker-compose

If you have trouble with the above commands, consider using `docker-compose up`,
which uses docker-compose for the web server as well.
This will not cache dependencies as well - in particular, you'll have to rebuild all 400 whenever the lockfile changes -
but makes sure that you're in a known environment so you should have fewer problems getting started.

Please file bugs for any trouble you have running docs.rs!

### Running tests

Tests are run outside of the docker-compose environment, and can be run with:
Tests are only supported via cargo, not docker-compose.
jyn514 marked this conversation as resolved.
Show resolved Hide resolved

```
cargo test
```

Some tests require access to the database. To run them, set the
Most tests require access to the database. To run them, set the
`CRATESFYI_DATABASE_URL` in `.env` to the url of a PostgreSQL database,
and set the `AWS_ACCESS_KEY_ID`, `S3_ENDPOINT`, and `AWS_SECRET_ACCESS_KEY` variables.
We have some reasonable default parameters in `.env.sample`.
Expand All @@ -101,7 +95,7 @@ If you don't want to use docker-compose, see the
for more information on how to setup this environment.
Note that either way, you will need docker installed for sandboxing with Rustwide.

[wiki-no-compose]: https://github.com/rust-lang/docs.rs/wiki/Developing-without-docker-compose
[wiki-no-compose]: https://forge.rust-lang.org/docs-rs/no-docker-compose.html

### Docker-Compose

Expand All @@ -111,7 +105,7 @@ Three services are defined:
| name | access | credentials | description |
|------|-------------------------------------------------|----------------------------|----------------------------------------|
| web | http://localhost:3000 | N/A | A container running the docs.rs binary |
| db | postgresql://cratesfyi:password@localhost:15432/ | - | Postgres database used by web |
| db | postgresql://cratesfyi:password@localhost:15432 | - | Postgres database used by web |
| s3 | http://localhost:9000 | `cratesfyi` - `secret_key` | Minio (simulates AWS S3) used by web |

[docker-compose.yml]: ./docker-compose.yml
Expand All @@ -133,12 +127,13 @@ This is probably because you have `git.autocrlf` set to true,

### CLI

#### Starting web server
See `cargo run -- --help` for a full list of commands.

#### Starting the web server

```sh
# This command will start web interface of docs.rs and you can access it from
# http://localhost:3000/`
docker-compose up
# This command will start web interface of docs.rs on http://localhost:3000
cargo run start-web-server
```

#### `build` subcommand
Expand All @@ -147,16 +142,16 @@ docker-compose up
# Builds <CRATE_NAME> <CRATE_VERSION> and adds it into database
# This is the main command to build and add a documentation into docs.rs.
# For example, `docker-compose run web build crate regex 1.1.6`
docker-compose run web build crate <CRATE_NAME> <CRATE_VERSION>
cargo run web build crate <CRATE_NAME> <CRATE_VERSION>

# Builds every crate and adds them into database
# Builds every crate on crates.io and adds them into database
# (beware: this may take months to finish)
docker-compose run web build world
cargo run web build world

# Builds a local package you have at <SOURCE> and adds it to the database.
# The package does not have to be on crates.io.
# The package must be on the local filesystem, git urls are not allowed.
docker-compose run -v "$(realpath <SOURCE>)":/build web build crate --local /build
cargo build crate --local /path/to/source
```

#### `database` subcommand
Expand All @@ -176,40 +171,32 @@ If you want to explore or edit database manually, you can connect to the databas
with the `psql` command.

```sh
# NOTE: this creates a new container that will keep running after you exit
# To remove the new container, run `docker-compose down`.
docker exec -it "$(docker-compose run -d db && sleep 1)" psql -U cratesfyi
# You only need to start the container once, in the future you can attach to it like so:
docker exec -it "$(docker container ps | grep postgres | cut -d ' ' -f 1)" psql -U cratesfyi
. .env
psql $CRATESFYI_DATABASE_URL
```

The database contains a blacklist of crates that should not be built.

```sh
# List the crates on the blacklist
docker-compose run web database blacklist list
cargo run web database blacklist list

# Adds <CRATE_NAME> to the blacklist
docker-compose run web database blacklist add <CRATE_NAME>
cargo run web database blacklist add <CRATE_NAME>

# Removes <CRATE_NAME> from the blacklist
docker-compose run web database blacklist remove <CRATE_NAME>
cargo run web database blacklist remove <CRATE_NAME>
```

#### `daemon` subcommand

```sh
# Run a persistent daemon which queues builds and starts a web server.
# Warning: This will try to queue hundreds of packages on crates.io, only start it
# if you have enough resources!
docker-compose run -p 3000:3000 web daemon
cargo run daemon --registry-watcher disabled
# Add crates to the queue
cargo run queue add <CRATE> <VERSION>
```

### Changing the build environment

To make a change to [the build environment](https://github.com/rust-lang/crates-build-env)
and test that it works on docs.rs, see [the wiki](https://github.com/rust-lang/docs.rs/wiki/Making-changes-to-the-build-environment).

### Contact

Docs.rs is run and maintained by the [docs.rs team](https://www.rust-lang.org/governance/teams/dev-tools#docs-rs).
Expand Down