diff --git a/README.md b/README.md index e37d80d03..bca9dc2d6 100644 --- a/README.md +++ b/README.md @@ -9,46 +9,33 @@ of crates for the Rust Programming Language. Docs.rs automatically builds crates' documentation released on crates.io using the nightly release of the Rust compiler. -The README of a crate is taken from the readme field defined in Cargo.toml. -If a crate doesn't have this field, no README will be displayed. +This readme is for developing docs.rs. See [the about page](https://docs.rs/about) for user-facing documentation. -### Redirections +## Changing the build environment -Docs.rs is using semver to parse URLs. You can use this feature to access -crates' documentation easily. Example of URL redirections for `clap` crate: - -| URL | Redirects to documentation of | -|-----------------------------------------------|--------------------------------------------------| -| | Latest version of clap | -| | 2.* version | -| | 2.9.* version | -| | 2.9.3 version (you don't need = unlike semver) | -| | Latest version of this page (if it still exists).| - -### Badges +To make a change to [the build environment](https://github.com/rust-lang/crates-build-env) +and test that it works on docs.rs, see [the wiki](https://forge.rust-lang.org/docs-rs/add-dependencies.html). -You can use badges to show state of your documentation to your users. -The default badge will be pointed at the latest version of a crate. -You can use `version` parameter to show status of documentation for -any version you want. +## Development -Badge will display in blue if docs.rs is successfully hosting your crate -documentation, and red if building documentation failing. +The recommended way to develop docs.rs is a combination of `cargo run` for +the main binary and [docker-compose](https://docs.docker.com/compose/) for the external services. +This gives you reasonable incremental build times without having to add new users and packages to your host machine. -Example badges for `mio` crate: +### Dependencies -| URL | Badge | -|-------|-------| -| Latest version: | ![mio](https://docs.rs/mio/badge.svg) | -| Version 0.4.4: | ![mio](https://docs.rs/mio/badge.svg?version=0.4.4) | -| Version 0.1.0: | ![mio](https://docs.rs/mio/badge.svg?version=0.1.0) | +Docs.rs requires at least the following native C dependencies. +- gcc +- g++ +- pkg-config +- git +- make +- cmake +- zlib +- openssl -## Development - -We strongly recommend using [docker-compose](https://docs.docker.com/compose/), -which will make it easier to get started without adding new users and packages -to your host machine. +There may be other dependencies that have not been documented. ### Getting started @@ -57,35 +44,42 @@ Make sure you have docker-compose and are able to download ~10GB data on the fir ```sh git clone https://github.com/rust-lang/docs.rs.git docs.rs cd docs.rs +# Configure the default settings for external services cp .env.sample .env - -docker-compose build # This builds the docs.rs binary - +# Builds the docs.rs binary +cargo build +# Start the extenal services +docker-compose up -d db s3 # Build a sample crate to make sure it works # This sets up the docs.rs build environment, including installing the nightly # Rust toolchain. This will take a while the first time but will be cached afterwards. -docker-compose run web build crate regex 1.3.1 - +cargo run web build crate regex 1.3.1 # This starts the web server but does not build any crates. -# If you want to build crates, see below under `build` subcommand. -# It should print a link to the website once it finishes initializing. -docker-compose up - +cargo run start-web-server ``` If you need to store big files in the repository's directory it's recommended to put them in the `ignored/` subdirectory, which is ignored both by git and Docker. +### Pure docker-compose + +If you have trouble with the above commands, consider using `docker-compose up`, +which uses docker-compose for the web server as well. +This will not cache dependencies as well - in particular, you'll have to rebuild all 400 whenever the lockfile changes - +but makes sure that you're in a known environment so you should have fewer problems getting started. + +Please file bugs for any trouble you have running docs.rs! + ### Running tests -Tests are run outside of the docker-compose environment, and can be run with: +Tests are only supported via cargo and do not work in docker-compose ``` cargo test ``` -Some tests require access to the database. To run them, set the +Most tests require access to the database. To run them, set the `CRATESFYI_DATABASE_URL` in `.env` to the url of a PostgreSQL database, and set the `AWS_ACCESS_KEY_ID`, `S3_ENDPOINT`, and `AWS_SECRET_ACCESS_KEY` variables. We have some reasonable default parameters in `.env.sample`. @@ -101,7 +95,7 @@ If you don't want to use docker-compose, see the for more information on how to setup this environment. Note that either way, you will need docker installed for sandboxing with Rustwide. -[wiki-no-compose]: https://github.com/rust-lang/docs.rs/wiki/Developing-without-docker-compose +[wiki-no-compose]: https://forge.rust-lang.org/docs-rs/no-docker-compose.html ### Docker-Compose @@ -111,7 +105,7 @@ Three services are defined: | name | access | credentials | description | |------|-------------------------------------------------|----------------------------|----------------------------------------| | web | http://localhost:3000 | N/A | A container running the docs.rs binary | -| db | postgresql://cratesfyi:password@localhost:15432/ | - | Postgres database used by web | +| db | postgresql://cratesfyi:password@localhost:15432 | - | Postgres database used by web | | s3 | http://localhost:9000 | `cratesfyi` - `secret_key` | Minio (simulates AWS S3) used by web | [docker-compose.yml]: ./docker-compose.yml @@ -133,12 +127,13 @@ This is probably because you have `git.autocrlf` set to true, ### CLI -#### Starting web server +See `cargo run -- --help` for a full list of commands. + +#### Starting the web server ```sh -# This command will start web interface of docs.rs and you can access it from -# http://localhost:3000/` -docker-compose up +# This command will start web interface of docs.rs on http://localhost:3000 +cargo run start-web-server ``` #### `build` subcommand @@ -147,16 +142,16 @@ docker-compose up # Builds and adds it into database # This is the main command to build and add a documentation into docs.rs. # For example, `docker-compose run web build crate regex 1.1.6` -docker-compose run web build crate +cargo run web build crate -# Builds every crate and adds them into database +# Builds every crate on crates.io and adds them into database # (beware: this may take months to finish) -docker-compose run web build world +cargo run web build world # Builds a local package you have at and adds it to the database. # The package does not have to be on crates.io. # The package must be on the local filesystem, git urls are not allowed. -docker-compose run -v "$(realpath )":/build web build crate --local /build +cargo build crate --local /path/to/source ``` #### `database` subcommand @@ -176,40 +171,32 @@ If you want to explore or edit database manually, you can connect to the databas with the `psql` command. ```sh -# NOTE: this creates a new container that will keep running after you exit -# To remove the new container, run `docker-compose down`. -docker exec -it "$(docker-compose run -d db && sleep 1)" psql -U cratesfyi -# You only need to start the container once, in the future you can attach to it like so: -docker exec -it "$(docker container ps | grep postgres | cut -d ' ' -f 1)" psql -U cratesfyi +. .env +psql $CRATESFYI_DATABASE_URL ``` The database contains a blacklist of crates that should not be built. ```sh # List the crates on the blacklist -docker-compose run web database blacklist list +cargo run web database blacklist list # Adds to the blacklist -docker-compose run web database blacklist add +cargo run web database blacklist add # Removes from the blacklist -docker-compose run web database blacklist remove +cargo run web database blacklist remove ``` #### `daemon` subcommand ```sh # Run a persistent daemon which queues builds and starts a web server. -# Warning: This will try to queue hundreds of packages on crates.io, only start it -# if you have enough resources! -docker-compose run -p 3000:3000 web daemon +cargo run daemon --registry-watcher disabled +# Add crates to the queue +cargo run queue add ``` -### Changing the build environment - -To make a change to [the build environment](https://github.com/rust-lang/crates-build-env) -and test that it works on docs.rs, see [the wiki](https://github.com/rust-lang/docs.rs/wiki/Making-changes-to-the-build-environment). - ### Contact Docs.rs is run and maintained by the [docs.rs team](https://www.rust-lang.org/governance/teams/dev-tools#docs-rs).