Add make docs and doc building instructions (#9534)

* Update doc building instructions and pin dependencies

This pins the dependencies for the docs and adds `pytest` as a
dependency which was missing when I built. Tested out the
requirements.txt with a fresh `ubuntu:focal` Docker image to verify that
the required depedencies work.

* Address comments, add Makefile for docs and add to instructions

* Use Python for running scripts

* Fix lint, add lint command

* Add option for cpu to only run the precheck, address comments

* Fix 'make doc' usage, add some -x's

* Fix bad condition on --cpu, add defaults for envs

* Fix another 'make doc'

* Fix for running on MacOS

Co-authored-by: driazati <[email protected]>

.gitignore

@@ -242,3 +242,8 @@ conda/pkg
 .tvm_test_data
 .dgl
 .caffe2
+
+# Local docs build
+_docs/
+jvm/target
+.config/configstore/

Makefile

@@ -19,7 +19,7 @@
 .PHONY: all \
         runtime vta cpptest crttest \
         lint pylint cpplint scalalint \
-	doc \
+	cppdoc docs \
 	web webclean \
 	cython cython3 cyclean \
         clean
@@ -114,7 +114,7 @@ scalalint:
 mypy:
 	tests/scripts/task_mypy.sh
 
-doc:
+cppdoc:
 	doxygen docs/Doxyfile
 
 
@@ -175,3 +175,6 @@ jvminstall:
 
 # Final cleanup rules, delegate to more specific rules.
 clean: cmake_clean cyclean webclean
+
+docs:
+	python3 tests/scripts/ci.py docs

docker/bash.sh

@@ -85,6 +85,10 @@ Usage: docker/bash.sh [-i|--interactive] [--net=host] [-t|--tty]
 
     Print the docker command to be run, but do not execute it.
 
+--env
+
+    Pass an environment variable through to the container.
+
 --name
 
     Set the name of the docker container, and the hostname that will
@@ -149,6 +153,7 @@ function parse_error() {
 # to overwrite the parent scope's behavior.
 break_joined_flag='if (( ${#1} == 2 )); then shift; else set -- -"${1#-i}" "${@:2}"; fi'
 
+DOCKER_ENV=( )
 
 while (( $# )); do
     case "$1" in
@@ -195,6 +200,11 @@ while (( $# )); do
             fi
             ;;
 
+        --env)
+            DOCKER_ENV+=( --env "$2" )
+            shift 2
+            ;;
+
         --dry-run)
             DRY_RUN=true
             shift
@@ -263,7 +273,6 @@ fi
 source "$(dirname $0)/dev_common.sh" || exit 2
 
 DOCKER_FLAGS=( )
-DOCKER_ENV=( )
 DOCKER_MOUNT=( )
 DOCKER_DEVICES=( )
 

docs/README.md

@@ -0,0 +1,101 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+# TVM Documentation
+This folder contains the source of TVM's documentation, hosted at https://tvm.apache.org/docs
+
+## Build Locally
+
+See also the instructions below to run a specific tutorial. Note that some of the tutorials need GPU support. Once build, either of these can be served using Python's build in HTTP server:
+
+```bash
+# Run this and then visit http://localhost:8000 in your browser
+cd docs/_build/html && python3 -m http.server
+```
+
+### With Docker (recommended)
+
+1. Build TVM and the docs inside the [tlcpack/ci-gpu image](https://hub.docker.com/r/tlcpack/ci-gpu)
+
+    ```bash
+    # If this runs into errors, try cleaning your 'build' directory
+    make docs
+    ```
+
+
+### Native
+
+1. [Build TVM](https://tvm.apache.org/docs/install/from_source.html) first in the repo root folder
+2. Install dependencies
+
+    ```bash
+    # Pillow on Ubuntu may require libjpeg-dev from apt
+    docker run tlcpack/ci-gpu:v0.78 bash -c \
+        'python3 -m pip install --quiet tlcpack-sphinx-addon==0.2.1 synr==0.5.0 && python3 -m pip freeze' > frozen-requirements.txt
+
+    pip install -r frozen-requirements.txt
+    ```
+
+3. Generate the docs
+
+    ```bash
+    # TVM_TUTORIAL_EXEC_PATTERN=none skips the tutorial execution to the build
+    # work on most environments (e.g. MacOS).
+    export TVM_TUTORIAL_EXEC_PATTERN=none
+
+    make html
+    ```
+
+
+## Only Execute Specified Tutorials
+The document build process will execute all the tutorials in the sphinx gallery.
+This will cause failure in some cases when certain machines do not have necessary
+environment. You can set `TVM_TUTORIAL_EXEC_PATTERN` to only execute
+the path that matches the regular expression pattern.
+
+For example, to only build tutorials under `/vta/tutorials`, run
+
+```bash
+TVM_TUTORIAL_EXEC_PATTERN=/vta/tutorials make html
+```
+
+To only build one specific file, do
+
+```bash
+# The slash \ is used to get . in regular expression
+TVM_TUTORIAL_EXEC_PATTERN=file_name\.py make html
+```
+
+## Helper Scripts
+
+You can run the following script to reproduce the CI sphinx pre-check stage.
+This script skips the tutorial executions and is useful to quickly check the content.
+
+```bash
+./tests/scripts/task_sphinx_precheck.sh
+```
+
+The following script runs the full build which includes tutorial executions.
+You will need a GPU CI environment.
+
+```bash
+python tests/scripts/ci.py --precheck --full
+```
+
+## Define the Order of Tutorials
+You can define the order of tutorials with `conf.py::subsection_order` and `conf.py::within_subsection_order`.
+By default, the tutorials within one subsection is sorted by filename.

python/tvm/script/parser.py

@@ -155,7 +155,9 @@ class TVMScriptParser(Transformer):
         ast.BuiltinOp.Not: tvm.tir.Not,
     }
 
-    def __init__(self, base_lineno, tir_namespace):
+    # pylint gets confused here with synr.Transformer which doesn't have a
+    # custom init, so just disable it
+    def __init__(self, base_lineno, tir_namespace):  # pylint: disable=super-init-not-called
         self.context = None
 
         self.base_lineno = base_lineno

tests/lint/cppdocs.sh

@@ -16,13 +16,14 @@
 # specific language governing permissions and limitations
 # under the License.
 
+set -euxo pipefail
 
 function cleanup() {
     rm -f /tmp/$$.log.txt /tmp/$$.logclean.txt
 }
 trap cleanup EXIT
 
-make doc 2>/tmp/$$.log.txt
+make cppdoc 2>/tmp/$$.log.txt
 
 grep -v -E "ENABLE_PREPROCESSING|unsupported tag" < /tmp/$$.log.txt > /tmp/$$.logclean.txt || true
 echo "---------Error Log----------"

tests/lint/pylint.sh

@@ -15,8 +15,8 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+set -euxo pipefail
 
-
-python3 -m pylint python/tvm --rcfile=$(dirname "$0")/pylintrc
-python3 -m pylint vta/python/vta --rcfile=$(dirname "$0")/pylintrc
-python3 -m pylint tests/python/unittest/test_tvmscript_type.py --rcfile=$(dirname "$0")/pylintrc
+python3 -m pylint python/tvm --rcfile="$(dirname "$0")"/pylintrc
+python3 -m pylint vta/python/vta --rcfile="$(dirname "$0")"/pylintrc
+python3 -m pylint tests/python/unittest/test_tvmscript_type.py --rcfile="$(dirname "$0")"/pylintrc