Build: implementation of build.commands

[humitos]

Initial implementation of build.commands as a beta feature to expose to user and collect more use cases.

Build without using any tool

version: 2

build:
  os: ubuntu-20.04
  commands:
    - mkdir output/
    - echo "Hello world! `date`" > output/index.html

  tools:  # <--- this shouldn't be required for these cases where no tools are needed for the build
    python: "3"

(https://github.com/readthedocs/test-builds/blob/build-commands/.readthedocs.yaml)

Build details page

index.html file served by El Proxito

Build using pelican

version: 2

build:
  os: ubuntu-22.04
  tools:
    python: "3.10"
  commands:
    - pip install pelican[markdown]
    # After installing Python packages with binaries on it, we need to call `asdf reshim`
    # (recreate shims for version of a package)
    # https://github.com/danhper/asdf-python/issues/37
    - asdf reshim
    - pelican --settings docs/pelicanconf.py --output output/ docs/

(https://github.com/readthedocs/test-builds/blob/pelican/.readthedocs.yaml)

Build details page

index.html file served by El Proxito

---

Decisions / Notes

• Expose the feature as beta, communicating this in the documentation and also on the build details page when we detect build.commands on the config file
• It defines an "implicit contract" where the output/ folder under the repository checkout will be uploaded to S3
• No HTML integrations will be done (no flyout, no ads, no search)
• No extra formats like ePUB, PDF and HTMLZip will be available
• Without the flyout integration, this is probably more useful for single-version projects
• build.os and build.tools are mandatory when using build.commands
• We should probably make build.tools not required since it's possible to generate HTML without any of these languages as shown in the first example where only echo command is used
• asdf requires re-shim the binaries after installing new CLI tools with Python to be able to find them when executing any of these commands. There was an attempt to re-shim automatically at Automatically run reshim? asdf-community/asdf-python#37 but they removed the post_<tool>_<shim> because Revert to using exec when running a shim asdf-vm/asdf#502. This may need some extra research to make the UX better. However, for now, I'm adding asdf reshim to my build.commands. We could execute asdf reshim python after each of the build.commands, for now as well
• I didn't update the "Build customization" page, but I'm happy to add a small example there (https://docs.readthedocs.io/en/stable/build-customization.html) if we consider it's good at this point. However, it may require a good amount of re-writing on that page to make the distinction between the two approaches clear

Related #9062 and others linked into that issue.

[ericholscher]

Neat how little code this is. Definitely needs a lot more done around the edges to make it nicer, but great that we already have all the basic infrastructure to make it work. 👍

[humitos]

I'm asking review for the three of you @ericholscher, @agjohnson, and @stsewd because I think this is an important change that will mold our path moving forward regarding the UX.

[ericholscher]

This looks close to me. I think we're close enough to get it out next week for testing. There's lots of additional work to be done in docs, etc. -- but this is enough for now.

[ericholscher]

This still feels really small compared to how important the information is, but we can improve it over time.

[humitos]

I agree. This has to be a lot more prominent. However,

1. I wasn't sure how to make it fit here in a good way
2. I'm not sure this is the right place for explaining this more. From the config file, we should point users to the "Build customization" page and explain all the things in detail there. Otherwise, we will be overloading the config file reference

I'm thinking we may need a section in the "Build customization" page that list all the considerations required to use build.commands or something like that we can link from here. I feel I'm not coming with a good idea of how to write this, tho 🤷🏼

[humitos]

I updated the PR with all the suggestions that I received and improved the code a little bit. Now, it handles reshiming automatically when it detects a Python package manager installing something and it also uses a 'generic' doctype for this type of documentation (build.commands).

For now, we are using template-based notifications but we plan to change that because none of us like this pattern. However, it's good enough, for now, to move forward and don't block this feature on that.

build.tools is required for now, but I opened an issue to track that and remove that restriction. I'm not doing it in this PR because it involves a good refactor of the ConfigFile object due that it differentiates legacy from the newer pattern based on build.tools instead of build.os.

I'm happy to merge this during this week and deploy it the next one 👍🏼

[agjohnson]

This is looking great. 👍 from me with the last bits of new feedback resolved