Expand ov documentation and replace environment setup docs with `ov…

…` instructions (#4567)

* Fix broken returns in stdout/err

* Greatly expand built-in `ov` documentation

* Revert "Fix broken returns in stdout/err"

This reverts commit 3210124.

* Do not provision docker tty when in a pipeline

* Update documentation for ov

Do not use a relative path. While it might be generally more consistent, it is much more cumbersome, and would require stating "from the repo root" each time to be accurate

* Fix broken links

* Replace references to old quickstart guide

* Explain blobless clone

* Update documentation/api/guides/quickstart.md

Co-authored-by: zack <[email protected]>

* Reword for clarity

* Fix incorrect or muddied usages of "dependent"

* Update link to non-existent prod subdomain

* Add `ov doctor`

---------

Co-authored-by: Krystle Salazar <[email protected]>
Co-authored-by: zack <[email protected]>

.devcontainer/devcontainer.json

@@ -9,7 +9,6 @@
   // More info: https://containers.dev/features
   // List of features and their default options: https://github.com/devcontainers/features/tree/main/src
   "features": {
-    // Openverse development environment requirement matrix: https://docs.openverse.org/general/general_setup.html#requirement-matrix
     "ghcr.io/guiyomh/features/just:0": {},
     "ghcr.io/devcontainers/features/python:1": {
       // Match the Python version defined in `*/Pipfile`.

.github/ISSUE_TEMPLATE/new_source_suggestion.yml

@@ -71,7 +71,7 @@ body:
         - The script must not use anything from `catalog/dags/retired/providers/provider_api_scripts/modules/etlMods.py`, since that module is deprecated.
         - If the provider API has can be queried by 'upload date' or something similar, the script should take a `--date` parameter when run as a script, giving the date for which we should collect images. The form should be `YYYY-MM-DD` (so, the script can be run via `python my_favorite_provider.py --date 2018-01-01`).
         - The script must provide a main function that takes the same parameters as from the CLI. In our example from above, we'd then have a main function `my_favorite_provider.main(date)`. The main should do the same thing calling from the CLI would do.
-        - The script *must* conform to [PEP8][pep8]. Please use `./ov just lint` before committing.
+        - The script *must* conform to [PEP8][pep8]. Please use `ov just lint` before committing.
         - The script should use small, testable functions.
         - The test suite for the script may break PEP8 rules regarding long lines where appropriate (e.g., long strings for testing).
 

.github/pull_request_template.md

@@ -24,9 +24,9 @@ Fixes #[issue number] by @[issue author]
 - [ ] I added or updated tests for the changes I made (if applicable).
 - [ ] I added or updated documentation (if applicable).
 - [ ] I tried running the project locally and verified that there are no visible errors.
-- [ ] I ran the DAG documentation generator (`./ov just catalog/generate-docs` for catalog
-      PRs) or the media properties generator (`./ov just catalog/generate-docs media-props`
-      for the catalog or `./ov just api/generate-docs` for the API) where applicable.
+- [ ] I ran the DAG documentation generator (`ov just catalog/generate-docs` for catalog
+      PRs) or the media properties generator (`ov just catalog/generate-docs media-props`
+      for the catalog or `ov just api/generate-docs` for the API) where applicable.
 
 [best_practices]:
   https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project#_commit_guidelines

.gitignore

@@ -13,7 +13,7 @@ logs/
 
 # Environment
 .env
-.ov_aliases.json
+.ov_profile.json
 
 # Python environments
 env/

.vale/README.md

@@ -12,7 +12,7 @@ the Openverse monorepo, refer to the README at
 To run Vale with Openverse's configuration, use the just recipe:
 
 ```bash
-./ov just .vale/run
+ov just .vale/run
 ```
 
 This recipe _always_ builds Vale. The Openverse Vale docker image is fast to

api/justfile

@@ -87,11 +87,11 @@ recreate:
 ########
 
 # Make a cURL GET request to service at the given path
-_curl-get path origin="http://localhost:50280":
+@_curl-get path origin="http://localhost:50280":
     curl "{{ origin }}/v1/{{ path }}"
 
 # Make a test cURL GET request to the API
-stats media="images" origin="http://localhost:50280":
+@stats media="images" origin="http://localhost:50280":
     just _curl-get "{{ media }}/stats/" {{ origin }}
 
 # Launch a `pgcli` shell in the web container

catalog/templates/template_test.py_template

@@ -3,7 +3,7 @@ TODO: Add additional tests for any methods you added in your subclass.
 Try to test edge cases (missing keys, different data types returned, Nones, etc).
 You may also need to update the given test names to be more specific.
 
-Run your tests locally with `./ov just catalog/test -k {provider_underscore}`
+Run your tests locally with `ov just catalog/test -k {provider_underscore}`
 """
 
 import json

catalog/tests/dags/providers/provider_api_scripts/test_cc_mixter.py

@@ -1,4 +1,4 @@
-"""Run these tests locally with `./ov just catalog/test -k cc_mixter`"""
+"""Run these tests locally with `ov just catalog/test -k cc_mixter`"""
 
 import json
 from pathlib import Path

catalog/tests/dags/providers/provider_api_scripts/test_inaturalist.py

@@ -37,7 +37,7 @@
 #    to pull a sample of records without breaking referential integrity.
 #  - Putting your final zipped test files in /tests/s3-data/inaturalist-open-data
 #    so that they will be synced over to minio.
-#  - Run `./ov just down -v` and then `./ov just recreate` to make sure that the test data gets to
+#  - Run `ov just down -v` and then `ov just recreate` to make sure that the test data gets to
 #    the test s3 instance. That process may take on the order of 15 minutes for the full
 #    dataset. You'll know that it's done when the s3-load container in docker exits.
 #  - Then, in airflow, trigger the dag possibly with configuration

docker/dev_env/exec.py

@@ -2,31 +2,111 @@
 import os
 import sys
 from pathlib import Path
+from textwrap import dedent
 
 
 SHARED_ALIASES = {
-    "j": ["just"],
-    "nuxt": ["j", "p", "frontend", "dev"],
+    "j": {
+        "cmd": ["just"],
+        "doc": "A common alias for `just` to save three keystrokes (it adds up!)",
+    },
+    "nuxt": {
+        "cmd": ["j", "p", "frontend", "dev"],
+        "doc": "Run the Openverse frontend local development site",
+    },
 }
 
 
+# Document alias usage and configuration here. Including it
+# in the command makes it easy to discover and noticeable when
+# making changes, so it will not go out of sync.
+# ov aliases is included in the root `ov` script to help make this visible
+# and is mentioned in the general setup documentation
+ALIAS_USAGE = """
+ov aliases
+
+USAGE
+    ov ALIAS [...args]
+        Run an aliased command configured for ov.
+
+    ov aliases [--help|-h]
+        List all available aliases. If --help is passed, display this help message instead.
+
+DESCRIPTION
+    `ov init` creates a ./ov_profile.json file at the root of the repository.
+    The "aliases" key of this JSON file should be a dictionary, with alias
+    configuration objects as the keys. The configuration object is a dictionary with
+    a "cmd" key assigned to a list of command arguments to expand the alias to.
+    You may also provide a "doc" key to document the alias. The documentation for
+    each alias is displayed by running `ov aliases`.
+
+    To run an alias, pass it as the command argument to ov. To run the built-in "j"
+    alias, run `ov j`.
+
+    Only the first argument passed to ov will be expanded using aliases. Aliases can
+    stack (that is, repeatedly expand the first argument, if an alias refers to another alias)
+    but only ever the leading argument of the expanded alias. For example, the built-in
+    alias "nuxt" expands to "j p frontend dev", and ov will expand "j" to "just". However,
+    if an alias "p", "frontend", or "dev" existed, ov will not expand those, as they are
+    not the leading argument of the expanded arguments list.
+""".strip()
+
+
+def alias_info(aliases: dict, args: list[str]):
+    if "--help" in args or "-h" in args:
+        print(ALIAS_USAGE)
+        return
+
+    # else, assume --list
+    output = "ov aliases\n"
+    longest_alias = len(sorted(aliases.keys())[-1])
+    for alias, cmd in aliases.items():
+        if isinstance(cmd, dict):
+            doc = cmd["doc"]
+            cmd = cmd["cmd"]
+        else:
+            doc = "No documentation found."
+
+        joined = " ".join(cmd)
+        output += dedent(
+            f"""
+            {alias}{' ' * (longest_alias - len(alias))} -> {joined}
+                {doc}
+                {'(Built-in alias)' if alias in SHARED_ALIASES else '(Personal alias)'}
+            """
+        )
+
+    print(output)
+
+
 def expand_aliases(args: list[str]):
-    ov_aliases = Path(os.getenv("OPENVERSE_PROJECT")) / ".ov_aliases.json"
-    aliases = SHARED_ALIASES
-    if ov_aliases.is_file():
-        aliases |= json.loads(ov_aliases.read_text())
+    ov_profile = Path(os.getenv("OPENVERSE_PROJECT")) / ".ov_profile.json"
+    aliases = SHARED_ALIASES.copy()
+    if ov_profile.is_file():
+        aliases |= json.loads(ov_profile.read_text()).get("aliases", {})
 
     args = sys.argv[1:]
 
+    if args[0] == "aliases":
+        return alias_info(aliases, args)
+
     while args[0] in aliases:
-        args = aliases.pop(args[0]) + args[1:]
+        alias = aliases.pop(args[0])
+        if isinstance(alias, dict):
+            args = alias["cmd"] + args[1:]
+        else:
+            args = alias + args[1:]
 
     return args
 
 
 if __name__ == "__main__":
     args = expand_aliases(sys.argv[1:])
     try:
-        os.execvp(args[0], args)
+        if args is not None:
+            os.execvp(args[0], args)
     except FileNotFoundError:
-        sys.stdout.buffer.write(args[0].encode() + b": command not found\n")
+        print(f"{args[0]}: command not found")
+        exit(1)
+    else:
+        exit(0)

docker/dev_env/run.sh

@@ -26,6 +26,8 @@ shared_args=(
 run_args=(
   -d
   --name "$container_name"
+  # Use host network so things like Sphinx and the frontend that run directly in `ov`
+  # will have ports available on the host (and this way we don't have to manually map each one)
   --network host
   # Bind the repo to the same exact location inside the container so that pre-commit
   # and others don't get confused about where files are supposed to be
@@ -44,7 +46,10 @@ run_args=(
 # environment doesn't have one to attach.
 # In other words, only tell Docker to attach a TTY to the container when
 # there's one to attach in the first place.
-if [ -t 0 ]; then
+# Additionally, we only want to attach a TTY if the command is actually interactive
+# and not, instead, as part of a pipe.
+# -t 1 is false if 1 is not stdout (it's something else! a file, a pipe, etc)
+if [[ -t 0 && -t 1 ]]; then
   shared_args+=(-t)
 fi
 

docker/minio/load_to_s3_entrypoint.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 # This does *not* allow for testing permissions issues that may come up in real AWS.
-# And, if you remove files from /tests/s3-data, you will need to use `./ov just down -v`
-# and `./ov just up` or `./ov just recreate` to see the minio bucket without those files.
+# And, if you remove files from /tests/s3-data, you will need to use `ov just down -v`
+# and `ov just up` or `ov just recreate` to see the minio bucket without those files.
 # Loop through subdirectories mounted to the volume and load them to s3/minio.
 # This takes care of filesystem delays on some local dev environments that may make
 # minio miss files included directly in the minio volume.

documentation/api/guides/https.md

@@ -10,26 +10,34 @@ Make sure you have gone through the
 [quickstart guide](/api/guides/quickstart.md) before attempting this as this is
 a slightly more advanced process.
 
-Additionally, you will need to install
-[mkcert](/general/general_setup.md#mkcert) as described in the
-[general setup guide](/general/general_setup.md).
+Additionally, you will need to install `mkcert`.
+[Follow `mkcert`'s installation guide](https://github.com/FiloSottile/mkcert?tab=readme-ov-file#installation)
+to do so.
+
+```{caution}
+`ov` does not yet support mkcert. You must run this command without `ov`
+on your host system for it to work.
+```
 
 ## Steps
 
 1. Create certificates for NGINX to use.
 
+   ```{caution}
+   Run this on your host system, `ov` does not support mkcert.
+   ```
+
    ```bash
-   ./ov just docker/nginx/cert
+   just docker/nginx/cert
    ```
 
    This will create a certificate file `openversse.crt` and a key file
    `openverse.key` in the `docker/nginx/certs/` directory.
 
-2. Bring the ingestion server and API up, along with all their dependent
-   services.
+2. Start the API along with its dependencies:
 
    ```bash
-   ./ov just api/up
+   ov just api/up
    ```
 
    The `api/up` recipe orchestrates the following services: `cache`, `db`,
@@ -41,8 +49,7 @@ Additionally, you will need to install
 3. Make an API call over HTTPS.
 
    ```bash
-   ./ov just api/stats images https://localhost:50243
-   ./ov just _curl-get "images/stats/" https://localhost:50243
-   curl "https://localhost:50243/v1/images/stats/"
+   ov just api/stats images https://localhost:50243
+   ov just _curl-get "images/stats/" https://localhost:50243
    [{"source_name":"flickr","display_name":"Flickr","source_url":"https://www.flickr.com","logo_url":null,"media_count":2500},{"source_name":"stocksnap","display_name":"StockSnap","source_url":"https://stocksnap.io","logo_url":null,"media_count":2500}]%
    ```