[documentation] undocumented environment variables

[majiang]

problem

Some environment variables to configure tox are undocumented as far as I searched on readthedocs. Some examples are:

• TOX_DISCOVER, which is in changelog 3.14.5 but not elsewhere. I searched in this repository for an environment-variable alternative to --discover command line option to find this environment variable.
• TOX_PROVISION

and others can be found by searching.

possible improvement

1. There is an Injected environment variables section in tox configuration specification. I found it is most straightforward to put them into this section. Since they don't seem to be environment variables "that you can use to test that your command is running within tox", the paragraph can also be rewritten.
2. Since TOX_DISCOVER corresponds to --discover, it is also possible to add it to help argument of parser.add_argument calls`. I haven't check if this applies to all the undocumented environment variables.

[majiang]

Which of the above possible improvement (or another) seems better?

[majiang]

If there is an easy way to extract all environment variables {used, documented} I could help drafting a PR.

[jugmac00]

Hi @majiang,

thank you for considering to contribute by updating the documentation! This is greatly appreciated!!

As for how to extract all environment variables... afaik all env names contain TOX, so this regex should be a good first approach:

❯ rg -o "_{0,1}TOX[A-Z_]*"
tasks/notify.py
122:TOX_DEV_GOOGLE_SECRET

tests/integration/test_provision_int.py
79:TOX_REPORTER_TIMESTAMP

tests/integration/test_parallel_interrupt.py
27:_TOX_SKIP_ENV_CREATION_TEST
28:TOX_REPORTER_TIMESTAMP

tests/unit/session/test_session.py
135:TOX_SKIP_ENV
155:TOX_SKIP_ENV
167:TOX_SKIP_ENV
285:TOX_WORK_DIR
287:TOX_ENV_NAME
288:TOX_ENV_DIR

tests/unit/session/test_parallel.py
89:_TOX_SKIP_ENV_CREATION_TEST
125:_TOX_SKIP_ENV_CREATION_TEST
144:_TOX_SKIP_ENV_CREATION_TEST
173:_TOX_SKIP_ENV_CREATION_TEST
218:TOX_PARALLEL_NO_SPINNER
225:TOX_PARALLEL_NO_SPINNER
232:TOX_PARALLEL_NO_SPINNER
292:TOX_WORK_DIR
293:TOX_WORK_DIR

tests/unit/package/test_package_parallel.py
128:TOX_WORK_DIR
129:TOX_WORK_DIR

tests/unit/config/test_config.py
1362:TOX_ENV_NAME
1363:TOX_ENV_DIR
1609:TOX_TESTENV_PASSENV
1623:TOX_TESTENV_PASSENV
1634:TOX_PARALLEL_NO_SPINNER
1638:TOX_PARALLEL_NO_SPINNER
1642:TOX_PARALLEL_NO_SPINNER
2672:TOXENV
2675:TOXENV
2890:TOX_ENV_DIR
2890:TOX_ENV_NAME
3172:TOX_ENV_DIR
3172:TOX_ENV_NAME

tests/unit/package/test_package.py
165:TOX_PACKAGE
170:TOX_PACKAGE

tests/unit/test_docs.py
22:TOX_ROOT
23:TOX_ROOT

azure-pipelines.yml
62:TOXENV

HOWTORELEASE.rst
52:TOX_DEV_GOOGLE_SECRET
54:TOX_DEV_GOOGLE_SECRET

docs/config.rst
122:TOXENV
128:TOX_SKIP_ENV
460:TOX_TESTENV_PASSENV
1082:TOX_LIMITED_SHEBANG
1084:TOX_LIMITED_SHEBANG
1095:TOX_WORK_DIR
1096:TOX_ENV_NAME
1097:TOX_ENV_DIR
1098:TOX_PACKAGE
1099:TOX_PARALLEL_ENV

src/tox/_pytestplugin.py
30:TOXENV
31:TOXENV
73:TOX_WORK_DIR

tests/unit/session/test_list_env.py
2:TOXENV
30:TOXENV
34:TOXENV
98:TOXENV
102:TOXENV

src/tox/config/__init__.py
58:TOX_PROVISION
449:TOXENV
619:TOX_ENV_NAME
620:TOX_ENV_DIR
776:TOX_WORK_DIR
785:TOX_TESTENV_PASSENV
1492:TOXENV

docs/changelog.rst
375:TOX_DISCOVER
455:_TOX_PARALLEL_ENV
456:TOX_PARALLEL_ENV
568:TOX_WORK_DIR
591:TOX_PARALLEL_NO_SPINNER
618:TOXENV
631:TOXENV
803:TOX_PARALLEL_ENV
812:TOX_REPORTER_TIMESTAMP
859:TOX_PACKAGE
986:TOX_SKIP_ENV
988:TOX_WORK_DIR
989:TOX_ENV_NAME
989:TOX_ENV_DIR
1014:TOX_LIMITED_SHEBANG
1128:TOX_LIMITED_SHEBANG
1129:TOX_LIMITED_SHEBANG
1415:TOX_TESTENV_PASSENV
1619:TOX_TESTENV_PASSENV

src/tox/session/__init__.py
67:TOX_WORK_DIR
145:TOX_SKIP_ENV
216:TOX_PACKAGE

src/tox/exception.py
47:TOX_MISSING_SUBSTITUTION

src/tox/session/commands/provision.py
15:TOX_PROVISION

src/tox/session/commands/help.py
7:TOXENV
8:TOX_SKIP_ENV
10:TOX_TESTENV_PASSENV
14:TOX_PARALLEL_NO_SPINNER

src/tox/reporter.py
20:TOX_REPORTER_TIMESTAMP

src/tox/session/commands/run/parallel.py
17:TOX_PARALLEL_NO_SPINNER

src/tox/interpreters/common.py
14:TOX_DISCOVER

src/tox/config/parallel.py
5:TOX_PARALLEL_ENV
6:_TOX_PARALLEL_ENV

src/tox/venv.py
592:TOX_LIMITED_SHEBANG
696:_TOX_SKIP_ENV_CREATION_TEST

docs/example/basic.rst
411:TOX_PARALLEL_NO_SPINNER

docs/example/general.rst
82:TOXENV
114:TOXENV
118:TOXENV

docs/example/jenkins.rst
18:TOXENV
18:TOXENV
29:TOXENV
64:TOXDEV
72:TOXDEV
186:TOX_LIMITED_SHEBANG

I am a bit in a hurry, but maybe you could massage the result, so you only get all env names in the py files, all env names in the rst files, and then compare them - or you do a manual comparison, or maybe @asottile got a minute - he got the black belt with regexes.

A good first step would be to find out which env variables are undocumented, maybe then it is also easier to decide where they should go to in the documentation.

Thanks again!

[majiang]

Thanks for extracting environment variables. There are more than I expected..

Is it better to wrap os.environ.get to automatically generate the list like command line options? I haven't used sphinx though.

[jugmac00]

sphinx is a mythical creature for me, both the app and the statue :-)

Unless @gaborbernat can help here, I think it is ok to manually list the env variables, and make sure that we do not add new env variables in future without documenting them.

[gaborbernat]

There's no way to automatically discover these env-vars, it must be manually maintained sadly.

[majiang]

Indeed there's no way, now. I meant by "wrap os.environ.get" something like:

class EnvVars:
    def __init__(self, all_env_vars: dict):
        self.all_env_vars = all_env_vars
    def __getitem__(self, name):
        env_var = self.all_env_vars[name] # which raises KeyError with unknown environment variable
        return os.environ.get(name, env_var.default)
    def document(self):
        return '\n'.join(f'`{name}`={env_var.default}\t{env_var.description}'
            for (name, env_var) in self.all_env_vars)

and use env_vars instance instead of where os.environ.get is now used. If there's some configuration sphinx can include the result of env_vars.document().

Description strings themselves need to be manually added anyway.

[gaborbernat]

Yeah, I'd go with just doing it manually 👍🏻

[jugmac00]

I played a bit with the regex pattern, and this does look a bit better. Note, the TOX_ is a false positive, but the others look good.

❯ diff <(egrep -oh -r --include \*.py "_{0,1}TOX_[A-Z_]*" | sort | uniq) <(egrep -oh -r --include \*.rst --exclude changelog.rst "_{0,1}TOX_[A-Z_]*" | sort | uniq)
1,3d0
< TOX_
< TOX_CONFIG_FILE
< TOX_CONFIG_FILE_ENV_VAR
5d1
< TOX_DISCOVER
9d4
< TOX_MISSING_SUBSTITUTION
11,12d5
< TOX_PACKAGE_ENV_ID
< _TOX_PARALLEL_ENV
15,18d7
< TOX_PROVISION
< TOX_REPORTER_TIMESTAMP
< TOX_ROOT
< _TOX_SHOW_THREAD
20d8
< _TOX_SKIP_ENV_CREATION_TEST

P.S.: @eumiro correctly noted that {0,1} could be replaced with ? 👍

P.P.S.: There are some more false positives... as we also use TOX_xxx as constants.

[gaborbernat]

Released via https://pypi.org/project/tox/3.24.0/