pip install --help fails

[Rosswell]

Environment

• pip version: 20.3
• Python version: 3.7.9 & 2.7.16
• OS: macOS Catalina 10.15.7

Description
pip install --help fails to parse the cli args, resulting in it raising with an AttributeError, for both py2.7 and py3.7

Expected behavior
normal help output for the install command, as in pip 20.2.4:

$ pip install --help

Usage:
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:

  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.

  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  -r, --requirement <file>    Install from the given requirements file. This option can be used multiple times.
{...}

How to Reproduce

1. set up and activate fresh virtual env
   $ python -m virtualenv ve && . ve/bin/activate
2. install old pip
   $ pip install pip==20.2.4
3. get expected help output
   $ pip install --help
4. install new pip
   $ pip install pip==20.3
5. try again, should fail
   $ pip install --help

Output

for python 3.7.9, pip install --help fails with the following error:

$ pip install --help
Traceback (most recent call last):
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/bin/pip", line 8, in <module>
    sys.exit(main())
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/main.py", line 75, in main
    return command.main(cmd_args)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 117, in main
    return self._main(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 131, in _main
    options, args = self.parse_args(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 111, in parse_args
    return self.parser.parse_args(args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1427, in _process_args
    self._process_long_opt(rargs, values)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1501, in _process_long_opt
    option.process(opt, value, values, self)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 785, in process
    self.action, self.dest, opt, value, values, parser)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 807, in take_action
    parser.print_help()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1647, in print_help
    file.write(self.format_help())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1635, in format_help
    result.append(self.format_option_help(formatter))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1618, in format_option_help
    result.append(group.format_help(formatter))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1101, in format_help
    result += OptionContainer.format_help(self, formatter)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1072, in format_help
    result.append(self.format_option_help(formatter))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1061, in format_option_help
    result.append(formatter.format_option(option))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 322, in format_option
    help_text = self.expand_default(option)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/parser.py", line 123, in expand_default
    default_value, redact_auth_from_url(default_value))
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 826, in redact_auth_from_url
    return _transform_url(url, _redact_netloc)[0]
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 786, in _transform_url
    purl = urllib_parse.urlsplit(url)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 418, in urlsplit
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 123, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'list' object has no attribute 'decode'

using python 2.7.16 produces a similar, but slightly different error:

$ pip install --help
Traceback (most recent call last):
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/bin/pip", line 8, in <module>
    sys.exit(main())
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 75, in main
    return command.main(cmd_args)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 117, in main
    return self._main(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 131, in _main
    options, args = self.parse_args(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 111, in parse_args
    return self.parser.parse_args(args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1400, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1440, in _process_args
    self._process_long_opt(rargs, values)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1515, in _process_long_opt
    option.process(opt, value, values, self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 789, in process
    self.action, self.dest, opt, value, values, parser)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 811, in take_action
    parser.print_help()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1650, in format_help
    result.append(self.format_option_help(formatter))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1633, in format_option_help
    result.append(group.format_help(formatter))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1114, in format_help
    result += OptionContainer.format_help(self, formatter)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1085, in format_help
    result.append(self.format_option_help(formatter))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1074, in format_option_help
    result.append(formatter.format_option(option))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 316, in format_option
    help_text = self.expand_default(option)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/parser.py", line 123, in expand_default
    default_value, redact_auth_from_url(default_value))
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 826, in redact_auth_from_url
    return _transform_url(url, _redact_netloc)[0]
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 786, in _transform_url
    purl = urllib_parse.urlsplit(url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urlparse.py", line 176, in urlsplit
    cached = _parse_cache.get(key, None)
TypeError: unhashable type: 'list'

[pradyunsg]

HUH. This is definitely worth fixing. ;)

Thanks for the bug report @Rosswell! Would it be possible to provide the output of pip config list?

[Rosswell]

@pradyunsg sure thing:

global.extra-index-url='https://pypi.python.org/simple/'
global.index-url='https://nexus.it.locusdev.net/repository/invitae-pypi/simple/'
global.no-cache-dir='true'
global.retries='1'
global.timeout='2'
global.trusted-host='nexus.it.locusdev.net'

[Rosswell]

I figured it would prolly be a good first issue, but didn't want to presume without looking into the cli parsing

[sbidoul]

#9163 touched that area in 20.3

[pradyunsg]

I'd start by reverting #9163, and we can revisit it after. :)