Merge branch 'master' into architecture

docs/html/development/architecture/index.rst

@@ -8,17 +8,17 @@ Architecture of pip's internals
     interested in helping out, please let us know in the `tracking issue`_.
 
 .. note::
-    Direct use of pip's internals is *not supported*.
-    For more details, see :ref:`Using pip from your program`.
+    Direct use of pip's internals is *not supported*, and these internals
+    can change at any time. For more details, see :ref:`Using pip from
+    your program`.
 
 
 .. toctree::
     :maxdepth: 2
 
     overview
     anatomy
-
-
+    package-finding
 
 
 .. _`tracking issue`: https://github.com/pypa/pip/issues/6831

docs/html/development/architecture/package-finding.rst

@@ -0,0 +1,202 @@
+Finding and choosing files (``index.py`` and ``PackageFinder``)
+---------------------------------------------------------------
+
+The ``index.py`` module is a top-level module in pip responsible for deciding
+what file to download and from where, given a requirement for a project. The
+module's functionality is largely exposed through and coordinated by the
+module's ``PackageFinder`` class.
+
+
+.. _index-py-overview:
+
+Overview
+********
+
+Here is a rough description of the process that pip uses to choose what
+file to download for a package, given a requirement:
+
+1. Access the various network and file system locations configured for pip
+   that contain package files. These locations can include, for example,
+   pip's :ref:`--index-url <--index-url>` (with default
+   https://pypi.org/simple/ ) and any configured
+   :ref:`--extra-index-url <--extra-index-url>` locations.
+   Each of these locations is a `PEP 503`_ "simple repository" page, which
+   is an HTML page of anchor links.
+2. Collect together all of the links (e.g. by parsing the anchor links
+   from the HTML pages) and create ``Link`` objects from each of these.
+3. Determine which of the links are minimally relevant, using the
+   :ref:`LinkEvaluator <link-evaluator-class>` class.  Create an
+   ``InstallationCandidate`` object (aka candidate for install) for each
+   of these relevant links.
+4. Further filter the collection of ``InstallationCandidate`` objects (using
+   the :ref:`CandidateEvaluator <candidate-evaluator-class>` class) to a
+   collection of "applicable" candidates.
+5. If there are applicable candidates, choose the best candidate by sorting
+   them (again using the :ref:`CandidateEvaluator
+   <candidate-evaluator-class>` class).
+
+The remainder of this section is organized by documenting some of the
+classes inside ``index.py``, in the following order:
+
+* the main :ref:`PackageFinder <package-finder-class>` class,
+* the :ref:`LinkEvaluator <link-evaluator-class>` class,
+* the :ref:`CandidateEvaluator <candidate-evaluator-class>` class,
+* the :ref:`CandidatePreferences <candidate-preferences-class>` class, and
+* the :ref:`BestCandidateResult <best-candidate-result-class>` class.
+
+
+.. _package-finder-class:
+
+The ``PackageFinder`` class
+***************************
+
+The ``PackageFinder`` class is the primary way through which code in pip
+interacts with ``index.py``. It is an umbrella class that encapsulates and
+groups together various package-finding functionality.
+
+The ``PackageFinder`` class is responsible for searching the network and file
+system for what versions of a package pip can install, and also for deciding
+which version is most preferred, given the user's preferences, target Python
+environment, etc.
+
+The pip commands that use the ``PackageFinder`` class are:
+
+* :ref:`pip download`
+* :ref:`pip install`
+* :ref:`pip list`
+* :ref:`pip wheel`
+
+The pip commands requiring use of the ``PackageFinder`` class generally
+instantiate ``PackageFinder`` only once for the whole pip invocation. In
+fact, pip creates this ``PackageFinder`` instance when command options
+are first parsed.
+
+With the excepton of :ref:`pip list`, each of the above commands is
+implemented as a ``Command`` class inheriting from ``RequirementCommand``
+(for example :ref:`pip download` is implemented by ``DownloadCommand``), and
+the ``PackageFinder`` instance is created by calling the
+``RequirementCommand`` class's ``_build_package_finder()`` method. ``pip
+list``, on the other hand, constructs its ``PackageFinder`` instance by
+calling the ``ListCommand`` class's ``_build_package_finder()``. (This
+difference may simply be historical and may not actually be necessary.)
+
+Each of these commands also uses the ``PackageFinder`` class for pip's
+"self-check," (i.e. to check whether a pip upgrade is available). In this
+case, the ``PackageFinder`` instance is created by the ``outdated.py``
+module's ``pip_version_check()`` function.
+
+The ``PackageFinder`` class is responsible for doing all of the things listed
+in the :ref:`Overview <index-py-overview>` section like fetching and parsing
+`PEP 503`_ simple repository HTML pages, evaluating which links in the simple
+repository pages are relevant for each requirement, and further filtering and
+sorting by preference the candidates for install coming from the relevant
+links.
+
+One of ``PackageFinder``'s main top-level methods is
+``find_best_candidate()``. This method does the following two things:
+
+1. Calls its ``find_all_candidates()`` method, which reads and parses all the
+   index URL's provided by the user, constructs a :ref:`LinkEvaluator
+   <link-evaluator-class>` object to filter out some of those links, and then
+   returns a list of ``InstallationCandidates`` (aka candidates for install).
+   This corresponds to steps 1-3 of the :ref:`Overview <index-py-overview>`
+   above.
+2. Constructs a ``CandidateEvaluator`` object and uses that to determine
+   the best candidate. It does this by calling the ``CandidateEvaluator``
+   class's ``compute_best_candidate()`` method on the return value of
+   ``find_all_candidates()``. This corresponds to steps 4-5 of the Overview.
+
+
+.. _link-evaluator-class:
+
+The ``LinkEvaluator`` class
+***************************
+
+The ``LinkEvaluator`` class contains the business logic for determining
+whether a link (e.g. in a simple repository page) satisfies minimal
+conditions to be a candidate for install (resulting in an
+``InstallationCandidate`` object). When making this determination, the
+``LinkEvaluator`` instance uses information like the target Python
+interpreter as well as user preferences like whether binary files are
+allowed or preferred, etc.
+
+Specifically, the ``LinkEvaluator`` class has an ``evaluate_link()`` method
+that returns whether a link is a candidate for install.
+
+Instances of this class are created by the ``PackageFinder`` class's
+``make_link_evaluator()`` on a per-requirement basis.
+
+
+.. _candidate-evaluator-class:
+
+The ``CandidateEvaluator`` class
+********************************
+
+The ``CandidateEvaluator`` class contains the business logic for evaluating
+which ``InstallationCandidate`` objects should be preferred. This can be
+viewed as a determination that is finer-grained than that performed by the
+``LinkEvaluator`` class.
+
+In particular, the ``CandidateEvaluator`` class uses the whole set of
+``InstallationCandidate`` objects when making its determinations, as opposed
+to evaluating each candidate in isolation, as ``LinkEvaluator`` does. For
+example, whether a pre-release is eligible for selection or whether a file
+whose hash doesn't match is eligible depends on properties of the collection
+as a whole.
+
+The ``CandidateEvaluator`` class uses information like the list of `PEP 425`_
+tags compatible with the target Python interpreter, hashes provided by the
+user, and other user preferences, etc.
+
+Specifically, the class has a ``get_applicable_candidates()`` method.
+This accepts the ``InstallationCandidate`` objects resulting from the links
+accepted by the ``LinkEvaluator`` class's ``evaluate_link()`` method, and
+it further filters them to a list of "applicable" candidates.
+
+The ``CandidateEvaluator`` class also has a ``sort_best_candidate()`` method
+that orders the applicable candidates by preference, and then returns the
+best (i.e. most preferred).
+
+Finally, the class has a ``compute_best_candidate()`` method that calls
+``get_applicable_candidates()`` followed by ``sort_best_candidate()``, and
+then returning a :ref:`BestCandidateResult <best-candidate-result-class>`
+object encapsulating both the intermediate and final results of the decision.
+
+Instances of ``CandidateEvaluator`` are created by the ``PackageFinder``
+class's ``make_candidate_evaluator()`` method on a per-requirement basis.
+
+
+.. _candidate-preferences-class:
+
+The ``CandidatePreferences`` class
+**********************************
+
+The ``CandidatePreferences`` class is a simple container class that groups
+together some of the user preferences that ``PackageFinder`` uses to
+construct ``CandidateEvaluator`` objects (via the ``PackageFinder`` class's
+``make_candidate_evaluator()`` method).
+
+A ``PackageFinder`` instance has a ``_candidate_prefs`` attribute whose value
+is a ``CandidatePreferences`` instance. Since ``PackageFinder`` has a number
+of responsibilities and options that control its behavior, grouping the
+preferences specific to ``CandidateEvaluator`` helps maintainers know which
+attributes are needed only for ``CandidateEvaluator``.
+
+
+.. _best-candidate-result-class:
+
+The ``BestCandidateResult`` class
+*********************************
+
+The ``BestCandidateResult`` class is a convenience "container" class that
+encapsulates the result of finding the best candidate for a requirement.
+(By "container" we mean an object that simply contains data and has no
+business logic or state-changing methods of its own.)
+
+The class is the return type of both the ``CandidateEvaluator`` class's
+``compute_best_candidate()`` method and the ``PackageFinder`` class's
+``find_best_candidate()`` method.
+
+
+.. _`PEP 425`: https://www.python.org/dev/peps/pep-0425/
+.. _`PEP 503`: https://www.python.org/dev/peps/pep-0503/

news/5306.bugfix

@@ -0,0 +1 @@
+Ignore errors copying socket files for local source installs (in Python 3).

news/6705.bugfix

@@ -0,0 +1,2 @@
+Fix ``--trusted-host`` processing under HTTPS to trust any port number used
+with the host.

news/6858.feature

@@ -0,0 +1 @@
+Make ``pip show`` warn about packages not found.

news/6869.trivial

@@ -0,0 +1 @@
+Clarify WheelBuilder.build() a bit

news/6883.trivial

@@ -0,0 +1 @@
+replace is_vcs_url function by is_vcs Link property

news/6885.bugfix

@@ -0,0 +1 @@
+Fix 'm' flag erroneously being appended to ABI tag in Python 3.8 on platforms that do not provide SOABI

news/6890.bugfix

@@ -0,0 +1,2 @@
+Hide security-sensitive strings like passwords in log messages related to
+version control system (aka VCS) command invocations.

src/pip/_internal/cli/req_command.py

@@ -62,7 +62,7 @@ def _build_session(self, options, retries=None, timeout=None):
                 if options.cache_dir else None
             ),
             retries=retries if retries is not None else options.retries,
-            insecure_hosts=options.trusted_hosts,
+            trusted_hosts=options.trusted_hosts,
             index_urls=self._get_index_urls(options),
         )
 
@@ -276,7 +276,6 @@ def _build_package_finder(
         return PackageFinder.create(
             search_scope=search_scope,
             selection_prefs=selection_prefs,
-            trusted_hosts=options.trusted_hosts,
             session=session,
             target_python=target_python,
         )

src/pip/_internal/commands/install.py

@@ -78,7 +78,7 @@ def build_wheels(
     # Always build PEP 517 requirements
     build_failures = builder.build(
         pep517_requirements,
-        autobuilding=True,
+        should_unpack=True,
     )
 
     if should_build_legacy:
@@ -87,7 +87,7 @@ def build_wheels(
         # install for those.
         builder.build(
             legacy_requirements,
-            autobuilding=True,
+            should_unpack=True,
         )
 
     return build_failures

src/pip/_internal/commands/list.py

@@ -126,7 +126,6 @@ def _build_package_finder(self, options, session):
         return PackageFinder.create(
             search_scope=search_scope,
             selection_prefs=selection_prefs,
-            trusted_hosts=options.trusted_hosts,
             session=session,
         )
 
@@ -192,7 +191,7 @@ def iter_packages_latest_infos(self, packages, options):
                 evaluator = finder.make_candidate_evaluator(
                     project_name=dist.project_name,
                 )
-                best_candidate = evaluator.get_best_candidate(all_candidates)
+                best_candidate = evaluator.sort_best_candidate(all_candidates)
                 if best_candidate is None:
                     continue
 

src/pip/_internal/commands/show.py

@@ -60,6 +60,11 @@ def search_packages_info(query):
         installed[canonicalize_name(p.project_name)] = p
 
     query_names = [canonicalize_name(name) for name in query]
+    missing = sorted(
+        [name for name, pkg in zip(query, query_names) if pkg not in installed]
+    )
+    if missing:
+        logger.warning('Package(s) not found: %s', ', '.join(missing))
 
     for dist in [installed[pkg] for pkg in query_names if pkg in installed]:
         package = {