Speed up backtracking by skipping unrelated states.

The current backtracking logic assumes that the last state in the stack
is the state that caused the incompatibility. This is not always the case.
For example, given three requirements A, B and C, with dependencies
A1, B1 and C1, where A1 and B1 are incompatible.
The requirements are processed one after the other, so the last state
is related to C, while the incompatibility is caused by B.

The current behavior causes significant slowdowns in case there are
many candidates for B and C, as all their combination are evaluated
before a compatible version of B can be found.

The new behavior discards a state if the packages that cause the
incompatibility are not found among the direct dependencies
of the candidate in the current state.
In our example, this causes the state related to C to be dropped
without evaluating any of its candidates, until a compatible candidate
of B is found.

Signed-off-by: Stefano Bennati [email protected]

src/resolvelib/resolvers.py

@@ -266,7 +266,7 @@ def _attempt_to_pin_criterion(self, name):
         # end, signal for backtracking.
         return causes
 
-    def _backtrack(self):
+    def _backtrack(self, causes):
         """Perform backtracking.
 
         When we enter here, the stack is like this::
@@ -283,22 +283,33 @@ def _backtrack(self):
 
         Each iteration of the loop will:
 
-        1.  Discard Z.
-        2.  Discard Y but remember its incompatibility information gathered
+        1.  Identify Z. The incompatibility is not necessarily caused by the latest state.
+            For example, given three requirements A, B and C, with dependencies
+            A1, B1 and C1, where A1 and B1 are incompatible: 
+            the last state might be related to C, so we want to discard the previous state.
+        2.  Discard Z.
+        3.  Discard Y but remember its incompatibility information gathered
             previously, and the failure we're dealing with right now.
-        3.  Push a new state Y' based on X, and apply the incompatibility
+        4.  Push a new state Y' based on X, and apply the incompatibility
             information from Y to Y'.
-        4a. If this causes Y' to conflict, we need to backtrack again. Make Y'
+        5a. If this causes Y' to conflict, we need to backtrack again. Make Y'
             the new Z and go back to step 2.
-        4b. If the incompatibilities apply cleanly, end backtracking.
+        5b. If the incompatibilities apply cleanly, end backtracking.
         """
         while len(self._states) >= 3:
             # Remove the state that triggered backtracking.
             del self._states[-1]
 
-            # Retrieve the last candidate pin and known incompatibilities.
-            broken_state = self._states.pop()
-            name, candidate = broken_state.mapping.popitem()
+            # Ensure to backtrack to a state that caused the incompatibility
+            incompatible_state = False
+            while not incompatible_state:
+                # Retrieve the last candidate pin and known incompatibilities.
+                broken_state = self._states.pop()
+                name, candidate = broken_state.mapping.popitem()
+                current_dependencies = set([d.name for d in self._p.get_dependencies(candidate)])
+                incompatible_deps = set([c.parent.name for c in causes]) # Parent should never be null
+                incompatible_state = current_dependencies.intersection(incompatible_deps) != set()
+
             incompatibilities_from_broken = [
                 (k, list(v.incompatibilities))
                 for k, v in broken_state.criteria.items()
@@ -406,7 +417,7 @@ def resolve(self, requirements, max_rounds):
                 # Backtrack if pinning fails. The backtrack process puts us in
                 # an unpinned state, so we can work on it in the next round.
                 self._r.resolving_conflicts(causes=causes)
-                success = self._backtrack()
+                success = self._backtrack(causes)
                 self.state.backtrack_causes[:] = causes
 
                 # Dead ends everywhere. Give up.