APPSI Solver Error: cannot remove variable q2 - it is still being used by constraints or the objective

[ZedongPeng]

Summary

APPSI solver will report the cannot remove variable q2 - it is still being used by constraints or the objective error if the variable is deleted and defined again.

Steps to reproduce the issue

# example.py
from pyomo.environ import *
from pyomo.solvers.tests.models.QCP_simple import QCP_simple

QCP_model = QCP_simple()
QCP_model._generate_model()
model = QCP_model.model
opt = SolverFactory('appsi_gurobi') # or 'appsi_cplex'
result = opt.solve(QCP_model.model,tee=True)
print(result)

QCP_model.model.del_component('q2')
model.q2 = Var(bounds=(-2, None))
result = opt.solve(QCP_model.model,tee=True)
print(result)

Error Message

$ # Output message here, including entire stack trace, if available
Model has 4 quadratic constraints
Coefficient statistics:
  Matrix range     [1e+00, 1e+00]
  QMatrix range    [1e+00, 1e+00]
  Objective range  [1e+00, 1e+00]
  Bounds range     [2e-01, 2e+00]
  RHS range        [1e+00, 1e+00]
  QRHS range       [2e-01, 5e+00]
Presolve removed 0 rows and 3 columns
Presolve time: 0.00s
Presolved: 7 rows, 7 columns, 14 nonzeros
Presolved model has 2 second-order cone constraints
Ordering time: 0.00s

Barrier statistics:
 AA' NZ     : 2.100e+01
 Factor NZ  : 2.800e+01
 Factor Ops : 1.400e+02 (less than 1 second per iteration)
 Threads    : 1

                  Objective                Residual
Iter       Primal          Dual         Primal    Dual     Compl     Time
   0   2.43809524e+00  2.43809524e+00  5.43e-01 4.40e-01  2.15e-01     0s
   1   2.49156971e+00  2.64695156e+00  3.89e-02 4.84e-07  2.42e-02     0s
   2   2.48970091e+00  2.50659507e+00  4.28e-08 5.31e-13  1.88e-03     0s
   3   2.49131949e+00  2.49163010e+00  9.30e-10 6.92e-14  3.45e-05     0s
   4   2.49156171e+00  2.49156262e+00  1.56e-12 4.00e-12  1.01e-07     0s

Barrier solved model in 4 iterations and 0.00 seconds (0.00 work units)
Optimal objective 2.49156171e+00


Problem: 
- Lower bound: 2.4915617113321344
  Upper bound: 2.4915617113321344
  Number of objectives: 1
  Number of constraints: 0
  Number of variables: 0
  Sense: -1
Solver: 
- Status: ok
  Termination condition: optimal
  Termination message: TerminationCondition.optimal
Solution: 
- number of solutions: 0
  number of solutions displayed: 0

Traceback (most recent call last):
  File "/Users/pengzedong1/Code/test.py", line 37, in <module>
    result = opt.solve(QCP_model.model,tee=True)
  File "/Users/pengzedong1/Github/pyomo/pyomo/contrib/appsi/base.py", line 1554, in solve
    results: Results = super(LegacySolverInterface, self).solve(model)
  File "/Users/pengzedong1/Github/pyomo/pyomo/contrib/appsi/solvers/gurobi.py", line 397, in solve
    self.update(timer=timer)
  File "/Users/pengzedong1/Github/pyomo/pyomo/contrib/appsi/solvers/gurobi.py", line 1146, in update
    super(Gurobi, self).update(timer=timer)
  File "/Users/pengzedong1/Github/pyomo/pyomo/contrib/appsi/base.py", line 1338, in update
    self.remove_variables(old_vars)
  File "/Users/pengzedong1/Github/pyomo/pyomo/contrib/appsi/base.py", line 1181, in remove_variables
    raise ValueError(
ValueError: cannot remove variable q2 - it is still being used by constraints or the objective

Information on your system

Pyomo version: 6.6.1.dev0
Python version: 3.9.10
Operating system: macOS Monterey Version 12.1
How Pyomo was installed (PyPI, conda, source): source
Solver (if applicable):appsi_cplex, appsi_gurobi

Additional information

[ZedongPeng]

@michaelbynum

[michaelbynum]

Isn't this correct? The new variable in the model is not the same one that is used in the constraints/objective.

[michaelbynum]

Just to be clear, when a variable is deleted and a new one is created with the same name, it is not the same variable:

>>> import pyomo.environ as pe
>>> m = pe.ConcreteModel()
>>> m.x = pe.Var()
>>> m.c = pe.Constraint(expr=m.x == 1)
>>> m.del_component('x')
>>> m.x = pe.Var()
>>> from pyomo.core.expr.visitor import identify_variables
>>> from pyomo.common.collections import ComponentSet
>>> m.x in ComponentSet(identify_variables(m.c.body))
False

[ZedongPeng]

There is a better example. model.q3 is newly defined and only appears in model.obj. If we delete both model.q3 and model.obj, then define them again. The same error will be reported.
Is it too strict to raise a ValueError here? I think in this case appsi_solver should at least solve the new model, right?

from pyomo.environ import *
from pyomo.solvers.tests.models.QCP_simple import QCP_simple

QCP_model = QCP_simple()
QCP_model._generate_model()
model = QCP_model.model
model.q3 = Var(bounds=(-2, None))
model.obj = Objective(expr=model.x + model.q1 - model.q2 - model.q3, sense=maximize)
opt = SolverFactory('appsi_gurobi') # or 'appsi_cplex'
result = opt.solve(QCP_model.model,tee=True)
print(result)


model.del_component('q3')
model.q3 = Var(bounds=(-2, None))
model.del_component('obj')
model.obj = Objective(expr=model.x + model.q1 - model.q2 - model.q3, sense=maximize)
# opt = SolverFactory('appsi_gurobi')
result = opt.solve(QCP_model.model,tee=True)
print(result)

[michaelbynum]

Thanks. This helps. I'll get this fixed.

[ZedongPeng]

Hi @michaelbynum . Any progress on this?

[ZedongPeng]

Hi @michaelbynum . #2903 didn't fix this issue.
If I run the following code, the same error will show up again with the latest Pyomo.(installed through python setup.py develop)

from pyomo.environ import *
from pyomo.solvers.tests.models.QCP_simple import QCP_simple

QCP_model = QCP_simple()
QCP_model._generate_model()
model = QCP_model.model
model.q3 = Var(bounds=(-2, None))
model.obj = Objective(expr=model.x + model.q1 - model.q2 - model.q3, sense=maximize)
opt = SolverFactory('appsi_gurobi') # or 'appsi_cplex'
result = opt.solve(QCP_model.model,tee=True)
print(result)


model.del_component('q3')
model.q3 = Var(bounds=(-2, None))
model.del_component('obj')
model.obj = Objective(expr=model.x + model.q1 - model.q2 - model.q3, sense=maximize)
# opt = SolverFactory('appsi_gurobi')
result = opt.solve(QCP_model.model,tee=True)
print(result)

Output

WARNING: Implicitly replacing the Component attribute obj (type=<class
'pyomo.core.base.objective.ScalarObjective'>) on block QCP_simple with a new
Component (type=<class 'pyomo.core.base.objective.ScalarObjective'>). This is
usually indicative of a modelling error. To avoid this warning, use
block.del_component() and block.add_component().
Version identifier: 22.1.1.0 | 2022-11-28 | 9160aff4d
CPXPARAM_Read_DataCheck                          1
Tried aggregator 1 time.
QCP Presolve eliminated 1 rows and 4 columns.
Reduced QCP has 7 rows, 7 columns, and 14 nonzeros.
Reduced QCP has 2 quadratic constraints.
Presolve time = 0.00 sec. (0.01 ticks)
Parallel mode: using up to 12 threads for barrier.
Number of nonzeros in lower triangle of A*A' = 21
Using Approximate Minimum Degree ordering
Total time for automatic ordering = 0.00 sec. (0.00 ticks)
Summary statistics for Cholesky factor:
  Threads                   = 12
  Rows in Factor            = 7
  Integer space required    = 7
  Total non-zeros in factor = 28
  Total FP ops to factor    = 140
 Itn      Primal Obj        Dual Obj  Prim Inf Upper Inf  Dual Inf Inf Ratio
   0   4.2000000e+00   4.2000000e+00  6.69e+00  0.00e+00  6.83e+00  1.00e+00
   1   4.4505445e+00   4.5170460e+00  6.69e+00  0.00e+00  6.83e+00  1.37e+02
   2   4.4887794e+00   4.4973082e+00  4.94e-01  0.00e+00  5.04e-01  6.77e+02
   3   4.4905466e+00   4.4920134e+00  6.69e-02  0.00e+00  6.83e-02  2.69e+03
   4   4.4915508e+00   4.4918356e+00  1.23e-02  0.00e+00  1.26e-02  9.06e+03
   5   4.4915775e+00   4.4916018e+00  2.64e-03  0.00e+00  2.70e-03  9.32e+04
   6   4.4915622e+00   4.4915628e+00  2.34e-04  0.00e+00  2.39e-04  1.81e+06
   7   4.4915623e+00   4.4915623e+00  7.44e-06  0.00e+00  7.60e-06  6.78e+07
   8   4.4915622e+00   4.4915622e+00  1.98e-07  0.00e+00  2.02e-07  6.27e+09

Problem: 
- Lower bound: 4.491562247172249
  Upper bound: 4.491562247172249
  Number of objectives: 1
  Number of constraints: 0
  Number of variables: 0
  Sense: -1
Solver: 
- Status: ok
  Termination condition: optimal
  Termination message: TerminationCondition.optimal
Solution: 
- number of solutions: 0
  number of solutions displayed: 0

Traceback (most recent call last):
  File "/Users/zedongpeng/Github/pyomo/pyomo/contrib/mindtpy/tests/test.py", line 19, in <module>
    result = opt.solve(QCP_model.model, tee=True)
  File "/Users/zedongpeng/opt/anaconda3/envs/py310/lib/python3.10/site-packages/Pyomo-6.6.2.dev0-py3.10.egg/pyomo/contrib/appsi/base.py", line 1554, in solve
    results: Results = super(LegacySolverInterface, self).solve(model)
  File "/Users/zedongpeng/opt/anaconda3/envs/py310/lib/python3.10/site-packages/Pyomo-6.6.2.dev0-py3.10.egg/pyomo/contrib/appsi/solvers/cplex.py", line 222, in solve
    self._writer.write(model, self._filename + '.lp', timer=timer)
  File "/Users/zedongpeng/opt/anaconda3/envs/py310/lib/python3.10/site-packages/Pyomo-6.6.2.dev0-py3.10.egg/pyomo/contrib/appsi/writers/lp_writer.py", line 168, in write
    self.update(timer=timer)
  File "/Users/zedongpeng/opt/anaconda3/envs/py310/lib/python3.10/site-packages/Pyomo-6.6.2.dev0-py3.10.egg/pyomo/contrib/appsi/base.py", line 1338, in update
    self.remove_variables(old_vars)
  File "/Users/zedongpeng/opt/anaconda3/envs/py310/lib/python3.10/site-packages/Pyomo-6.6.2.dev0-py3.10.egg/pyomo/contrib/appsi/base.py", line 1181, in remove_variables
    raise ValueError(
ValueError: cannot remove variable q3 - it is still being used by constraints or the objective

[michaelbynum]

Strange. I just ran this, and I did not get an error.

[ZedongPeng]

Yes. It's strange. I reinstalled appsi, but the error still showed up.

# Name                    Version                   Build  Channel
pyomo                     6.6.2.dev0                dev_0    <develop>

[michaelbynum]

You are on the main branch?

[ZedongPeng]

Yes. I am on the main branch. I found the problem.
If I define opt = SolverFactory('appsi_cplex') again after the model has changed, the script will work well.

from pyomo.environ import *
from pyomo.solvers.tests.models.QCP_simple import QCP_simple

QCP_model = QCP_simple()
QCP_model._generate_model()
model = QCP_model.model
model.q3 = Var(bounds=(-2, None))
model.obj = Objective(expr=model.x + model.q1 - model.q2 - model.q3, sense=maximize)
opt = SolverFactory('appsi_cplex')  # or 'appsi_cplex'
result = opt.solve(QCP_model.model, tee=True)
print(result)


model.del_component('q3')
model.q3 = Var(bounds=(-2, None))
model.del_component('obj')
model.obj = Objective(expr=model.x + model.q1 - model.q2 - model.q3, sense=maximize)
opt = SolverFactory('appsi_cplex')
result = opt.solve(QCP_model.model, tee=True)
print(result)

However, the key of appsi_solver is defining opt only once and repeatedly solving the model, right?

[michaelbynum]

Correct. You should not have to reconstruct the solver. That defeats the purpose of Appsi. Something is wrong. It's hard to debug when I can't reproduce the issue though.

[ZedongPeng]

I tried to start from the beginning.

1. Create a new virtual Python=3.9.16 environment using Anaconda.
2. Install Pyomo from Github (main branch).
3. Install appsi_solver.
4. Run the code again.

The error still showed up.

[ZedongPeng]

@michaelbynum Any suggestions to further test on this?

[ZedongPeng]

It suddenly works as expected. Thanks for your help.
Just one more question, I use anaconda. The base environment is Python 3.7 and works well. However, when I changed to another virtual environment with Python 3.10, it failed.
Do I need to rebuild the appsi_solver in the new virtual environment?

[ZedongPeng]

It worked well after I rebuilt it in the new virtual environment. Thanks for your effort.