High dual infeasibility and barrier parameters issue for non-Linear optimization

[uttamcadambi07]

Hi,

I am using IPOPT to solve a non-linear optimization problem. It is actually part of a bigger package called DAFoam that provides the Navier-Stokes equations in a discrete format as input to IPOPT as the constraints (so one equality constraint) and I am trying to optimize a scalar field \beta (where the Navier-Stokes equations in the constraint is a function of this \beta field).

The objective function is simply a difference between the reference field of one of the quantities and a baseline case.

I am attaching the results of an optimization case.

List of user-set options:

                                    Name   Value                used
                          acceptable_tol = 0.4                   yes
                             alpha_for_y = full                  yes
                           compl_inf_tol = 0.04                  yes
                            dual_inf_tol = 0.04                  yes
                        file_print_level = 5                     yes
                   hessian_approximation = limited-memory        yes
              limited_memory_max_history = 26                    yes
                           linear_solver = mumps                 yes
                                max_iter = 25                    yes
                             mu_strategy = adaptive              yes
                      nlp_scaling_method = gradient-based        yes
                             output_file = opt_IPOPT.out         yes
                             print_level = 5                     yes
                      print_user_options = yes                   yes
                                recalc_y = yes                   yes
                                      sb = yes                   yes
                                     tol = 0.4                   yes
This is Ipopt version 3.13.5, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:     3500
Number of nonzeros in Lagrangian Hessian.............:        0

Total number of variables............................:     3500
                     variables with only lower bounds:        0
                variables with lower and upper bounds:     3500
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        1
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  1.0371885e+00 0.00e+00 1.00e+02   0.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1  4.1485929e-01 0.00e+00 4.68e+04  -2.3 5.00e-03    -  1.00e+00 2.48e-03f  4
   2  4.1437480e-01 0.00e+00 5.25e+04  -2.9 8.98e-05    -  1.00e+00 1.18e-04f 14
   3  4.1427524e-01 0.00e+00 5.43e+04  -2.7 1.39e-04    -  1.00e+00 9.51e-06f 17
   4  4.1453945e-01 0.00e+00 1.14e+03  -2.7 8.44e-05    -  1.00e+00 1.00e+00S1076
   5  3.2972612e+00 0.00e+00 2.07e+02  -2.7 2.71e-03    -  3.46e-02 3.46e-02S1071
   6  3.1859352e+00 0.00e+00 5.75e+02  -2.7 4.43e-05    -  1.00e+00 1.00e+00f  1
   7  3.1859205e+00 0.00e+00 9.73e+01  -2.7 1.11e-05    -  1.00e+00 1.00e+00f  1
   8  1.9401865e+00 0.00e+00 1.92e+02  -2.7 3.51e-05    -  8.44e-01 1.00e+00f  1
   9  1.9401762e+00 0.00e+00 3.01e+02  -2.7 2.44e-04    -  1.00e+00 6.47e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  10  3.5940374e-01 0.00e+00 4.20e+03  -2.7 3.70e-04    -  2.76e-01 2.49e-01f  1
  11  3.5940499e-01 0.00e+00 3.05e+04  -1.0 5.49e-03    -  1.00e+00 1.18e-02f  1
  12  3.0792260e-01 0.00e+00 1.30e+04  -1.1 1.28e-03    -  5.55e-02 4.91e-02f  2
  13  2.9766395e-01 0.00e+00 8.11e+03  -1.0 1.56e-04    -  1.00e+00 2.04e-01f  2
  14  3.8941467e-01 0.00e+00 1.06e+04  -1.0 2.50e-05    -  8.63e-01 1.00e+00f  1
  15  4.8049127e-01 0.00e+00 3.33e+03  -1.1 4.83e-05    -  1.00e+00 2.50e-01f  3
  16  4.8048698e-01 0.00e+00 1.14e+04  -1.1 2.51e-05    -  1.00e+00 1.00e+00f  1
  17  7.6803257e-01 0.00e+00 1.07e+04  -1.1 5.04e-05    -  1.00e+00 2.50e-01f  3
  18  8.0541075e-01 0.00e+00 2.29e+05  -1.1 8.42e-05    -  9.06e-01 1.06e-01f  4
  19  7.4864205e-01 0.00e+00 2.27e+04  -1.1 1.82e-03    -  5.58e-02 2.06e-04f  9
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  20  7.4573496e-01 0.00e+00 1.70e+04  -1.1 9.37e-05    -  9.87e-01 1.83e-04f 13
  21  6.4411565e-01 0.00e+00 5.59e+04  -1.1 5.91e-05    -  8.91e-01 2.50e-01f  3
  22  6.8698766e-01 0.00e+00 1.20e+04  -1.1 4.79e-04    -  1.56e-01 1.56e-01S1073
  23  6.3708282e-01 0.00e+00 7.79e+03  -1.1 9.32e-05    -  1.00e+00 2.12e-01f  3
  24  5.8256807e-01 0.00e+00 6.79e+04  -1.1 2.15e-05    -  1.00e+00 1.00e+00f  1
  25  5.7154546e-01 0.00e+00 8.97e+03  -1.1 9.46e-05    -  1.00e+00 1.49e-01f  3

Number of Iterations....: 25

                                   (scaled)                 (unscaled)
Objective...............:   1.3350410232052107e-02    5.7154546418680674e-01
Dual infeasibility......:   8.9697401987472276e+03    3.8400425428275601e+05
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   1.6001646979332093e-01    6.8504776943846641e+00
Overall NLP error.......:   1.1274446845830216e+03    3.8400425428275601e+05


Number of objective function evaluations             = 3374
Number of objective gradient evaluations             = 26
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 3374
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 26
Number of Lagrangian Hessian evaluations             = 0
Total CPU secs in IPOPT (w/o function evaluations)   =     96.504
Total CPU secs in NLP function evaluations           =   8157.938

EXIT: Maximum Number of Iterations Exceeded.

In this case, I notice somethings that I am finding hard to understand. Please keep these in mind before going over my questions

1. I have a decent understanding of the concepts related to how IPOPT works (about 50-60%).
2. I have ran a lot of test cases to complement my understanding that I obtained through reading the documentation of IPOPT.

Here are my questions

1. The barrier parameter seems to reduce initially but then suddenly increases to -1.1 and stays the same. IPOPT solves successive barrier problems by reducing the barrier parameter, right? So why is this observed?
2. I notice that this is accompanied by a high dual infeasibility. Now if my understanding is correct, the high value of dual infeasibility means that the solution is in the infeasible region (so a high constraint violation of the barrier problem, which is very well in line with the fact that the barrier parameter does not reduce). Now my question is – does this value of high dual infeasibility mean that my solution space for the parameter that I am trying to optimize (which is \beta) is VERY close to OR NOT in the bounds [\beta_lowerbound, \beta_upperbound]? Basically I do not care what the limits are (this limit on upper and lower values of the \beta field is my only inequality constraint). Interestingly my objective reduces in magnitude and my error goes to a very small value for this case. Should I be concerned about the dual infeasibility?
3. One of the methods that we came up with to try and tackle the dual infeasibility problem (and possibly improve our objective function value reduction) is to change the bounds relative to the existing value of \beta that we obtain after a few iterations (I am not sure if this corresponds to warm re-starting). This would mean that the bounds are altered to ensure that the feasible space of \bneta is not what it originally was. Do you think this might help?

Please do let me know. I want to try to fix this issue with my case and your answers will be helpful in doing so.

P.S I am attaching another case that was similar.

List of user-set options:

                                    Name   Value                used
                          acceptable_tol = 1e-05                 yes
                             alpha_for_y = full                  yes
                         constr_viol_tol = 1e-05                 yes
                        file_print_level = 5                     yes
                   hessian_approximation = limited-memory        yes
              limited_memory_max_history = 26                    yes
                           linear_solver = mumps                 yes
                                max_iter = 25                    yes
                             mu_strategy = adaptive              yes
                      nlp_scaling_method = gradient-based        yes
                             output_file = opt_IPOPT.out         yes
                             print_level = 5                     yes
                      print_user_options = yes                   yes
                                recalc_y = yes                   yes
                                      sb = yes                   yes
                                     tol = 1e-05                 yes
This is Ipopt version 3.13.5, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).

Number of nonzeros in equality constraint Jacobian...:        0
Number of nonzeros in inequality constraint Jacobian.:     3500
Number of nonzeros in Lagrangian Hessian.............:        0

Total number of variables............................:     3500
                     variables with only lower bounds:        0
                variables with lower and upper bounds:     3500
                     variables with only upper bounds:        0
Total number of equality constraints.................:        0
Total number of inequality constraints...............:        1
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  4.9789015e-01 0.00e+00 1.00e+02   0.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1  1.9914833e-01 0.00e+00 4.68e+04  -2.3 5.00e-03    -  1.00e+00 2.48e-03f  4
   2  1.9891576e-01 0.00e+00 5.25e+04  -2.9 8.98e-05    -  1.00e+00 1.18e-04f 14
   3  1.9886797e-01 0.00e+00 5.43e+04  -2.7 1.39e-04    -  1.00e+00 9.51e-06f 17
   4  1.9899479e-01 0.00e+00 1.14e+03  -2.7 8.44e-05    -  1.00e+00 1.00e+00S1076
   5  1.5828057e+00 0.00e+00 2.07e+02  -2.7 2.71e-03    -  3.46e-02 3.46e-02S1071
   6  1.5293759e+00 0.00e+00 5.75e+02  -2.7 4.43e-05    -  1.00e+00 1.00e+00f  1
   7  1.5293688e+00 0.00e+00 9.73e+01  -2.7 1.11e-05    -  1.00e+00 1.00e+00f  1
   8  9.3122204e-01 0.00e+00 1.91e+02  -2.7 3.51e-05    -  8.44e-01 1.00e+00f  1
   9  9.3121708e-01 0.00e+00 3.02e+02  -2.7 2.44e-04    -  1.00e+00 6.48e-01f  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  10  1.7235384e-01 0.00e+00 4.24e+03  -2.7 3.71e-04    -  2.76e-01 2.49e-01f  1
  11  1.7235445e-01 0.00e+00 1.95e+05  -1.0 5.55e-03    -  1.00e+00 1.16e-02f  1
  12  1.4884554e-01 0.00e+00 6.88e+03  -1.1 8.73e-03    -  1.41e-02 3.25e-05f  9
  13  1.4884601e-01 0.00e+00 1.65e+03  -1.9 1.10e-05    -  1.00e+00 1.00e+00f  1
  14  1.6641308e-01 0.00e+00 2.16e+03  -2.0 4.96e-05    -  8.87e-01 5.00e-01f  2
  15  1.6462551e-01 0.00e+00 1.59e+03  -2.0 2.48e-05    -  1.00e+00 1.25e-01f  4
  16  1.6424528e-01 0.00e+00 1.24e+03  -2.0 2.13e-05    -  1.00e+00 1.25e-01f  4
  17  1.6434360e-01 0.00e+00 1.16e+03  -2.0 1.09e-04    -  9.90e-01 7.55e-03f  8
  18  1.6586501e-01 0.00e+00 2.88e+03  -2.0 1.15e-04    -  1.00e+00 1.42e-02f  7
  19  1.6505703e-01 0.00e+00 1.31e+03  -2.0 2.45e-04    -  3.58e-01 1.63e-03f  9
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  20  1.7171035e-01 0.00e+00 3.25e+03  -2.0 9.41e-04    -  1.17e-01 4.51e-02f  3
  21  1.6097834e-01 0.00e+00 2.73e+03  -2.0 1.15e-04    -  1.00e+00 1.55e-02f  7
  22  1.6203955e-01 0.00e+00 1.83e+03  -2.0 9.15e-05    -  9.00e-01 3.12e-02f  6
  23  1.6657540e-01 0.00e+00 1.12e+03  -2.0 4.44e-05    -  1.00e+00 1.25e-01f  4
  24  1.7966976e-01 0.00e+00 3.17e+03  -2.0 2.69e-05    -  1.00e+00 1.00e+00f  1
  25  1.8351557e-01 0.00e+00 5.82e+03  -2.0 3.09e-04    -  3.00e-01 3.00e-01S1074

Number of Iterations....: 25

                                   (scaled)                 (unscaled)
Objective...............:   8.9297826896246411e-03    1.8351556962932322e-01
Dual infeasibility......:   5.8211037058816610e+03    1.1962924514360621e+05
Constraint violation....:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   1.0695771018231761e-02    2.1980831776748591e-01
Overall NLP error.......:   5.4119191598579000e+03    1.1962924514360621e+05


Number of objective function evaluations             = 3396
Number of objective gradient evaluations             = 26
Number of equality constraint evaluations            = 0
Number of inequality constraint evaluations          = 3396
Number of equality constraint Jacobian evaluations   = 0
Number of inequality constraint Jacobian evaluations = 26
Number of Lagrangian Hessian evaluations             = 0
Total CPU secs in IPOPT (w/o function evaluations)   =    106.083
Total CPU secs in NLP function evaluations           =   7514.556

EXIT: Maximum Number of Iterations Exceeded.

[svigerske]

There is something odd in the problem statistics:

Total number of equality constraints.................:        0
Total number of inequality constraints...............:        1
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

You wrote something about an equality constraint, but it seems you have one free constraint, i.e., left-hand side at -infinity and right-hand side at +infinity. It's not surprising then that there inf_pr is always 0.

Since you choose mu_strategy = adaptive (https://coin-or.github.io/Ipopt/OPTIONS.html#OPT_mu_strategy), there is no guarantee that mu will always decrease. If you increase print_level to 6, you should get more info on how the Barrier parameter is chosen.
If you want to follow in the code, then start at https://github.com/coin-or/Ipopt/blob/stable/3.14/src/Algorithm/IpAdaptiveMuUpdate.cpp#L252
There are two different modes the adaptive mu oracle choice may be in. If it is in free mode, then the actual calculation of mu seems to happen in https://github.com/coin-or/Ipopt/blob/stable/3.14/src/Algorithm/IpQualityFunctionMuOracle.cpp by default.
Details on the adaptive mu strategy seem to be described in https://optimization-online.org/?p=9717

The high dual infeasibility means that Ipopt couldn't confirm that the current point is locally optimal. It can also find a direction in which to decrease the objective function. But going in that direction doesn't seem to help much to get the dual infeasibility down. Maybe the direction vector is not much accurate or the objective is very nonlinear and nonconvex. That the Hessian is only approximated doesn't help on getting fast convergence. You should also check whether the derivatives given to Ipopt seem correct: https://coin-or.github.io/Ipopt/SPECIALS.html#DERIVCHECK

[uttamcadambi07]

Excellent. Thank you very much. These points are very helpful. My Navier--Stokes equations are my equality constraints and I have bounds on my design variable (upper and lower) so it is strange that its is not reflected in the log. I will contact the developers of DAFoam to clarify this.
I will also try changing the mu oracle.