Expanded expression template support

[jsiirola]

Fixes #N/A

(but provides the first steps for #752 and #1153)

Summary/Motivation:

This PR expands the template system to support hierarchical models and embedded sum expressions. Templates can now be formed for hierarchical models where the IndexTemplate(s) appear in the blocks and not just in Var/Param components. This also adds support for generating unexpanded templates of expressions containing summations. For example, considering the following "complex" rule:

m = ConcreteModel()
m.I = RangeSet(3)
m.J = RangeSet(3)
m.K = Set(m.I, initialize={1:[10], 2:[10,20], 3:[10,20,30]})
m.x = Var(m.I,m.J,[10,20,30])
@m.Constraint()
def c(m):
    return sum( sum(m.x[i,j,k] for k in m.K[i])
                for j in m.J for i in m.I ) <= 0

The following now works:

>>> template, indices = templatize_constraint(m.c)
>>> print(str(template))
SUM(SUM(x[_2,_1,_3] for _3 in K[_2]) for _1 in J for _2 in I)  <=  0.0

To implement this, the PR proposes several core changes:

1. Add a new PyomoModelingObject base class. This slotized interface is a new common base class for NumericValue and _ComponentBase class hierarchies and defines the fundamental is_*_type methods (all of which return False). By implementing this as a common base class, it makes it easier to support classes that "blur" the distinction between numeric types, component types, and expression types. This declares the minimal set of methods that Pyomo classes (that is, classes appearing in models / expression trees that are not in native_types) are expected to implement. These are:

   • is_expression_type(): True if this object is a non-leaf expression node. If True, the object is expected to implement the ExpressionBase API.
   • is_numeric_type(): True if the object can represent a numeric value. If True, then this object is expected to implement the NumericValue API.
   • is_component_type(): True if this object is a Pyomo component or ComponentData. If True, the object is expected to implement the _ComponentBase API.
   • is variable_type() : Preserved for backwards compatibility. A future PR will look at the possibility of deprecating / removing this method.
   • is_parameter_type(): Preserved for backwards compatibility. It is currently not used anywhere in the codebase, and will likely be deprecated in a future PR.
   • is_named_expression_type(): Preserved for backwards compatibility. A future PR will look at the possibility of deprecating / removing this method.

2. Expanding the StreamBasedExpressionVisitor. This adds a new initializeWalker callback, and expands the API for the beforeChild, acceptChildResult, and afterChild methods to also include the index of the child node in the parent node's args list. The walker also allows nodes to return args that implement a context manager api (without forcing the walker to define enterNode & exitNode).

3. Expanded IndexedComponent_slice class. This class is now "public", more robust, and hashable (can be a key in a dict)

Changes proposed in this PR:

• Updates to the StreamBasedExpressionVisitor
  • pass the index if the child in the node's args list to the beforeChild, acceptChildResult, and afterChild methods
  • Add initializeWalker callback
  • Add support for the context manager API for expression node args data
• Updates to _IndexedComponent_slice
  • make _IndexedComponent_slice a "public" class (rename, dropping the leading _)
  • allow IndexedComponent_slice objects as dict keys
  • update IndexedComponent_slice internal data structures to make them less susceptible to unintended side effects by borrowing concepts from SumExpression views.
• Add a new PyomoModelingObject base class from which both NumericValue and _ComponentBase derive.
  • standardize the definitionof the fundamental is_*_type methods.
• Flush out the _UnindexedComponent_set class with additional methods from the Set API
• Switch the Set API to have derived sets implement _iter_impl instead of directly overloading __iter__.
• Move all template logic (including definition of expression tree nodes and the resolution walkers) into pyomo.core.expr.template_expr (deprecate pyomo.core.base.template_expr)
• Add GetAttrExpression and TempateSumExpression expression nodes
• Add resolve_template(), and templatize_constraint() methods

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[codecov]

Codecov Report

Merging #1433 into master will decrease coverage by 2.37%.
The diff coverage is 91.73%.

@@            Coverage Diff             @@
##           master    #1433      +/-   ##
==========================================
- Coverage   71.63%   69.26%   -2.38%     
==========================================
  Files         590      596       +6     
  Lines       83042    85026    +1984     
==========================================
- Hits        59491    58897     -594     
- Misses      23551    26129    +2578     

Impacted Files	Coverage Δ
pyomo/core/base/param.py	81.67% <ø> (-1.62%)	⬇️
pyomo/core/base/template_expr.py	0.00% <0.00%> (-91.92%)	⬇️
pyomo/core/base/var.py	88.09% <ø> (+0.06%)	⬆️
pyomo/core/expr/logical_expr.py	81.30% <50.00%> (-18.70%)	⬇️
pyomo/core/base/global_set.py	90.00% <75.00%> (-6.67%)	⬇️
pyomo/contrib/satsolver/satsolver.py	83.41% <85.71%> (ø)
pyomo/core/expr/template_expr.py	89.81% <89.81%> (ø)
pyomo/core/pyomoobject.py	92.85% <92.85%> (ø)
pyomo/core/expr/visitor.py	96.54% <96.19%> (-0.77%)	⬇️
pyomo/core/base/indexed_component_slice.py	97.20% <96.70%> (-0.85%)	⬇️
... and 143 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f51f1f3...da86137. Read the comment docs.

[qtothec]

@jsiirola I can't seem to track down documentation on how the template expressions work. Do you happen to know where that lives?

[jsiirola]

@qtothec, alas there is very little formal documentation written up (in part because things are still evolving).

The short version is a "template" expression is just an extended expression tree; that is normal expression tree that also contains GetItemExpression nodes (and with this PR, potentially also GetAttrExpression and TempateSumExpression nodes). You trigger template generation by creating IndexTemplate objects and using them when retrieving indices (e.g., by passing them to rules). IndexTemplate objects change the behavior of IndexedComponent's __getitem__ to return GetItemExpression nodes instead of returning a "Data" object. To avoid expanding internal sum expressions, you need to leverage the templatize_rule method from this PR (it temporarily overrides the sum() builtin and Pyomo Set iteration so that we can detect and get back a TemplateSumExpression node.

[carldlaird]

This looks reasonable to me. I like the idea of a common base class where the is_... methods are documented - it makes the interfaces clearer.

[carldlaird]

I like this concept, not sure about the name. How about just PyomoObject? Can we also move this to a separate file - it is the base of all pyomo objects - components, expression nodes, etc.

[carldlaird]

I find the name of this method confusing - this is a separate conversation, but now that we have a base class PyomoModelObject, we can have a discussion about the names and intent of these methods.

[carldlaird]

Good to me if tests pass.