You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At present, any given mathematical expression in aegir is formed as a (potentially very large) tree vis-a-vis symbolic differentiation. This leads to many challenges, but doesn't have to be a blocker if implemented correctly despite what many people claim [1]. The problem is, we are currently missing one crucial feature to avoid redundant computation.
Caching is fundamentally important for efficient computation in symbolic approaches to autodiff. The idea is to avoid computing the output every single operator if we can assert that the result has already been computed. This facilitates the transformation of expression trees into expression forests which are known to be equivalent to the DAGs used in more traditional implementations [1].
[1] Laue, Sören. "On the equivalence of forward mode automatic differentiation and symbolic differentiation." arXiv preprint arXiv:1904.02990 (2019).
To-Do List
Add trait for mutating entires in Context types.
Implement an operator, say Cached, that skips redundant computation.
Add Hash implementations for operator trees to facilitate identification.
Extend Context support for Hash-based access to cached results.
Facilitate propagation of Cached operator through Differentiable::adjoint.
The text was updated successfully, but these errors were encountered:
Progress is being made here: see commits 9ad7fcc through 380678c. Major roadblock will be in how to propagate Cached through Differentiable::adjoint. Similar challenge to #5 in which rewrites to the tree are not permitted when each operator is a unique type rather than an algebraic data type.
Summary
At present, any given mathematical expression in
aegir
is formed as a (potentially very large) tree vis-a-vis symbolic differentiation. This leads to many challenges, but doesn't have to be a blocker if implemented correctly despite what many people claim [1]. The problem is, we are currently missing one crucial feature to avoid redundant computation.Caching is fundamentally important for efficient computation in symbolic approaches to autodiff. The idea is to avoid computing the output every single operator if we can assert that the result has already been computed. This facilitates the transformation of expression trees into expression forests which are known to be equivalent to the DAGs used in more traditional implementations [1].
[1] Laue, Sören. "On the equivalence of forward mode automatic differentiation and symbolic differentiation." arXiv preprint arXiv:1904.02990 (2019).
To-Do List
Context
types.Cached
, that skips redundant computation.Hash
-based access to cached results.Cached
operator throughDifferentiable::adjoint
.The text was updated successfully, but these errors were encountered: