The central research question this paper addresses is how to jointly optimize the structure and parameters of tensor networks to minimize an arbitrary loss function. The key ideas are:
-
Formulating the novel problem of minimizing a loss function over the space of tensor network structures under a constraint on the number of parameters.
-
Proposing a greedy algorithm to optimize this challenging bi-level (discrete + continuous) optimization problem. The greedy algorithm starts from a rank-1 tensor network and successively identifies the most promising edges to incrementally increase the ranks.
-
Introducing a weight transfer mechanism to efficiently restart the optimization where it left off after each greedy iteration.
-
Allowing the greedy algorithm to explore tensor networks with internal nodes by truncating singular values of matricizations of core tensors.
-
Evaluating the proposed greedy algorithm on tensor decomposition, completion and neural network compression tasks, showing it can find efficient tensor network structures that outperform common baselines like CP and Tucker decompositions.
In summary, this paper introduces a novel formulation and practical algorithm for adaptively learning tensor network structures directly from data/tasks rather than using pre-defined structures like TT or TR. The key innovation is the greedy bi-level optimization approach and weight transfer mechanism to efficiently search the discrete space of tensor network topologies.
The main contribution of this paper is proposing an adaptive greedy algorithm to efficiently learn the structure and parameters of a tensor network for various tensor learning tasks. The key ideas are:
-
Formulating the novel problem of minimizing a loss function over the space of tensor network structures under a constraint on the number of parameters. This results in a bi-level optimization problem.
-
Proposing a greedy algorithm to tackle this problem. It starts from a rank-1 tensor network and iteratively identifies the most promising edge to increment the rank of, transferring weights across greedy iterations.
-
Allowing the greedy algorithm to create internal nodes in the tensor network by splitting cores using truncated SVD.
-
Showcasing the effectiveness of the proposed greedy algorithm on tensor decomposition, completion and neural network compression tasks. The method is able to outperform specialized algorithms by adaptively finding well-suited tensor network structures for each task.
The main novelty is the formulation of the adaptive tensor network learning problem and the simple yet effective greedy algorithm. This provides a general and flexible framework for tensor learning problems rather than relying on pre-defined decomposition models like Tucker or Tensor Train. Experiments demonstrate the potential of the approach to find better tensor network structures than commonly used models.