Implicit acceleration by overparameterization

This project is a simple demonstration of the implicit acceleration by overparameterization of neural networks. We reimplement some of the experiments and push some ideas further from the following paper:

Arora, S., Cohen, N., & Hazan, E. (2018, July). On the optimization of deep networks: Implicit acceleration by overparameterization. In International Conference on Machine Learning (pp. 244-253). PMLR.

The report is available here. Slides are available here.

Installation

This project is written in Python 3.11. To install the required packages, run the following command:

python -m venv .venv
source .venv/bin/activate # or .venv/Scripts/activate on Windows
pip install -r requirements.txt