flownet2

FlowNet2

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Abstract

The FlowNet demonstrated that optical flow estimation can be cast as a learning problem. However, the state of the art with regard to the quality of the flow has still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. In this paper, we advance the concept of end-to-end learning of optical flow and make it work really well. The large improvements in quality and speed are caused by three major contributions: first, we focus on the training data and show that the schedule of presenting data during training is very important. Second, we develop a stacked architecture that includes warping of the second image with intermediate optical flow. Third, we elaborate on small displacements by introducing a sub-network specializing on small motions. FlowNet 2.0 is only marginally slower than the original FlowNet but decreases the estimation error by more than 50%. It performs on par with state-of-the-art methods, while running at interactive frame rates. Moreover, we present faster variants that allow optical flow computation at up to 140fps with accuracy matching the original FlowNet.

Citation

@inproceedings{ilg2017flownet,
  title={Flownet 2.0: Evolution of optical flow estimation with deep networks},
  author={Ilg, Eddy and Mayer, Nikolaus and Saikia, Tonmoy and Keuper, Margret and Dosovitskiy, Alexey and Brox, Thomas},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={2462--2470},
  year={2017}
}

Results and Models

Models	Training datasets	FlyingChairs	Sintel (training)		KITTI2012 (training)	KITTI2015 (training)		Log	Config	Download
			clean	final	EPE	Fl-all	EPE
FlowNet2CS	FlyingChairs	1.59	-	-	-	-	-	log	Config	Model
FlowNet2CS	Flying Chairs + FlyingThing3d subset	-	1.96	3.69	3.50	28.28%	8.23	log	Config	Model
FlowNet2CSS	FlyingChairs	1.55	-	-	-	-	-	log	Config	Model
FlowNet2CSS	Flying Chairs + FlyingThing3d subset	-	1.85	3.57	3.13	25.76%	7.72	log	Config	Model
FlowNet2CSS-sd	Flying Chairs + FlyingThing3d subset + ChairsSDHom	-	1.81	3.69	2.98	25.66%	7.99	log	Config	Model
FlowNet2	FlyingThing3d subset		1.78	3.31	3.02	25.18%	8.02	log	Config	Model

Models	Training datasets	ChairsSDHom	Log	Config	Download
						Flownet2sd	ChairsSDHom	0.37	log	Config	Model