We recommend to firstly start with one or two Machine Learning classes, because Deep Learning is related to them and it will be easier for you.
Table of contents
- Andrew Ng's deeplearning.ai class
- fast.ai
- Stanford cs231n
- Richard Socher's Deep Learning and Natural Language Processing
- Hugo Larochelle's course
- Nando de Freita's class on Machine/Deep Learning
- Hinton's Neural Network Machine Learning
- Oxfords Deep NLP
If you don't know which DL library to start with, we suggest you to check Ranking Popular DL Libraries from 10/2017 where the winner is Tensorflow, followed by Keras and Caffe.
| Library | Rank | Overall | Github | Stack Overflow | Google Results |
|---|---|---|---|---|---|
| Tensorflow | 1 | 10.8676777173 | 4.25282914794 | 4.371905768 | 2.24294280139 |
| Keras | 2 | 1.92768682345 | 0.613405340454 | 0.830444013135 | 0.483837469861 |
| Caffe | 3 | 1.85536658344 | 1.00172325244 | 0.301598379669 | 0.552044951334 |
All 23 libraries and their usage (based on Data Incubator ranking)
| Library | Rank | Overall | Github | Stack Overflow | Google Results |
|---|---|---|---|---|---|
| Tensorflow | 1 | 10.8676777173 | 4.25282914794 | 4.371905768 | 2.24294280139 |
| Keras | 2 | 1.92768682345 | 0.613405340454 | 0.830444013135 | 0.483837469861 |
| Caffe | 3 | 1.85536658344 | 1.00172325244 | 0.301598379669 | 0.552044951334 |
| Theano | 4 | 0.757142065184 | -0.156657475854 | 0.361637072631 | 0.552162468406 |
| Pytorch | 5 | 0.481418742361 | -0.198079135346 | -0.30225967424 | 0.981757551946 |
| Sonnet | 6 | 0.427865682184 | -0.326074511957 | -0.361634296039 | 1.11557449018 |
| Mxnet | 7 | 0.0987996914674 | 0.121327235453 | -0.306328604959 | 0.283801060973 |
| Torch | 8 | 0.00559731666893 | -0.153332101969 | -0.00824393023136 | 0.167173348869 |
| Cntk | 9 | -0.0205203098963 | 0.0965088202554 | -0.282173869559 | 0.165144739407 |
| Dlib | 10 | -0.599823512154 | -0.39578194316 | -0.223382454956 | 0.0193408859617 |
| Caffe2 | 11 | -0.671062928351 | -0.274071118159 | -0.359648165565 | -0.0373436446266 |
| Chainer | 12 | -0.70151841136 | -0.400397905813 | -0.234603397931 | -0.0665171076164 |
| Paddlepaddle | 13 | -0.833003782881 | -0.267123408237 | -0.366884083295 | -0.198996291348 |
| Deeplearning4j | 14 | -0.893319117931 | -0.0575131634759 | -0.321347169592 | -0.514458784863 |
| Lasagne | 15 | -1.10606125475 | -0.381150749139 | -0.287853956451 | -0.437056549158 |
| Bigdl | 16 | -1.12821350465 | -0.458674544538 | -0.367555905286 | -0.301983054824 |
| Dynet | 17 | -1.25088837288 | -0.465671394541 | -0.367690269684 | -0.417526708658 |
| Apache Singa | 18 | -1.33963459336 | -0.502246959001 | -0.367824634082 | -0.469563000276 |
| Nvidia Digits | 19 | -1.39248467556 | -0.407011549848 | -0.346078273813 | -0.639394851898 |
| Matconvnet | 20 | -1.41327975079 | -0.487125591647 | -0.346308395531 | -0.579845763615 |
| Tflearn | 21 | -1.44982650865 | -0.226089464016 | -0.282710110548 | -0.941026934086 |
| Nervana Neon | 22 | -1.65176202195 | -0.39497574163 | -0.366989720498 | -0.889796559818 |
| Opennn | 23 | -1.97015587693 | -0.53381703821 | -0.366068321175 | -1.07027051754 |
- A guide to DL is very handy because all concepts are ranked by its difficulty
- Awesome DL
- Deep Learning Book, Ian Goodfellow, Yoshua Bengio and Aaron Courville
- Neural Networks and Deep Learning, Michael Nielsen
- Geoffrey Hinton
- Yoshua Bengio
- Yann Lecun
- Andrew Ng
- Andrew Karpathay
- Richard Socher
- Hugo Larochelle
- Nando de Freitas
Based on Awesome DL Papers, counted since 2012 (last 5 years).
- Visualizing and understanding convolutional networks, M. Zeiler and R. Fergus, 2014
- Decaf: A deep convolutional activation feature for generic visual recognition, J. Donahue et al., 2014
- CNN features off-the-Shelf: An astounding baseline for recognition, A. Razavian et al., 2014
- How transferable are features in deep neural networks?, J. Yosinski et al., 2014
- Learning and transferring mid-Level image representations using convolutional neural networks, M. Oquab et al., 2014
- Distilling the knowledge in a neural network, G. Hinton et al., 2015
- Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, A. Nguyen et al., 2015
- Adam: A method for stochastic optimization, D. Kingma and J. Ba, 2014
- Dropout: A simple way to prevent neural networks from overfitting, N. Srivastava et al., 2014
- Batch normalization: Accelerating deep network training by reducing internal covariate shift, S. Loffe and C. Szegedy, 2015
- Improving neural networks by preventing co-adaptation of feature detectors, G. Hinton et al., 2012
- Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, K. He et al., 2015
- Random search for hyper-parameter optimization, J. Bergstra and Y. Bengio, 2012
- Training very deep networks, R. Srivastava et al., 2015
- Generative adversarial nets, I. Goodfellow et al., 2014
- Building high-level features using large scale unsupervised learning, Q. Le et al., 2013
- Auto-encoding variational Bayes, D. Kingma and M. Welling, 2013
- Unsupervised representation learning with deep convolutional generative adversarial networks, A. Radford et al., 2015
- DRAW: A recurrent neural network for image generation, K. Gregor et al.
- Improved techniques for training GANs, T. Salimans et al., 2016
- Pixel recurrent neural networks, A. Oord et al., 2016
- Gradient ∇ (Nabla)
- Backpropagation
- Sigmoid σ
- Rectifier (Rectified Linear Units or ReLU)
- Tanh
- Gated recurrent unit (GRU)
- Long short-term memory
- Softmax
- Regularization
- Batch Normalization
- Objective Functions
- F1 score
- Cross entropy
- Dynamic Routing Between Capsules, Sara Sabour, Nicholas Frosst, Geoffrey E Hinton, 2017
- Matrix capsules with EM routing, Blind Submission of CapsNets with Reviews, 2017
- Aurélien Géron's tutorial, recommended by Geffrey Hinton himself
- Understanding Hinton’s Capsule Networks