Rough list of my favorite deep learning resources, useful for revisiting topics or for reference. I (Guillaume Chevalier) have got through all of the content listed there, carefully, and built this list.
Here are the all-time Google Trends, from 2004 up to now, April 2017:
You might also want to look at Andrej Karpathy's new post about trends in Machine Learning research.
I believe that Deep learning is the key that can make computers think more like humans and has a lot of potential. Some hard automation tasks can be solved easily with that while this was impossible to achieve earlier with classical algorithms.
Moore's Law about exponential progress rates in computer science hardware is now more affecting GPUs than CPUs because of physical limits on how tiny an atomic transistor can be. We are shifting toward parallel architectures [read more]. Deep learning exploits parallel architectures as such under the hood by using GPUs. On top of that, deep learning algorithms may be applied to Quantum Computing.
The key of intelligence and cognition is a very interesting subject to explore and is not yet well understood.
- Machine Learning by Andrew Ng on Coursera (Good entry-level online class with certificate. Taught by: Andrew Ng, Associate Professor, Stanford University; Chief Scientist, Baidu; Chairman and Co-founder, Coursera)
- Deep Learning by Google (Very good class covering high-level deep learning concepts, it is the logical next step after Andrew Ng's Machine Learning class)
- Machine Learning for Trading by Georgia Tech (Interesting class for acquiring basic knowledge of machine learning applied to trading and some AI and finance concepts)
- Neural networks class - Université de Sherbrooke (Very interesting class available online for free by Hugo Larochelle, yet I have watched a few of those videos)
- How to Create a Mind (The audio version is nice to listen to while commuting)
- Deep Learning - An MIT Press book (Outstanding, yet halfway through the book)
- Some other books I have read (Some may be less related to deep learning but still somehow relevant)
- The Unreasonable Effectiveness of Recurrent Neural Networks (MUST READ post by Andrej Karpathy - this is what motivated me to learn RNNs)
- Neural Networks, Manifolds, and Topology (Fresh look on how neurons map information)
- Understanding LSTM Networks (Walktrough the LSTM cells' inner workings, plus interesting links in conclusion)
- Attention and Augmented Recurrent Neural Networks (Interesting for visual animations)
- Recommending music on Spotify with deep learning (Awesome for doing clustering on audio - post by an intern at Spotify)
- Announcing SyntaxNet: The World’s Most Accurate Parser Goes Open Source (Parsey McParseface's birth)
- Improving Inception and Image Classification in TensorFlow (Very interesting CNN architecture)
- WaveNet: A Generative Model for Raw Audio (Realistic talking machines)
- François Chollet's Twitter (Author of Keras - has interesting Twitter posts)
- TensorFlow's GitHub repository (Most known deep learning framework, both flexible and high-level)
- skflow (TensorFlow wrapper à la scikit-learn)
- Keras (Keras is another very intersting deep learning framework like TensorFlow)
- carpedm20's repositories (Many interesting neural network architectures are implemented by the Korean guy Taehoon Kim - A.K.A. carpedm20)
- carpedm20/NTM-tensorflow (Neural Turing Machine TensorFlow implementation)
- Deep learning for lazybones (Transfer learning tutorial in TensorFlow for vision from high-level embeddings of a pretrained CNN, AlexNet 2012)
- LSTM for Human Activity Recognition (HAR) (Tutorial of mine on using LSTMs on time series for classification)
- Deep residual bidirectional LSTM for HAR (Improvements on the previous project)
- ML / DL repositories I starred (GitHub is full of nice code samples & projects)
Those are resources I have found that seems interesting to develop models onto.
- UCI Machine Learning Repository (TONS of datasets for ML)
- Cornell Movie--Dialogs Corpus (Could be used for a chatbot)
- SQuAD The Stanford Question Answering Dataset (Interesting QA dataset)
- Aligned Hansards of the 36th Parliament of Canada (Aligned text chunks useful for FR-EN machine translation)
- LibriSpeech ASR corpus (Huge free English speech dataset with balanced genders and speakers that seems to be of high quality)
- Neural Networks and Deep Learning (Overview)
- Yes you should understand backprop (Exposing backprop's caveats)
- Artificial Neural Networks: Mathematics of Backpropagation (Picturing backprop mathematically)
- Deep Learning Lecture 12: Recurrent Neural Nets and LSTMs (Unfolding of the RNN graphs is explained properly and expose potential problems about gradient descent algorithms)
- Gradient descent algorithms in a saddle point (Interesting image)
- Gradient descent algorithms in an almost flat landscape (Complementary interesting image to the previous one)
- Gradient Descent (Okay, I already listed Andrew NG's Coursera class above, but this video is quite pertinent)
- Gradient Descent: Intuition (continued)
- Gradient Descent in Practice 2: Learning Rate (continued - adjusting learning rate)
- The Problem of Overfitting (continued - intro to overfitting)
- Diagnosing Bias vs Variance (continued - heuristics for optimizing)
Okay, signal processing might not be directly related to deep learning, but studying it is interesting for have better intuition for developing neural architectures based on signal.
- Window Functions (Interesting Wikipedia page listing known window functions)
- MathBox, Tools for Thought Graphical Algebra and Fourier Analysis (New look on Fourier analysis)
- How to Fold a Julia Fractal (Very interesting animations dealing with complex numbers and wave equations)
- Animate Your Way to Glory, Math and Physics in Motion (Look on convergence methods in physic engines and interaction design)
- Animate Your Way to Glory - Part II, Math and Physics in Motion (Nice animations for rotation and rotation interpolation with Quaternions)
- Filtering signal, plotting the STFT and the Laplace transform (Simple Python demo on signal processing)
- Deep Learning in Neural Networks: An Overview (You_Again's DL summary, mostly about RNNs)
- Bidirectional Recurrent Neural Networks (Better classifications with RNNs!)
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (Two networks in one combined into a seq2seq (sequence to sequence) Encoder–Decoder architecture. RNN Encoder–Decoder with 1000 hidden units. Adadelta optimizer.)
- Sequence to Sequence Learning with Neural Networks (4 stacked LSTM cells of 1000 hidden size with reversed input sentences, and with beam search, on the WMT’14 English to French dataset)
- Exploring the Limits of Language Modeling (Nice recursive models using word level LSTMs on character level CNN using an overkill amount of GPU power)
- Exploring the Depths of Recurrent Neural Networks with Stochastic Residual Learning (Basically, residual connections can be better than stacked RNNs in the presented case of sentiment analysis)
- Neural Machine Translation by Jointly Learning to Align and Translate (Attention mechanism for LSTMs! Mostly, figures and formulas and their explanations revealed to be useful to me.)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (LSTM attention mechanism on CNNs)
- Effective Approaches to Attention-based Neural Machine Translation (Exploring different attention mechanisms)
- Neural Turing Machines (Outstanding for implementing simple neural algorithms with seemingly good generalisation)
- Teaching Machines to Read and Comprehend (A very interesting and creative work about textual question answering, there is something to do with that)
- Pixel Recurrent Neural Networks (Nice for photoshop-like "content aware fill" to fill missing patches in images)
- Adaptive Computation Time for Recurrent Neural Networks (Very interesting, I would love to see how well would it combines to Neural Turing Machines. Interesting interactive visualizations on the subject can be found here.)
- Hybrid computing using a neural network with dynamic external memory (Improvements on differientable memory based on NTM: now differentiable neural computer)
- Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (In 2016: stacked residual LSTMs with attention mechanisms encoder/decoder are the best for NMT - Neural Machine Translation)
- Neural Machine Translation and Sequence-to-sequence Models: A Tutorial (Interesting tutorial, I mostly read part 8 about RNNs with attention as a refresher for NMT.)
- What is the Best Multi-Stage Architecture for Object Recognition? (For the use of "local contrast normalization")
- ImageNet Classification with Deep Convolutional Neural Networks (AlexNet, 2012 ILSVRC, breakthrough of the ReLU activation function)
- Visualizing and Understanding Convolutional Networks (For the "deconvnet layer")
- Very Deep Convolutional Networks for Large-Scale Image Recognition (For the idea of stacking multiple 3x3 conv+ReLU before pooling for a bigger filter size with few parameters, also there is a nice table for "ConvNet Configuration")
- Going Deeper with Convolutions (GoogLeNet: Appearance of "Inception" layers/modules, the idea is of parallelizing conv layers into many mini-conv of different size with "same" padding, concatenated on depth)
- Highway Networks (Highway networks: residual connections)
- Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift (Batch normalization (BN): to normalize a layer's output by also summing over the entire batch, and then performing a linear rescaling and shifting of a certain trainable amount)
- Deep Residual Learning for Image Recognition (Very deep residual layers with batch normalization layers - a.k.a. "how to overfit any vision dataset with too many layers")
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning (For improving GoogLeNet with residual connections)
- WaveNet: a Generative Model for Raw Audio (Epic raw voice/music generation with new architectures based on dilated causal convolutions to capture more audio lenght)
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (3D-GAN for 3D model generation and fun 3D furniture arithmetics from embeddings)
- Deep Learning & Machine Learning (Advanced topics) (Awesome videos)
- Tensor Calculus and the Calculus of Moving Surfaces (Generalize properly how Tensors work, yet watching a few videos helps)
- Signal Processing YouTube Playlist (A YouTube playlist I composed about DFT/FFT, STFT and the Laplace transform - I was mad about my software engineering bachelor not including signal processing classes)
- Computer Science (Yet another YouTube playlist I composed, this time about various CS topics just slightly related)
- Siraj's YouTube Channel (Siraj has some entertaining videos)
- Two Minute Papers' Youtube Channel (Interesting but very shallow overview of some papers, for example about WaveNet or Neural Style Transfer)
- Quora.com (Finest question/answer site)
- Hacker News (Maybe how I discovered ML - Interesting trends appear on that site way before they get to be a big deal)
- DataTau (A hub similar to Hacker News, but specific to data science)
- Naver (Korean search engine - best used with Google Translate, ironically. Surprisingly, sometimes deep learning search results and comprehensible advanced math content shows up more easily there than on Google search)
- Arxiv Sanity Preserver (arXiv browser with TF/IDF features)