ResNet (2015): residual connections to go deeper
He et al. introduced residual networks that enabled training hundreds of layers and won ILSVRC 2015.
2012–2016
Deep Learning
Deep nets, ImageNet, and the rebirth of large-scale machine learning.
GANs (2014): Goodfellow and the generator–discriminator game
Ian Goodfellow et al. introduced generative adversarial networks, opening an era of realistic image synthesis.
Word2Vec (2013): the algebra of words
Tomas Mikolov and Google colleagues released Word2Vec, popularizing dense embeddings and king–man+woman≈queen analogies.
AlexNet (2012): deep learning’s big bang in vision
Krizhevsky, Sutskever, and Hinton won ILSVRC 2012 with a deep CNN on GPUs, crashing classification error.
Deep pretraining before AlexNet: the table is set
Before 2012, Hinton, Bengio, and LeCun pushed greedy pretraining, autoencoders, and evidence that deep nets could learn.
From HMMs to DNNs in speech recognition
In the early 2010s, deep nets replaced key parts of HMM-GMM systems and sharply cut speech error rates.