Deep Learning Paper Implementations

A collection of my personal implementations and mental models of deep learning papers.

Note: The markdown notes in each folder are my rough mental dumps written while reading the papers.

Some notes may be sparse or confusing. Details may occasionally be left out if they are common knowledge.

The notes are subject to change (e.g., being edited during a re-read)

Paper List

Deep Learning Foundations

• Deep Learning Review (2015)
  LeCun, Bengio, Hinton

• Backpropagation (1986)
  Rumelhart, Hinton, Williams

---

Optimization & Regularization

• Xavier Initialization (2010)
  Glorot, Bengio

• Adam Optimizer (2014)
  Kingma & Ba

• Deep Sparse Rectifier Networks (2011)
  Glorot & Bengio

• Dropout (2014)
  Srivastava et al.

---

Convolutional Neural Networks

• Gradient-Based Learning/LeNet5 (1998)
  LeCun, Bengio

• AlexNet (2012)
  Krizhevsky, Sutskever, Hinton

• VGGNet (2014)
  Simonyan, Zisserman

---

Residual Networks

• Kaiming Init (2015)
  He et al.

• Batch Normalization (2015)
  Ioffe, Szegedy

• ResNet (2016)
  He et al.

---

NLP Foundations

• RNN Overview (2019)
  Robin M. Schmidt

• LSTM (1997)
  Hochreiter, Schmidhuber

• Word2Vec Word Estimations (2013)
  Mikolov et al.

• Word2Vec Negative Sampling (2013)
  Mikolov et al.

• RNN Encoder-Decoders (2014)
  Cho, Bengio

• Seq2Seq (2014)
  Sutskever, Vinyals, Le

• Attention (2014)
  Bahdanau, Cho, Bengio

• Efficient Attention-Based NMT (2015)
  Luong et al.

---

Modern NLP

• Transformers (2017)
  Vaswani et al.

• BERT (2018)
  Devlin et al.

• GPT (2018)
  Radford et al.

---

Generative Models

• Variational Autoencoders (2013)
  Kingma et al.

• Generative Adversarial Networks (2014)
  Goodfellow et al.

Folder Organization

/paper_title/notes.md or /paper_title/[model/algorithm].ipynb