Neural Foundations · 2014
Recurrent Neural Network Regularization
Showed that dropout works for recurrent networks if it is applied only to the non-recurrent (layer-to-layer) connections of an LSTM and not to the recurrent state, cutting overfitting on language modelling, translation, speech and captioning without damaging long-range memory.
Editorial record
Plain-language summary
Before this paper, dropout was believed to hurt recurrent networks because noise injected into the recurrent connections corrupts the information the network needs to carry across time steps. Zaremba, Sutskever and Vinyals applied dropout only where information moves between layers at the same time step and left the recurrent path untouched. Large regularised LSTMs then beat smaller unregularised ones across several sequence tasks, and the recipe became the default way to train big recurrent models for the next few years.
Source record
Provenance
- Record ID
- P-670
- Record created
- 2026-09-26
- Last reviewed
- 2026-09-26
- Record version
- 1
- https://arxiv.org/abs/1409.2329
- arXiv:1409.2329
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.