Neural Foundations · 2016
Identity Mappings in Deep Residual Networks
Analysed signal propagation in residual networks and showed that keeping the skip connection a clean identity, with normalisation and activation applied before each weight layer (pre-activation), lets networks over a thousand layers deep train easily and generalise better.
Editorial record
Plain-language summary
The original ResNet added a shortcut around each block, but the shortcut still passed through an activation. This follow-up examines what happens to gradients as they flow through many blocks and finds that any operation on the shortcut path compounds across depth and hurts training. Moving normalisation and activation inside the block, ahead of the convolutions, keeps the shortcut pure and makes extremely deep networks practical. The same pre-normalisation pattern is what modern transformers use in their residual streams.
Source record
Provenance
- Record ID
- P-676
- Record created
- 2026-09-26
- Last reviewed
- 2026-09-26
- Record version
- 1
- https://arxiv.org/abs/1603.05027
- arXiv:1603.05027
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.