Origins & Computability · 1993
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Treated learning as compression: by adding Gaussian noise to the weights and minimising the total description length of the weights plus the data misfit, one obtains a principled regulariser that trades model complexity against fit, an early form of variational Bayes for neural networks.
Editorial record
Plain-language summary
The paper asks how much information a set of trained weights really carries. Using the minimum description length principle, it penalises networks whose weights would take many bits to transmit, and shows that adding noise to the weights lets the sender and receiver share a cheaper code. The construction is now recognised as the origin of the bits-back argument and of variational weight-noise methods, and it is the paper most often cited when the compression view of intelligence is traced to neural networks.
Source record
Provenance
- Record ID
- P-671
- Record created
- 2026-09-26
- Last reviewed
- 2026-09-26
- Record version
- 1
Citation caveat: Citation metadata is approximate and marked unverified in the source dataset.