Relational recurrent neural networks

Editorial record

Plain-language summary

LSTMs and memory-augmented networks store information but give it few ways to interact. The Relational Memory Core keeps a small matrix of memory slots and lets them attend to one another and to the current input using the same attention mechanism as transformers. This lets the model, for example, track which entities were involved in earlier events. Gains on program evaluation, reinforcement learning and language modelling showed that attention-based interaction inside recurrent memory was worthwhile, shortly before transformers made recurrence itself optional.