Colendi AIResearch
Menu

Research archive

The Intelligence Papers

A navigable record of the ideas, artifacts, and institutions that formed the modern intelligence stack—reviewed as evidence, not arranged as a reading list.

294reviewed records

22research domains

48 records

0011986Neural Foundations

Foundational paper

Backpropagation (Learning representations by back-propagating errors)

David Rumelhart, Geoffrey Hinton, Ronald Williams

The paper applies the chain rule to compute, for every weight in a layered network, how much it contributed to the output error, then adjusts weights in the direction that reduces that error. This gave hidden units a way to learn useful internal representations instead of being hand-designed. It made training networks with one or more hidden layers a routine gradient-descent procedure and became the standard method for supervised neural network learning.

UnknownDifficulty 5/10Verified
Read record Primary source
0022014Neural Foundations

Peer reviewed

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau attention)

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

Instead of forcing the encoder to compress a whole sentence into one vector, this model learns an alignment: for each output word the decoder scores and softly selects relevant source positions to build a context vector. This kept translation quality from degrading as sentences got longer and produced interpretable soft alignments between source and target words. The mechanism became the core building block that the Transformer later generalized.

UnknownDifficulty 5/10Verified
Read record Primary source
0032015Neural Foundations

Peer reviewed

Deep Residual Learning for Image Recognition (ResNet)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

The authors added shortcut connections that add a layer block's input to its output, so each block only has to learn the difference from identity. This made gradients flow through very deep stacks and allowed networks of 50 to 152 layers to train and improve rather than degrade, winning ImageNet 2015. Residual blocks became a default component in most later deep vision and language architectures.

UnknownDifficulty 4/10Verified
Read record Primary source
0042014Neural Foundations

Peer reviewed

Adam: A Method for Stochastic Optimization

Diederik Kingma, Jimmy Ba

Adam keeps exponentially decaying averages of past gradients and squared gradients, applies bias correction to those estimates, and uses them to scale each parameter's step. This gives well-behaved step sizes across parameters with different gradient magnitudes and works with little tuning across many problems. Its combination of low memory cost and robust default hyperparameters made it the default optimizer for training deep networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0052014Neural Foundations

Peer reviewed

Sequence to Sequence Learning with Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc Le

The paper used one multilayer LSTM to read an entire source sentence into a single vector and a second LSTM to generate the target sentence from that vector. A key practical trick was reversing the order of source words, which shortened the average distance between corresponding input and output tokens and made optimization easier, pushing WMT'14 English-to-French BLEU to competitive levels. It showed that a purely neural, general-purpose sequence model could rival phrase-based statistical MT and became the template for later encoder-decoder systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0062016Neural Foundations

Preprint

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey Hinton

The method computes each layer's normalization statistics over the units within one example, so behavior is identical during training and testing and does not depend on batch composition. This made it effective for RNNs and sequence models where batch normalization worked poorly, stabilizing and speeding up training. Layer normalization became standard in Transformer architectures.

UnknownDifficulty 4/10Verified
Read record Primary source
0072015Neural Foundations

Peer reviewed

Neural Machine Translation of Rare Words with Subword Units (BPE)

Rico Sennrich, Barry Haddow, Alexandra Birch

BPE starts from characters and iteratively merges the most frequent adjacent symbol pairs to build a vocabulary of subword units, so any word can be encoded as a sequence of known pieces. This let translation models handle rare words, compounds, and morphology without a huge word vocabulary or a separate back-off mechanism. Subword segmentation via BPE became standard tokenization for most later NMT and language models.

UnknownDifficulty 4/10Verified
Read record Primary source
0082012Neural Foundations

Peer reviewed

AlexNet: ImageNet Classification with Deep CNNs

Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton

AlexNet is a deep convolutional network with roughly 60 million parameters trained on over a million labeled images across 1,000 categories, using ReLU activations for faster training, dropout to limit overfitting, data augmentation, and two GPUs to fit the model in memory. Combining these let it cut the ImageNet top-5 error far below the previous best. Its result shifted computer vision toward deep learning and drove wide adoption of GPU-trained convolutional networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0091997Neural Foundations

Peer reviewed

Long Short-Term Memory (LSTM)

Sepp Hochreiter, Jürgen Schmidhuber, Felix Gers

Each LSTM unit keeps a memory cell whose value is protected by gates that control when new information is written in, retained, or read out, so the cell can hold information across many time steps. Because the cell passes its state forward with near-constant behavior when the gates allow, gradients do not shrink away over long sequences the way they do in plain recurrent networks. This let recurrent networks learn dependencies spanning hundreds of steps and made them effective for sequence tasks such as speech and language modeling.

UnknownDifficulty 5/10Verified
Read record Primary source
0102009Neural Foundations

Peer reviewed

ImageNet: A Large-Scale Hierarchical Image Database

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei

ImageNet organised millions of human-labelled images into thousands of categories following the WordNet hierarchy, assembled through large-scale crowdsourcing. It supplied the data and the annual ILSVRC challenge that made it possible to train and fairly compare very large vision models. AlexNets 2012 win on this benchmark is widely taken as the moment deep learning became the dominant approach in vision.

UnknownDifficulty 3/10Verified
Read record Primary source
0112003Neural Foundations

Peer reviewed

A Neural Probabilistic Language Model

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin

Instead of treating words as discrete symbols, the model maps each word to a real-valued vector and feeds the vectors of the preceding words into a neural network that outputs a probability distribution over the next word. Words used in similar contexts end up with similar vectors, so the model can assign reasonable probability to word sequences it never saw in training. This showed that a single network could learn word features and the probability function together, and set the template later used by neural word embeddings and language models.

UnknownDifficulty 5/10Verified
Read record Primary source
0122013Neural Foundations

Preprint

Efficient Estimation of Word Representations (Word2Vec)

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean

The models drop the hidden nonlinear layers of earlier neural language models and instead train a shallow network to predict a word from its surrounding context (CBOW) or the surrounding context from a word (skip-gram). This makes training fast enough to run on billions of words, producing vectors where semantic and syntactic relationships appear as consistent offsets (the vector arithmetic 'king minus man plus woman' lands near 'queen'). It made high-quality word embeddings cheap to compute and widely reusable as input features for other NLP systems.

UnknownDifficulty 4/10Verified
Read record Primary source
0132014Neural Foundations

Peer reviewed

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov

During training, each unit is kept with some probability and dropped otherwise, so the network cannot rely on any specific combination of units and must learn features that are useful on their own. At test time all units are used with their outputs scaled to match, which approximates averaging over the many thinned networks seen during training. This reduced overfitting on a range of vision and speech tasks and became a standard technique for training large networks with limited data.

UnknownDifficulty 3/10Verified
Read record Primary source
0142010Neural Foundations

Peer reviewed

Understanding the Difficulty of Training Deep Nets (Xavier/He init)

Xavier Glorot, Yoshua Bengio, Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

The paper measures how the variance of activations and back-propagated gradients changes from layer to layer under different setups, showing that poor scaling makes deep networks with saturating activations (like sigmoid) hard to train. It proposes initializing weights with a variance set from the number of incoming and outgoing connections so that signals neither vanish nor blow up as they pass through the layers. This initialization let deeper networks converge more reliably and became a default starting point for many architectures.

UnknownDifficulty 5/10Verified
Read record Primary source
0152010Neural Foundations

Peer reviewed

Rectified Linear Units (ReLU)

Vinod Nair, Geoffrey Hinton, Xavier Glorot, Antoine Bordes, Yoshua Bengio

A ReLU passes positive inputs through unchanged and outputs zero for negative inputs, so its gradient is a constant one wherever the unit is active instead of shrinking toward zero the way sigmoid and tanh gradients do. This keeps gradients flowing through many layers and produces sparse activations that are cheap to compute. Using ReLUs let deep networks train faster and reach lower error, and it became the default activation for most feedforward and convolutional networks.

UnknownDifficulty 3/10Verified
Read record Primary source
0162015Neural Foundations

Peer reviewed

Batch Normalization

Sergey Ioffe, Christian Szegedy

For each mini-batch, the method normalizes a layer's pre-activation values to zero mean and unit variance, then applies a learned scale and shift so the layer can still represent what it needs. This keeps the distribution of inputs to each layer more stable as the weights below it change, which allows much higher learning rates and less sensitivity to initialization. It sped up training substantially, added a mild regularizing effect, and became a standard component of deep convolutional networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0171998Neural Foundations

Peer reviewed

LeNet: Gradient-Based Learning Applied to Document Recognition

Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner

The network stacks convolutional layers that apply the same small learned filters across the image with pooling layers that reduce resolution, so it detects local patterns regardless of position while keeping the parameter count low through weight sharing. Trained on labeled digit images, it learns its own feature detectors instead of relying on manually designed ones. It achieved strong accuracy on handwritten digit recognition and was deployed to read amounts on bank checks, demonstrating a practical gradient-trained vision system.

UnknownDifficulty 4/10Verified
Read record Primary source
0182006Neural Foundations

Peer reviewed

Deep Belief Nets / Reducing Dimensionality

Geoffrey Hinton, Simon Osindero, Yee-Whye Teh, Ruslan Salakhutdinov

The method builds a deep belief network by training each layer in turn to model the activations of the layer below, giving the full network a sensible starting point before ordinary gradient fine-tuning. Applied to an autoencoder, this two-stage approach learns a compact code that reconstructs the input better than linear methods like PCA. The work provided a practical way to train deep networks before better initialization and activation functions existed, and helped renew interest in deep learning.

UnknownDifficulty 5/10Verified
Read record Primary source
0191988Neural Foundations

Peer reviewed

Learning to Predict by the Methods of Temporal Differences

Richard S. Sutton

Sutton showed how to learn multi-step predictions incrementally: instead of comparing a prediction only to the eventual outcome, TD updates each prediction using the difference between successive predictions along the way. This let learning happen online, step by step, and made better use of experience than earlier supervised methods. TD became the computational core of most later reinforcement learning, including value estimation in games and control.

UnknownDifficulty 6/10Verified
Read record Primary source
0201992Neural Foundations

Peer reviewed

Q-learning

Christopher J. C. H. Watkins, Peter Dayan

Watkins and Dayan defined a rule that updates an estimate of the long-run reward for each state-action pair (the Q-value) from observed transitions and rewards. Because updates use the best available next action rather than the action actually taken, the agent can learn the optimal policy while still exploring. They gave a convergence proof, giving reinforcement learning a model-free method with mathematical guarantees that underpins much later work.

UnknownDifficulty 5/10Verified
Read record Primary source
0211992Neural Foundations

Peer reviewed

Simple Statistical Gradient-Following Algorithms (REINFORCE)

Ronald J. Williams

Williams derived how to nudge the weights of a stochastic network so that actions leading to higher reward become more likely, using only the reward signal and the probability of the chosen action. The estimate of the gradient is unbiased and needs no model of the environment. This policy-gradient approach became the basis for later actor-critic and deep policy-optimization methods used to train agents and language models.

UnknownDifficulty 6/10Verified
Read record Primary source
0222015Neural Foundations

Peer reviewed

Human-level Control through Deep RL (DQN)

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al.

Mnih and colleagues trained a network to estimate action values from screen images, using two stabilizing tricks: an experience replay buffer that reuses and decorrelates past transitions, and a periodically updated target network. The same model and hyperparameters learned to play 49 Atari games from pixels and reward alone. It showed that deep networks could serve as reliable function approximators in reinforcement learning, launching the deep RL field.

UnknownDifficulty 6/10Verified
Read record Primary source
0232016Neural Foundations

Peer reviewed

Mastering the Game of Go (AlphaGo/AlphaZero)

David Silver, Aja Huang, Chris J. Maddison, Julian Schrittwieser, Thomas Hubert, Demis Hassabis

AlphaGo paired a policy network (which moves to consider) and a value network (who is winning) with tree search to defeat top human Go players. AlphaZero simplified this into a single network trained purely by playing against itself, guided by search, and generalized to chess and shogi. The work showed that self-play plus search could reach superhuman strength without human examples or handcrafted evaluation.

UnknownDifficulty 6/10Verified
Read record Primary source
0242013Neural Foundations

Peer reviewed

Auto-Encoding Variational Bayes (VAE)

Diederik P. Kingma, Max Welling

Kingma and Welling framed generation as learning a latent variable model and approximating its intractable posterior with a neural encoder. Their key move, the reparameterization trick, rewrote random sampling so gradients could flow through it, allowing the encoder and decoder to be trained together by ordinary backpropagation on a single objective (the evidence lower bound). This gave a scalable way to learn continuous latent representations and generate new data, widely used for images and representation learning.

UnknownDifficulty 6/10Verified
Read record Primary source
0252014Neural Foundations

Peer reviewed

Generative Adversarial Networks (GAN)

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al.

Goodfellow and colleagues set up a game between two networks: a generator that turns noise into samples and a discriminator that tries to tell real data from generated data. Training the two in competition drives the generator to match the real data distribution without needing an explicit likelihood. This adversarial framework produced sharp, realistic image generation and spawned a large family of follow-on models for synthesis and translation.

UnknownDifficulty 6/10Verified
Read record Primary source
0262014Neural Foundations

Peer reviewed

GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher Manning

GloVe builds a matrix counting how often each word appears near each other word across the whole corpus, then trains word vectors so that the dot product of two vectors approximates the logarithm of their co-occurrence count, weighting frequent pairs so common words do not dominate. This uses aggregate statistics directly rather than scanning context windows one at a time like word2vec. The resulting vectors placed related words near each other and encoded analogies as vector offsets, and became a widely used pretrained input for later NLP models.

UnknownDifficulty 4/10Verified
Read record Primary source
0272015Neural Foundations

Peer reviewed

Effective Approaches to Attention-based NMT (Luong attention)

Minh-Thang Luong, Hieu Pham, Christopher Manning

The paper lets a translation decoder, at each output step, compute a weighted average of encoder hidden states where the weights come from a similarity score between the current decoder state and each source state, then combines that context with the decoder state to predict the next word. It compared scoring functions (dot product, general, concat) and introduced a 'local' attention that only looks at a small window of source positions to save computation. These variants improved translation quality and clarified the design space, and the dot-product scoring it favored became standard in later attention and Transformer work.

UnknownDifficulty 5/10Verified
Read record Primary source
0281989Neural Foundations

Peer reviewed

Universal Approximation Theorem

George Cybenko, Kurt Hornik

Cybenko (1989) showed this for sigmoid-type activations and Hornik (1991) generalized it to broad classes of activation functions, proving that finite sums of scaled, shifted activations are dense in the space of continuous functions. The result is an existence guarantee: for any target accuracy some network with enough hidden units achieves it. It does not say how many units are needed or how to find the weights by training, but it removed the theoretical objection that networks might be fundamentally incapable of representing complex functions.

UnknownDifficulty 5/10Verified
Read record Primary source
0291990Neural Foundations

Peer reviewed

Finding Structure in Time (Elman RNN)

Jeffrey Elman

The network copies its hidden-layer state at each step into a 'context' layer that is fed back in alongside the next input, so the hidden units learn representations that depend on prior elements of the sequence. Trained on tasks like predicting the next word or discovering word boundaries in a stream, it developed internal representations that reflected grammatical categories and structure without being told them. This showed that temporal structure could be handled by recurrence rather than by explicit time-delay windows, and the architecture became a foundation for later recurrent networks.

UnknownDifficulty 4/10Verified
Read record Primary source
0301994Neural Foundations

Peer reviewed

Learning Long-Term Dependencies is Difficult (vanishing gradients)

Sepp Hochreiter, Yoshua Bengio, Patrice Simard, Paolo Frasconi

The paper showed mathematically that for a recurrent network to store information robustly over time its dynamics must contract, but contraction causes error gradients passed backward through many time steps to shrink toward zero, so distant past inputs contribute almost nothing to weight updates. When the dynamics instead expand, gradients blow up. This trade-off means standard gradient descent cannot reliably link causes and effects separated by many steps. The analysis motivated later remedies such as gating (LSTM, GRU), gradient clipping, and better initialization.

UnknownDifficulty 5/10Verified
Read record Primary source
0312014Neural Foundations

Peer reviewed

Gated Recurrent Unit (GRU) / RNN Encoder-Decoder

Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, Yoshua Bengio

One recurrent network (the encoder) reads a source sentence and compresses it into a fixed-length vector, and a second recurrent network (the decoder) generates the target sentence from that vector, trained jointly to maximize the probability of the correct output. To make the recurrence trainable over longer spans the paper introduced the GRU, which uses update and reset gates to control how much past state is kept or overwritten, achieving LSTM-like behavior with a simpler structure. The encoder-decoder setup became the template for neural machine translation and sequence-to-sequence learning, and its learned phrase representations improved a statistical translation system.

UnknownDifficulty 4/10Verified
Read record Primary source
0322011Neural Foundations

Peer reviewed

Natural Language Processing (almost) from Scratch

Ronan Collobert, Jason Weston, Léon Bottou, et al.

The system maps words to learned vector embeddings, processes sentences with a convolutional network, and is trained jointly across tasks like part-of-speech tagging, chunking, named-entity recognition, and semantic role labeling. Crucially it pretrains word embeddings on large unlabeled corpora using a ranking objective, then reuses them, so the model discovers useful features rather than relying on task-specific hand-crafted inputs. It reached competitive accuracy across tasks with one architecture, demonstrating the transfer value of unsupervised pretrained word representations that later work built on.

UnknownDifficulty 5/10Verified
Read record Primary source
0331993Neural Foundations

Peer reviewed

The Mathematics of Statistical MT (IBM alignment models)

Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer

The paper frames translation as finding the target sentence most probable given the source, decomposed via Bayes' rule into a translation model and a language model, and introduces a hidden alignment variable saying which source word generated each target word. The five IBM models add successive detail (word translation probabilities, then absolute position, fertility of how many words a source word produces, and distortion of position), and their parameters are estimated from unaligned sentence pairs using the EM algorithm. These alignment models became the statistical foundation of machine translation for roughly two decades and the basis for later phrase-based systems.

UnknownDifficulty 6/10Verified
Read record Primary source
0341995Neural Foundations

Peer reviewed

TD-Gammon

Gerald Tesauro

The program uses a neural network to estimate the expected outcome of a board position and trains it with TD(lambda), which adjusts predictions so that each position's estimated value moves toward the value of positions that follow it during play. By playing hundreds of thousands of games against itself it improved without expert-labeled positions or a hand-tuned evaluation function. It reached strong tournament-level play and even influenced human opening theory, providing an early demonstration that reinforcement learning combined with function approximation could master a large-state-space game.

UnknownDifficulty 5/10Verified
Read record Primary source
0352014Neural Foundations

Preprint

Neural Turing Machines / Memory Networks

Alex Graves, Greg Wayne, Ivo Danihelka, Jason Weston, Sumit Chopra, Antoine Bordes

A controller network emits read and write operations addressed to memory locations using soft, differentiable attention weights (based on content similarity and location shifting), so gradients can flow through the memory access and the model learns how to use its memory end-to-end. This separates computation from an addressable storage bank, letting the network learn procedures like copying, sorting, and associative recall and generalize them to longer inputs than seen in training. It, alongside Memory Networks, showed that networks augmented with external memory could learn algorithm-like behavior, influencing later memory-augmented and attention-based models.

UnknownDifficulty 6/10Verified
Read record Primary source
0362012Neural Foundations

Peer reviewed

Deep Neural Networks for Acoustic Modeling in Speech

Geoffrey Hinton, Li Deng, Dong Yu, et al.

In a speech recognizer the acoustic model estimates how likely each short audio frame corresponds to each sub-phonetic state; the paper describes replacing the standard Gaussian-mixture estimator with a deep neural network that takes several frames of audio features and outputs those state probabilities, often pretrained layer by layer and then fine-tuned. Reported jointly by teams at Microsoft, Google, IBM, and the University of Toronto, the deep networks lowered word error rates on multiple large-vocabulary benchmarks. The shared results gave the field consistent evidence that deep acoustic models were a general improvement, and they were adopted in commercial speech systems.

UnknownDifficulty 5/10Verified
Read record Primary source
0372017Neural Foundations

Peer reviewed

Neural Message Passing for Quantum Chemistry

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, George E. Dahl

Several groups had proposed different ways to run neural networks over molecules and other graphs. This paper shows they are all instances of one scheme: each node collects messages from its neighbours, updates its state, and a readout function pools node states into a prediction. Framing it this way made the design space explicit and gave a strong, general recipe. MPNNs became the reference abstraction for graph learning and remain the standard vocabulary for describing graph neural networks.

UnknownDifficulty 5/10Verified
Read record Primary source
0382016Neural Foundations

Peer reviewed

Identity Mappings in Deep Residual Networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

The original ResNet added a shortcut around each block, but the shortcut still passed through an activation. This follow-up examines what happens to gradients as they flow through many blocks and finds that any operation on the shortcut path compounds across depth and hurts training. Moving normalisation and activation inside the block, ahead of the convolutions, keeps the shortcut pure and makes extremely deep networks practical. The same pre-normalisation pattern is what modern transformers use in their residual streams.

Fully open sourceDifficulty 4/10Verified
Read record Primary source
0392015Neural Foundations

Peer reviewed

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Adam Coates, Greg Diamos, Erich Elsen, Awni Hannun, Andrew Ng, Sanjeev Satheesh, Shubho Sengupta, Zhenyao Zhu

Speech recognisers used to be assembled from separate acoustic, pronunciation and language models built by specialists. Deep Speech 2 trains a single network from spectrograms to characters and shows that the same recipe works for two very different languages. Much of the paper is about engineering at scale: synchronous training across many GPUs, custom kernels, and a batching scheme that made the model cheap to serve. It was an early, influential demonstration that data, compute and end-to-end learning could displace expert-built systems.

UnknownDifficulty 4/10Verified
Read record Primary source
0401997Neural Foundations

Peer reviewed

Bidirectional Recurrent Neural Networks

Mike Schuster, Kuldip Paliwal

The architecture processes a sequence left-to-right with one recurrent network and right-to-left with a second, then merges the two hidden states at each position before making a prediction. This gives every output access to the entire input sequence rather than only the elements seen so far. Because it requires the whole sequence up front it suits offline labeling tasks like speech and text tagging, and the bidirectional design later combined with LSTMs to become a standard component for sequence labeling.

UnknownDifficulty 4/10Verified
Read record Primary source
0412006Neural Foundations

Peer reviewed

Connectionist Temporal Classification (CTC)

Alex Graves, Santiago Fernández, Faustino Gomez, Jürgen Schmidhuber

CTC adds a special 'blank' symbol and defines the probability of a target label sequence as the sum over all frame-level paths that collapse to it (by removing blanks and merging repeats), computed efficiently with a forward-backward dynamic-programming algorithm. This lets the network output a probability distribution over labels at every time step and be trained end-to-end on unsegmented pairs of input and target sequences. It removed the requirement for hand-aligned data and enabled direct recurrent-network sequence transcription in speech and handwriting recognition.

UnknownDifficulty 5/10Verified
Read record Primary source
0422014Neural Foundations

Preprint

Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals

Before this paper, dropout was believed to hurt recurrent networks because noise injected into the recurrent connections corrupts the information the network needs to carry across time steps. Zaremba, Sutskever and Vinyals applied dropout only where information moves between layers at the same time step and left the recurrent path untouched. Large regularised LSTMs then beat smaller unregularised ones across several sequence tasks, and the recipe became the default way to train big recurrent models for the next few years.

Fully open sourceDifficulty 3/10Verified
Read record Primary source
0432015Neural Foundations

Peer reviewed

Pointer Networks

Oriol Vinyals, Meire Fortunato, Navdeep Jaitly

Standard sequence-to-sequence models choose each output from a fixed vocabulary, which fails when the answer is a set of indices into the input. Pointer networks use the attention distribution directly as that choice: at each step the decoder points at one input element. This small change let neural networks tackle combinatorial geometry problems and later inspired copy mechanisms in summarisation and question answering, where the model copies spans from its input.

UnknownDifficulty 4/10Verified
Read record Primary source
0442015Neural Foundations

Peer reviewed

Multi-Scale Context Aggregation by Dilated Convolutions

Fisher Yu, Vladlen Koltun

Image classification networks shrink their feature maps with pooling, which loses the detail needed for tasks like semantic segmentation. Dilated convolutions skip input positions inside the kernel so each layer sees a wider area without pooling or extra parameters. Stacking layers with increasing dilation gives a large receptive field at full resolution. The idea was adopted widely, from later segmentation networks to WaveNet's audio generation.

Fully open sourceDifficulty 3/10Verified
Read record Primary source
0452015Neural Foundations

Peer reviewed

Order Matters: Sequence to sequence for sets

Oriol Vinyals, Samy Bengio, Manjunath Kudlur

Many problems involve sets rather than sequences, yet models were trained by picking an arbitrary order. The authors show experimentally that this arbitrary choice can swing results, for example when sorting numbers or parsing. They introduce a set encoder that reads all elements, processes them with attention over a memory for several steps, and writes outputs with a pointer network. The paper is an early, concrete argument that inductive biases about ordering and permutation invariance matter, a theme that reappears in graph networks and in positional encodings for transformers.

UnknownDifficulty 4/10Verified
Read record Primary source
0462017Neural Foundations

Peer reviewed

A simple neural network module for relational reasoning

Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Timothy Lillicrap

Questions like 'is the ball the same colour as the cube?' require comparing objects, which standard convolutional and recurrent networks do poorly. A Relation Network takes all pairs of object representations, runs each pair through the same small network, and adds the outputs. This explicit all-pairs computation is a simple inductive bias for relational reasoning, and it is a close cousin of the pairwise interactions that self-attention later made general and scalable.

UnknownDifficulty 3/10Verified
Read record Primary source
0472016Neural Foundations

Peer reviewed

Variational Lossy Autoencoder

Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, Pieter Abbeel

When a VAE's decoder is expressive enough to model the data on its own, the encoder has no incentive to put information into the latent variable. The authors show this follows from the cost of encoding the latent, then turn the effect into a design tool: by restricting what the decoder can see locally, they force the latent to carry global information such as shape while discarding texture. Combined with autoregressive flows for the prior, the model also set likelihood records at the time. The paper is a clear example of information-theoretic reasoning guiding architecture design.

UnknownDifficulty 6/10Verified
Read record Primary source
0482018Neural Foundations

Peer reviewed

Relational recurrent neural networks

Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Théophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, Timothy Lillicrap

LSTMs and memory-augmented networks store information but give it few ways to interact. The Relational Memory Core keeps a small matrix of memory slots and lets them attend to one another and to the current input using the same attention mechanism as transformers. This lets the model, for example, track which entities were involved in earlier events. Gains on program evaluation, reinforcement learning and language modelling showed that attention-based interaction inside recurrent memory was worthwhile, shortly before transformers made recurrence itself optional.

UnknownDifficulty 5/10Verified
Read record Primary source