Who gave John Carmack the reading list?

OpenAI co-founder Ilya Sutskever gave John Carmack a list of around 30 foundational deep-learning papers, telling him that mastering them would cover about 90% of what matters in modern AI.

How many papers are on the list?

The circulating, community-reconstructed list contains roughly 27–30 papers, blog posts, a course and a couple of books, spanning convolutional and recurrent networks, attention and Transformers, scaling laws and information theory.

Can I listen to the papers as audio?

Yes. ListenDock turned the papers into free, AI-narrated audio episodes with key-insight explanations you can follow while walking or commuting.

THE AI READING LIST

Ilya’s 30 papers as audio

The famous list of papers Ilya Sutskever gave John Carmack. He said, “If you really learn all of these, you’ll know 90% of what matters today”.

The audio episodes explain the key insights, giving you a clear overview of every chapter before reading the full PDFs.

22 audio episodes27 works from the listFree to stream

Listen to the episodes

AI-narrated, key-insights chapter by chapter. Free to stream and download.

Scaling Laws for Neural Language Models
Jared Kaplan et al. (OpenAI) · 2020
The empirical power laws linking model size, data and compute to loss — the playbook for scaling up LLMs.
Listen PaperVideo
Relational Recurrent Neural Networks
Adam Santoro et al. (DeepMind) · 2018
Giving memory-based networks a way to let stored memories interact — self-attention inside the recurrent core.
Listen PaperVideo
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Yanping Huang et al. (Google) · 2018
How to split a model too big for one accelerator across many, with micro-batch pipelining and near-linear speedups.
Listen PaperVideo
Attention Is All You Need
Ashish Vaswani et al. (Google) · 2017
The Transformer. Drop recurrence entirely, keep only attention — the architecture behind every modern LLM.
Listen PaperVideo
A Simple Neural Network Module for Relational Reasoning
Adam Santoro et al. (DeepMind) · 2017
A tiny plug-in module (the Relation Network) that gives models superhuman relational reasoning on tasks like CLEVR.
Listen PaperVideo
Neural Message Passing for Quantum Chemistry
Justin Gilmer et al. (Google) · 2017
The framework that unified graph neural networks — passing messages along molecular bonds to predict chemistry.
Listen PaperVideo
Identity Mappings in Deep Residual Networks
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · 2016
The follow-up that explains why residual networks train so cleanly, and how to make the identity path even better.
Listen PaperVideo
Variational Lossy Autoencoder
Xi Chen et al. (OpenAI) · 2016
Combining autoregressive models with VAEs to control exactly what information the latent code keeps or throws away.
Listen PaperVideo
The Unreasonable Effectiveness of Recurrent Neural Networks
Andrej Karpathy · 2015
A character-level RNN trained on Shakespeare, Linux source and math papers — and the jaw-drop of watching it learn.
Listen ArticleVideo
Understanding LSTM Networks
Christopher Olah · 2015
The clearest visual explanation ever written of how an LSTM cell decides what to remember and what to forget.
Listen ArticleVideo
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · 2015
ResNet — the skip connection that let networks go 100+ layers deep and still train.
Listen PaperVideo
Multi-Scale Context Aggregation by Dilated Convolutions
Fisher Yu, Vladlen Koltun · 2015
Dilated convolutions expand the receptive field exponentially without losing resolution — a staple of dense prediction.
Listen PaperVideo
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Dario Amodei et al. (Baidu Research) · 2015
One end-to-end model that learns speech recognition for two very different languages, straight from audio.
Listen PaperVideo
Order Matters: Sequence to Sequence for Sets
Oriol Vinyals, Samy Bengio, Manjunath Kudlur · 2015
What happens when the input or output is a set with no natural order — and why order still matters for learning.
Listen PaperVideo
Pointer Networks
Oriol Vinyals, Meire Fortunato, Navdeep Jaitly · 2015
An architecture that outputs pointers into its own input — solving problems where the vocabulary is the input itself.
Listen PaperVideo
Recurrent Neural Network Regularization
Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals · 2014
How to apply dropout to LSTMs correctly — only on the non-recurrent connections — and finally regularize RNNs.
Listen PaperVideo
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014
The paper that invented attention — letting a translator softly align to the source words it needs.
Listen PaperVideo
Neural Turing Machines
Alex Graves, Greg Wayne, Ivo Danihelka · 2014
A neural network coupled to external memory it can read and write — learning simple algorithms end-to-end.
Listen PaperVideo
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton
Scott Aaronson, Sean Carroll, Lauren Ouellette · 2014
The follow-up experiment: mixing coffee and cream in a cellular automaton to actually measure complexity over time.
Listen PaperVideo
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton · 2012
AlexNet — the paper that won ImageNet by a landslide and kicked off the entire deep-learning revolution.
Listen PaperVideo
The First Law of Complexodynamics
Scott Aaronson · 2011
Why does complexity rise and then fall as a system moves toward equilibrium? A physicist-computer-scientist muses.
Listen ArticleVideo
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights
Geoffrey Hinton, Drew van Camp · 1993
The 1993 paper connecting generalization to compression — the MDL roots of why smaller descriptions win.
Listen PaperVideo

Also on the list

Courses, books and code-heavy pieces that don’t fit an audio episode — read them at the source.

THE STORY

As the story goes, when legendary game programmer John Carmack (Doom, Quake) decided to move into AI, he asked OpenAI co-founder Ilya Sutskever what he should read. Ilya handed him a list of around thirty papers and said that if Carmack really learned all of them, he’d understand 90% of what matters in modern deep learning.

Carmack has confirmed the exchange in interviews, though he’s said the original list was lost. The version circulated today was reconstructed by the community from his and others’ recollections. It’s a remarkably coherent tour from convolutional and recurrent nets, through attention and Transformers, to scaling laws and the information-theoretic roots of learning.

Reconstructions and background: community reading list, Aman’s AI Journal, Ilya’s List.

Have a paper of your own?

Upload any PDF or paste a link and ListenDock turns it into a clear audio episode.

Turn a paper into audio

Ilya’s 30 papers as audio

Listen to the episodes

Scaling Laws for Neural Language Models

Relational Recurrent Neural Networks

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Attention Is All You Need

A Simple Neural Network Module for Relational Reasoning

Neural Message Passing for Quantum Chemistry

Identity Mappings in Deep Residual Networks

Variational Lossy Autoencoder

The Unreasonable Effectiveness of Recurrent Neural Networks

Understanding LSTM Networks

Deep Residual Learning for Image Recognition

Multi-Scale Context Aggregation by Dilated Convolutions

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Order Matters: Sequence to Sequence for Sets

Pointer Networks

Recurrent Neural Network Regularization

Neural Machine Translation by Jointly Learning to Align and Translate

Neural Turing Machines

Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton

ImageNet Classification with Deep Convolutional Neural Networks

The First Law of Complexodynamics

Keeping Neural Networks Simple by Minimizing the Description Length of the Weights

Also on the list

The Annotated Transformer

Kolmogorov Complexity and Algorithmic Randomness

CS231n: Convolutional Neural Networks for Visual Recognition

Machine Super Intelligence

A Tutorial Introduction to the Minimum Description Length Principle

THE STORY

Have a paper of your own?