ImageNet Classification with Deep Convolutional Neural Networks

Host: We are going to dive right into one of the most famous breakthroughs in artificial intelligence, a paper that completely changed how computers see the world. Written by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, it details a massive leap forward in image classification. Guest: I've heard this was a major milestone. What exactly were they trying to get the computer to recognize? Host: They took on the ImageNet contest, training a deep convolutional neural network to categorize 1.2 million high-resolution images into a thousand different classes. And by doing so, they absolutely shattered the previous state-of-the-art records. Guest: The text mentions they achieved top-1 and top-5 error rates of 37.5 percent and 17 percent. What does "top-5" mean in this context? Host: A top-5 error rate just means the correct answer wasn't even in the model's top five highest-confidence guesses. Getting that down to 17 percent in 2010, and later 15.3 percent in 2012, was completely unprecedented compared to the runner-up's 26.2 percent. Guest: That is a massive gap in performance. What did their neural network have that the others didn't? Host: Unprecedented scale, made possible by raw computing power. The network had 60 million parameters spread across five convolutional layers and three fully-connected layers, all trained using highly efficient GPUs. Guest: With a model containing 60 million parameters, wouldn't it just memorize the training images instead of actually learning the underlying patterns? Host: That is a major risk known as overfitting, but they successfully tackled it using a newly developed method called "dropout." They also used specific "non-saturating" neurons to make this gigantic model train much faster, proving that deep neural networks were finally practical.

Host: To kick things off, let's look at how computers actually learn to identify objects in images. Historically, the recipe for better object recognition has been pretty straightforward: get more data, build more powerful models, and find better ways to stop those models from just memorizing the training data. Guest: That makes sense, but what actually counted as "a lot of data" in the early days? Host: Not that long ago, a big dataset meant tens of thousands of images, like the famous MNIST dataset used for recognizing handwritten digits. Models got really good at simple tasks with those—even approaching human performance—but real-world objects are just too variable for small datasets to cover everything. Guest: So they needed way more images to handle the messiness of the real world? Host: Exactly, which brings us to the breakthrough of massive datasets like ImageNet, packing over 15 million labeled, high-resolution images across 22,000 categories. But if you want to learn from millions of images, you need a learning model with a massive capacity to absorb all that information. Guest: That sounds like a heavy lift. What kind of model can actually handle that scale? Host: This is where Convolutional Neural Networks, or CNNs, come into play. Even with 15 million images, object recognition is so incredibly complex that the model needs some built-in "prior knowledge" about how images work to fill in the gaps. Guest: What do you mean by prior knowledge about how images work? Host: CNNs assume two main things: that visual patterns can happen anywhere in an image, and that pixels close to each other are strongly connected. Because they make these smart, naturally correct assumptions, they need far fewer connections than a standard neural network. Guest: So they're basically more efficient because they aren't starting completely from scratch when looking at a picture? Host: Spot on. They are much easier to train because they have fewer parameters to calculate, making them the perfect tool for tackling these massive new datasets.

Host: Let's explore what drove the researchers to build this model and the specific breakthroughs they brought to the table. For a long time, Convolutional Neural Networks, or CNNs, were simply too computationally expensive to apply to high-resolution images at a large scale. Guest: What finally changed to make them practical for those bigger images? Host: The arrival of powerful GPUs, paired with massive new labeled datasets like ImageNet. This combination allowed the authors to train what was then one of the largest CNNs ever, achieving by far the best results reported in the 2010 and 2012 ImageNet competitions. Guest: Just how big was this record-breaking network? Host: It consists of five convolutional layers and three fully-connected layers. The researchers found this exact depth was absolutely crucial, noting that removing even a single convolutional layer resulted in worse performance. Guest: That is interesting, especially since those layers probably did not hold the majority of the network's parameters. Host: Exactly, each convolutional layer contains no more than one percent of the model's parameters, yet removing any of them hurt the results. But with a model this deep, overfitting became a massive problem, even with 1.2 million training images. Guest: How did they manage to get around the overfitting? Host: They had to write highly-optimized GPU code and introduce a number of new, unusual features just to prevent that overfitting and keep training times manageable. They actually made all of those operations publicly available. Guest: How long did it actually take to train something like this back then? Host: Between five and six straight days using two three-gigabyte GTX 580 GPUs. The network's size was strictly limited by that GPU memory and the researchers' own patience. Guest: It sounds like they pushed the hardware of that era to its absolute limit. Host: They really did. In fact, all their experiments led them to one final conclusion: these results could be improved even further simply by waiting for faster GPUs and bigger datasets to become available in the future.

Host: To understand how this visual recognition model learned to identify objects, we first need to examine the massive collection of pictures it trained on. They used a dataset called ImageNet, which contains over 15 million high-resolution images across roughly 22,000 categories, all labeled by humans online. Guest: That is a staggering amount of data, but did they actually feed all 15 million images into their model? Host: They mostly focused on a specific subset created for an annual competition called the ImageNet Large-Scale Visual Recognition Challenge, or ILSVRC. That subset trims the data down to exactly 1,000 categories, leaving about 1.2 million images for training and another 200,000 for validation and testing. Guest: Having a standardized benchmark like that makes sense for comparing results. How did they measure if the model was actually guessing the right categories? Host: They tracked two main error rates known as top-1 and top-5. Top-1 is just how often the model's absolute best guess is wrong, while the top-5 error rate measures how often the correct label is completely missing from the model's top five guesses. Guest: Measuring the top five makes sense, since an image might have multiple objects going on in the background. What about the images themselves, did they all start out the exact same size? Host: No, they were originally all different resolutions, but this neural network requires a constant, uniform input size. They solved this by down-sampling every single image to a fixed resolution of 256 by 256 pixels. Guest: How do you force a rectangular image into a square without completely warping how the objects look? Host: They first scaled the image so the shorter side was exactly 256 pixels long, and then they simply cropped out the center 256-by-256 square. After that, the only other pre-processing they did was subtracting the average pixel value of the whole training set from each pixel. Guest: So there was no edge-detection or complicated filtering? They just centered those raw red, green, and blue pixel values before feeding them in? Host: Exactly, they kept it incredibly simple. They trained the network directly on those raw RGB pixel values.

Host: Let's look under the hood at the actual structure of this neural network. At a high level, the authors built a system with exactly eight "learned" layers. Guest: By "learned" layers, do they mean the parts of the network that actually adapt and train on the data? Host: Exactly, these are the layers where the weights are updated during the learning process. Out of those eight layers, the first five are convolutional, and the final three are fully-connected. Guest: I know convolutional layers usually scan an image for specific visual patterns, but what do those fully-connected ones do at the end? Host: They take all the high-level patterns found by the convolutional layers and combine them to make the final classification. So it's essentially five pattern-finding steps followed by three decision-making steps. Guest: That makes sense as a sequence, but is there something specific that makes this eight-layer setup so special? Host: There is, and the authors point out several highly unusual features about how this network is designed. What's helpful is that as they list out these novel features, they explicitly rank them by importance. Guest: That's great, so they put the absolute biggest architectural breakthrough right at the top of their list? Host: Precisely. They lead with their most critical design choice, making sure we grasp the biggest improvements first before digging into the rest of the structure.

Host: Let's look at the specific mathematical function that decides whether a neuron fires or not. Traditionally, neural networks modeled a neuron's output using equations like the sigmoid or hyperbolic tangent, but these were notoriously slow to train. Guest: What exactly made those traditional functions so slow? Host: They are what we call saturating nonlinearities. This means that as the input values get very large or very small, the output flattens out, which makes the learning signal shrink to almost zero. Guest: So the network basically stalls out in those flat areas. What did they use instead to fix that? Host: They switched to a non-saturating function called a Rectified Linear Unit, or ReLU for short. The math is incredibly simple, it just takes the maximum of zero and the input value. Guest: Wait, so if the input is negative, the output is zero, and if it is positive, it just passes the number right through? Host: Exactly, and because it never flattens out on the positive side, the gradient descent process does not slow down. A deep convolutional network using ReLUs actually reaches its target error rate several times faster than one using the older functions. Guest: Did anyone else realize you could swap out these neuron functions before this? Host: Some earlier researchers had experimented with different functions to prevent networks from overfitting on smaller datasets. But the authors here noticed a completely different benefit, which was a massive acceleration in the ability to fit the training set. Guest: And I imagine that kind of raw speed is exactly what you need when you are building really big models. Host: That is the whole key here. Without the faster learning provided by ReLUs, experimenting with networks of this unprecedented size would have simply taken too long to be practical.

Host: To process such a massive amount of data, we need to look at the physical hardware making it all possible. Back when this architecture was built, a standard GTX 580 GPU only had 3 gigabytes of memory, which simply wasn't enough to hold the giant neural network they needed. Guest: Three gigabytes really isn't much by today's standards! So if the network was too big for one GPU, how did they manage to train it? Host: They solved this by splitting the network down the middle and spreading it across two GPUs. What made this efficient was that these GPUs could read and write directly to each other's memory, completely bypassing the main computer's slower memory. Guest: That sounds incredibly fast. Did the two halves of the network just constantly share data back and forth as they processed the images? Host: Actually, they deliberately restricted the communication to avoid bottlenecks. They used a clever parallelization scheme where the GPUs only shared data during very specific layers of the network. Guest: How did that play out in practice? Host: For example, the neurons in layer three would pull data from all the available neurons in layer two across both GPUs. But in layer four, they would only look at the layer three neurons located on their own isolated GPU. Guest: I see, they intentionally limited the cross-talk to keep the calculation speed high while still sharing some context. Did splitting the network like this improve the overall results? Host: It absolutely did, dropping their top error rates by over a full percent compared to running a half-sized network on just one GPU. And surprisingly, despite being a much larger model, training on the two GPUs actually took slightly less time than the smaller single-GPU version.

Host: Let's look at a clever biological trick the authors used to make the network's features pop. Even though ReLU activation functions don't strictly require normalization to prevent them from maxing out, the team found that adding a specific local normalization step helped the model generalize better to new data. Guest: If ReLUs don't need it to keep learning, what exactly is this normalization step doing? Host: It mimics something found in real biological brains called lateral inhibition. When one neuron fires very strongly, it suppresses the activity of its neighbors, creating a competition where only the most dominant features survive. Guest: Oh, so the network actively dampens weaker signals around a strong one? How is that calculated mathematically? Host: After the ReLU function outputs a positive value, that activity is divided by a scaling factor. This factor is based on the sum of the squared activations from a few "adjacent" kernel maps. Guest: When you say adjacent, do you mean neighboring pixels in the physical image? Host: Great question, but no; it means adjacent feature channels looking at the exact same spatial pixel. So if one filter detects a strong vertical edge at a specific spot, it suppresses other filters looking for different things at that exact same spot. Guest: That makes sense, creating a sort of local contrast effect. Did this extra math actually improve the model's performance? Host: It did, though they preferred the term "brightness normalization" since they only scale the values without subtracting the average. This competition reduced their top error rates by over a full percent, proving it was well worth the effort.

Host: Let's explore how the network compresses and summarizes the visual information it processes. Pooling layers are used in these networks to summarize small, neighboring groups of neurons into a single output. Guest: So if I picture a grid of pixels, how does the network decide which groups to summarize together? Host: Think of a square summarizing window of a certain size, let's call it *z*, that jumps across the image by a specific step size, called *s*. Traditionally, the step size equals the window size, so the windows sit side-by-side without overlapping. Guest: Like cutting a pan of brownies into perfectly separate squares. What happens if the step size is smaller than the window? Host: Then you get overlapping pooling, which is exactly the approach the authors took here. They set their step size to 2 pixels, but used a slightly larger 3-by-3 window. Guest: Oh, so the windows share some of the same pixels as they slide across. Does that overlap actually improve the model's performance? Host: It does, as it dropped their top-1 error rate by 0.4% and their top-5 error rate by 0.3%. They proved this by comparing it to a traditional non-overlapping setup that produces the exact same output dimensions. Guest: That is a solid accuracy boost just from tweaking how the windows move. Did it change anything else about the training process? Host: Yes, they observed a very helpful secondary benefit. They found that models using this overlapping pooling approach were slightly more resistant to overfitting the training data.

Host: Let's step back and look at the complete blueprint of this neural network. It's a massive structure containing eight layers with weights—five convolutional layers that process the image, followed by three fully-connected layers. Guest: So the convolutional layers act like filters for the visual patterns, and the fully-connected ones make the final decision? Host: Exactly, ending in a final layer that outputs a probability distribution across all 1000 possible image categories. But because this network is so uniquely huge, they actually had to divide the computation across two separate GPUs. Guest: How does splitting the network across two GPUs affect how the layers talk to each other? Host: To minimize communication delays, certain layers—specifically the second, fourth, and fifth convolutional layers—only connect to data processed on their own GPU. However, the third convolutional layer and the fully-connected layers are allowed to cross over and connect to everything. Guest: That makes sense as a way to avoid a hardware bottleneck. Where do the other techniques, like the ReLU activation function, fit into this layout? Host: They apply that ReLU function to the output of every single convolutional and fully-connected layer to keep training fast. They also sprinkle in max-pooling and normalization layers, mostly just after the early convolutional layers. Guest: It sounds like a dense obstacle course for the data, starting from the raw image all the way to that final category prediction. Host: It really is, progressively filtering a raw 224 by 224 pixel image through hundreds of kernels and layers with over 4,000 neurons, all to maximize the probability of guessing the correct label.

Host: Let's look at one of the biggest hurdles in training massive AI models, which is preventing them from just memorizing the training data. The neural network we are examining is incredibly large, coming in at 60 million parameters. Guest: That sounds like a massive number of dials to tune. Does having that much capacity cause problems during training? Host: It definitely does, because it creates a severe risk of overfitting. The network has so much flexibility that it can easily memorize the specific training images instead of learning useful, general visual patterns. Guest: But doesn't the training dataset have a thousand different categories to learn from? Host: It does, and those thousand categories provide what the authors call 10 bits of constraint per image. But compared to 60 million parameters, that tiny amount of feedback per example just isn't enough to keep the model on track. Guest: Wait, what exactly does 10 bits of constraint mean in this context? Host: Think of it like a game of 20 Questions. Since there are a thousand possible image categories, it takes about 10 yes-or-no questions to narrow down the correct answer, because two to the power of ten is a thousand and twenty-four. Guest: Ah, I see. So each image only forces the model to learn 10 bits of information, while the model has 60 million parameters it can freely adjust. Host: Exactly, and that huge mathematical imbalance gives the network way too much freedom to cheat and overfit. To solve this, the authors had to rely on two primary techniques to keep those parameters in check.

Host: To stop the neural network from just memorizing the training images, the creators used a clever trick to artificially expand their dataset. This is called data augmentation, and the authors used two distinct methods that were incredibly efficient. Guest: How do you artificially expand a dataset without actually going out and collecting more photos? Host: You do it by slightly altering the images you already have. For their first method, they took the original 256 by 256 pixel images and extracted smaller random patches of 224 by 224 pixels, and then flipped those patches horizontally. Guest: Oh, so one image suddenly becomes hundreds of slightly shifted or mirrored versions. But wouldn't storing all those new images take up a massive amount of hard drive space? Host: It definitely would, which is why they generated these transformations on the fly. The CPU created these altered images in Python code while the GPU was busy training on the previous batch, making the whole process essentially computationally free. Guest: That is a brilliant way to avoid a bottleneck. Since the network learns on all these random patches, how does it make a final decision when looking at a completely new image? Host: At test time, they extract ten specific patches from the new image, specifically the four corners, the center, and the horizontal reflections of all five. The network computes the probabilities using its softmax layer for all ten patches, and averages those predictions to make its final call. Guest: Very thorough. You mentioned a second augmentation method too, what was that one? Host: The second method involved altering the actual colors. They used Principal Component Analysis, or PCA, to identify the main color variations across the dataset, and then randomly shifted the RGB pixel intensities for each training image. Guest: So an image might look a bit more red or a bit more blue each time the network sees it? Host: Exactly. This mimics natural changes in lighting and illumination. It taught the network that an object's identity doesn't change just because the lighting does, which alone reduced their top-1 error rate by over one percent.

Host: Let's look at a clever trick the researchers used to keep their massive neural network from just memorizing the training data. Usually, combining the predictions of many different models is a great way to reduce errors, but training multiple huge networks takes days and is way too expensive. Guest: So how do they get the accuracy boost of multiple models without the massive computing cost? Host: They used a highly efficient technique called "dropout." During training, every time an input is processed, they randomly turn off—or drop out—the output of each hidden neuron with a 50% probability. Guest: Wait, they just randomly shut down half of the network's neurons during training? Doesn't that disrupt the learning process? Host: It actually improves it by preventing what they call complex "co-adaptations." Because a neuron can never guarantee its neighbors will be active, it can't rely on them and is forced to learn robust, independent features. Guest: Ah, so because the active neurons change on every single pass, the system is essentially sampling a brand-new, smaller network each time? Host: Exactly, though all those random architectures share the same underlying weights. Then, at test time, they turn all the neurons back on to make their final predictions. Guest: If all the neurons are suddenly on at test time, wouldn't the combined signal be way too strong compared to training? Host: Spot on, which is why they multiply the outputs by 0.5 at test time. This halves the signal to keep it balanced, serving as a great approximation for averaging the predictions of all those smaller dropout networks. Guest: That is a brilliant workaround. Did they use this dropping out everywhere, and were there any downsides? Host: They applied it specifically to their first two fully-connected layers, which successfully stopped the network from severely overfitting. The main trade-off was that dropout roughly doubled the number of iterations needed for the network to fully train.

Host: Let's look under the hood at exactly how this neural network was actually trained. The researchers used a standard optimization method called stochastic gradient descent, feeding the model 128 images at a time, but they added a few specific tweaks to make it learn effectively. Guest: What kind of tweaks did they need to make? Host: One major tweak was how they used "weight decay," which is a tiny penalty applied to the network's parameters. Usually, weight decay is just a regularizer to prevent the model from memorizing the data, but here, they found it actually reduced the primary training error. Guest: Oh, so it actively helped the model learn the training data better, rather than just helping it generalize. Did they have to do anything special when first setting up the network? Host: Yes, initialization was crucial to getting the training off the ground. They set the initial weights to very small random numbers, but for several layers, they set the starting neuron biases to exactly 1 instead of 0. Guest: Why specifically use the number 1 for those starting biases? Host: It was a clever trick to give their ReLU activation functions a jumpstart. ReLUs only activate if their input is positive, so starting with a positive bias of 1 ensured the neurons were actively passing signals right from the very first training step. Guest: That makes sense, so they aren't stuck at zero. How did they control the learning speed as the model progressed? Host: They started with a learning rate of 0.01 and kept a very close eye on the error rate. Whenever the network stopped improving, they manually intervened and divided that learning rate by 10. Guest: So they forced the model to take smaller, more careful steps as it got closer to the final solution. How long did the whole training process take? Host: They cycled through the entire 1.2 million image dataset about 90 times. Even using two top-of-the-line GPUs for that era, the entire process took five to six days of non-stop computing.

Host: It is time to look at how this network actually performed when put to the test. When the authors evaluated their model on the 2010 ImageNet dataset, they measured their success using what are called top-1 and top-5 error rates. Guest: Before we get into the actual numbers, what is the difference between a top-1 and a top-5 error? Host: A top-1 error means the model's absolute first guess was wrong, while a top-5 error means the correct answer was not even in the model's top five guesses. On the 2010 data, their network achieved a top-1 error of 37.5 percent and a top-5 error of just 17 percent. Guest: That sounds pretty good, but how does it compare to what other researchers were doing at the time? Host: The previous best top-1 error was 47.1 percent, which relied on older machine learning techniques combining multiple sparse-coding models. So this single convolutional neural network dropped the error rate by almost ten full percentage points, which is a massive leap. Guest: That is a huge breakthrough. Did they see the same kind of success in the big 2012 competition? Host: They did, and the gap was even more dramatic. A single version of their network hit an 18.2 percent top-5 error rate, but they managed to push that all the way down to 15.3 percent by averaging the predictions of seven different variations of their network. Guest: So they basically created a panel of network variations and had them vote on the image to cancel out individual mistakes? Host: Exactly, that technique is called an ensemble. Some of those networks were even fine-tuned after being pre-trained on an even larger dataset of 15 million images. Guest: What did the runner-up get in that 2012 contest? Host: The second-best entry, which used older feature-extraction methods, was way behind with a 26.2 percent error rate. Winning by such a gigantic margin is what firmly proved to the world that these deep neural networks were the new standard.

Host: Let's step away from the raw math for a moment and actually look at the visual patterns this network figured out. When the researchers examined the initial layers of their trained model, they saw it had naturally learned to detect some very distinct features. Guest: What kind of features are we talking about? Does it recognize full objects right away? Host: Not in these early stages; instead, it learns basic filters called convolutional kernels. These act like tiny feature detectors that specifically look for sharp edges, angled lines, or simple colored blobs. Guest: So it's breaking the image down into fundamental building blocks. Does splitting the network across two graphics cards change how it learns those blocks? Host: It does, and it leads to a really fascinating division of labor. Because the two GPUs were restricted from fully talking to each other, they actually ended up specializing their tasks. Guest: Specialized how? Did they divide up the work by color or something? Host: You nailed it. The first GPU became almost entirely color-agnostic, focusing heavily on shapes and black-and-white patterns, while the second GPU specialized in detecting specific colors. Guest: That's a brilliant way for the system to organize itself! Was that just a fluke of how they started that specific training run? Host: Amazingly, no. This exact split between shape detection and color detection happens every single time they train the model, completely independent of the random starting weights. Guest: It's like the network naturally discovers the most efficient way to see. How did this specific setup perform on its tests? Host: As a baseline, if they test standard images without using a special trick of averaging ten different cropped views together, the error rates sit at exactly 39% and 18.3%.

Host: Let's explore how we can visually judge what this neural network has actually learned. The researchers checked the network's top five predictions on various test images and found it could easily spot off-center objects, like a tiny mite in the corner of a photo. Guest: That is impressive, but what happens when the network gets the main prediction wrong? Are its mistakes at least logical? Host: Yes, most of its backup guesses make total sense. For example, if it looks at a picture of a leopard, the other labels it considers plausible are just different types of cats. Guest: So it clearly understands the broader category. Is looking at those final text labels the only way to test its visual knowledge? Host: Not at all, since we can also look under the hood at the final hidden layer, which outputs a unique signature of 4,096 numbers for any given image. If two different images produce very similar number patterns in that layer, the network considers them closely related. Guest: Does closely related just mean the pictures have the exact same colors and edges? Host: That is the real breakthrough here, because the answer is no. Two images can look totally different at the raw pixel level—like elephants standing in completely different poses—but this layer still matches them up because it grasps the actual subject. Guest: It sounds like that would make for an amazing, smart reverse image search tool. Host: It absolutely would. Comparing those massive lists of numbers is currently a bit slow, but the authors note you could easily compress them into short binary codes to build a highly efficient, meaning-based image search.

Host: Let's step back and reflect on what these results actually mean for the field. The big takeaway is that a large, deep convolutional neural network can shatter performance records on a difficult dataset using purely supervised learning. Guest: It's amazing that just feeding it labeled data was enough. Was the specific design of the network the main key to that success? Host: Absolutely, and specifically its depth. The authors found that if they removed even a single middle convolutional layer, the network's top accuracy dropped by about two percent. Guest: Wow, so every single layer is pulling its weight. Did they use any other tricks, like having the model learn from unlabeled data first? Host: They actually skipped unsupervised pre-training entirely to keep their experiments simple. However, they suspect it will be really helpful in the future, especially if they build even larger networks without getting more labeled data to match. Guest: So for now, just making the network bigger and training it longer did the job. Does that mean we're getting close to human-level vision? Host: Not quite, as we are still many orders of magnitude away from matching the human brain's visual pathways. But these results do give researchers a clear direction for what to tackle next. Guest: What is that next frontier for these deep neural networks? Host: Applying them to video sequences instead of static images. The way objects move over time provides incredibly helpful context that is completely missing when you just look at a still photograph.

References. [1] R.M. Bell and Y. Koren. Lessons from the netflix prize challenge. ACM SIGKDD Explorations Newsletter, 9(2):75-79, 2007. [2] A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010. www.imagenet.org/challenges. 2010. [3] L. Breiman. Random forests. Machine learning, 45(1):5-32, 2001. [4] D. Cireşan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. Arxiv preprint arXiv:1202.2745, 2012. [5] D.C. Cireşan, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber. High-performance neural networks for visual object classification. Arxiv preprint arXiv:1102.0183, 2011. [6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009. [7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012, 2012. URL http://www.image-net.org/challenges/LSVRC/2012/. [8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understand-ing, 106(1):59–70, 2007. [9] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, Cali-fornia Institute of Technology, 2007. URL http://authors.library.caltech.edu/7694. [10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. Improving neural net-works by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012. [11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In International Conference on Computer Vision, pages 2146–2153. IEEE, 2009. [12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009. [13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 2010. [14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image retrieval. In ESANN, 2011. [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Hand-written digit recognition with a back-propagation network. In Advances in neural information processing systems, 1990. [16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II-97. IEEE, 2004. [17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253-256. IEEE, 2010. [18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for scalable unsuper-vised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609-616. ACM, 2009. [19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classifi-cation: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer Vision, Florence, Italy, October 2012. [20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning, 2010. [21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computa-tional biology, 4(1):e27, 2008. [22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS computational biology, 5(11):e1000579, 2009. [23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157–173, 2008. [24] J. Sánchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1665–1672. IEEE, 2011. [25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003. [26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Con-volutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22(2):511-538, 2010.