TensorFlow/MNIST

MNIST Convolutional Neural Network

Concept: Simple, end-to-end, LeNet-5-like convolutional MNIST model example. Meant as a tutorial for simple convolutional models.

Link to code: https://github.com/tensorflow/models/blob/master/tutorials/image/mnist/convolutional.py

Link to tutorial(s): https://www.tensorflow.org/tutorials/

Link to original data set: http://yann.lecun.com/exdb/mnist/

License

# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

Import Statements and Variables

Import statements:

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import argparse
import gzip
import os
import sys
import time

import numpy
from six.moves import urllib
from six.moves import xrange  # pylint: disable=redefined-builtin
import tensorflow as tf

Variable definitions for use in the rest of the model:

SOURCE_URL = 'http://yann.lecun.com/exdb/mnist/'
WORK_DIRECTORY = 'data'
IMAGE_SIZE = 28
NUM_CHANNELS = 1
PIXEL_DEPTH = 255
NUM_LABELS = 10
VALIDATION_SIZE = 5000  # Size of the validation set.
SEED = 66478  # Set to None for random seed.
BATCH_SIZE = 64
NUM_EPOCHS = 10
EVAL_BATCH_SIZE = 64
EVAL_FREQUENCY = 100  # Number of steps between evaluations.
FLAGS = None

Obtaining the Data

Several functions are defined to help obtain the data. First, define the variable types we will use in the model:

def data_type():
  """Return the type of the activations, weights, and placeholder variables."""
  if FLAGS.use_fp16:
    return tf.float16
  else:
    return tf.float32

Now define a function that will attempt to download the data if it does not already exist on disk. This uses urllib to obtain the MNIST files, and TensorFlow's gfile module to interact with the file and filesystem.

def maybe_download(filename):
  """Download the data from Yann's website, unless it's already here."""
  if not tf.gfile.Exists(WORK_DIRECTORY):
    tf.gfile.MakeDirs(WORK_DIRECTORY)
  filepath = os.path.join(WORK_DIRECTORY, filename)
  if not tf.gfile.Exists(filepath):
    filepath, _ = urllib.request.urlretrieve(SOURCE_URL + filename, filepath)
    with tf.gfile.GFile(filepath) as f:
      size = f.size()
    print('Successfully downloaded', filename, size, 'bytes.')
  return filepath

Once the data is downloaded, it must be converted to a format convenient for Tensorflow - in particular, a 4D tensor in which the first index is the image number, the second and third are the width and height, and the fourth dimension is each channel of the image.

These values are then normalized and re-scaled.

def extract_data(filename, num_images):
  """Extract the images into a 4D tensor [image index, y, x, channels].
  Values are rescaled from [0, 255] down to [-0.5, 0.5].
  """
  print('Extracting', filename)
  with gzip.open(filename) as bytestream:
    bytestream.read(16)
    buf = bytestream.read(IMAGE_SIZE * IMAGE_SIZE * num_images * NUM_CHANNELS)
    data = numpy.frombuffer(buf, dtype=numpy.uint8).astype(numpy.float32)
    data = (data - (PIXEL_DEPTH / 2.0)) / PIXEL_DEPTH
    data = data.reshape(num_images, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS)
    return data

The labels - the predictions - must also be put into a format conducive for TensorFlow - a 1D vector:

def extract_labels(filename, num_images):
  """Extract the labels into a vector of int64 label IDs."""
  print('Extracting', filename)
  with gzip.open(filename) as bytestream:
    bytestream.read(8)
    buf = bytestream.read(1 * num_images)
    labels = numpy.frombuffer(buf, dtype=numpy.uint8).astype(numpy.int64)
  return labels

There's also a utility for creating a fake data set.

Error Rate

There is a function defined to compute the error rate. It computes the accuracy first: sums up the number of correctly-labeled digits, divides by the total number of digits, multiplies by 100 to convert to percent. Last, it subtracts the accuracy from 100 to get a percent error.

def error_rate(predictions, labels):
  """Return the error rate based on dense predictions and sparse labels."""
  return 100.0 - (
      100.0 *
      numpy.sum(numpy.argmax(predictions, 1) == labels) /
      predictions.shape[0])

Note that this metric is NOT used for training the convolutional network, it is only used for printing purposes.

Main Method

Get Data

The main method starts by checking if it is in self-test mode (debug mode), in which case, it generates fake data:

  if FLAGS.self_test:
    print('Running self-test.')
    train_data, train_labels = fake_data(256)
    validation_data, validation_labels = fake_data(EVAL_BATCH_SIZE)
    test_data, test_labels = fake_data(EVAL_BATCH_SIZE)
    num_epochs = 1

Otherwise, it extracts data from the downloaded training/testing MNIST images. None of this is using TensorFlow functionality yet.

  else:
    # Get the data.
    train_data_filename = maybe_download('train-images-idx3-ubyte.gz')
    train_labels_filename = maybe_download('train-labels-idx1-ubyte.gz')
    test_data_filename = maybe_download('t10k-images-idx3-ubyte.gz')
    test_labels_filename = maybe_download('t10k-labels-idx1-ubyte.gz')

    # Extract it into numpy arrays.
    train_data = extract_data(train_data_filename, 60000)
    train_labels = extract_labels(train_labels_filename, 60000)
    test_data = extract_data(test_data_filename, 10000)
    test_labels = extract_labels(test_labels_filename, 10000)

    # Generate a validation set.
    validation_data = train_data[:VALIDATION_SIZE, ...]
    validation_labels = train_labels[:VALIDATION_SIZE]
    train_data = train_data[VALIDATION_SIZE:, ...]
    train_labels = train_labels[VALIDATION_SIZE:]
    num_epochs = NUM_EPOCHS
  train_size = train_labels.shape[0]

Variable Definitions

To use the input variables we have on the computational graph that TensorFlow will build, we have to declare those input variables. We do that using a TensorFlow placeholder type.

We also created a placeholder variable in the TensorFlow/Adversarial Crypto class, in the Creation of Message and Key section. There, the placeholder variable was batch size. (However, the input variables in that network were different because we used that somewhat mysterious function, "tf.contrib.framework.arg_scope()", and passed it TensorFlow variables to initialize on each of the three graph we had.)

Here, we are explicitly creating a variable placeholder for the trianing data (inputs), training labels (outputs), and evaluation data. (Remember, the data_type() function was defined above and just returns a float16 or float32 type.)

Again, notice the four dimensions - meaning train_data_node is a placeholder representing a 4D tensor.

  # This is where training samples and labels are fed to the graph.
  # These placeholder nodes will be fed a batch of training data at each
  # training step using the {feed_dict} argument to the Run() call below.
  train_data_node = tf.placeholder(
      data_type(),
      shape=(BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))
  train_labels_node = tf.placeholder(tf.int64, shape=(BATCH_SIZE,))
  eval_data = tf.placeholder(
      data_type(),
      shape=(EVAL_BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))

Next, we create a set of tensors that will hold the weights and biases of the convolutional neural network layers. (When we use high-level neural net APIs like Keras, this is all taken care of for us, but here we do it all manually.)

Variables for Convolutional Layers

First, convolutional layer 1: in this layer we are defining a 5 x 5 filter - meaning the 28 x 28 pixel images in the NIST data set will be convoluted down to a 5 x 5 image set. But the number of feature maps is 32, meaning we're going to convolute the 28 x 28 pixel images down into 32 different 5 x 5 pixel images. The idea is that each of those feature maps will pick out a particular feature about the image topology that is important, and "represent" it, or "allow" that signal to pass through the first layer of the convolutional neural network.

The bias determines the importance of each of those feature maps, so we need the number of biases to match the number of feature maps (32).

  # The variables below hold all the trainable weights. They are passed an
  # initial value which will be assigned when we call:
  # {tf.global_variables_initializer().run()}
  conv1_weights = tf.Variable(
      tf.truncated_normal([5, 5, NUM_CHANNELS, 32],  # 5x5 filter, depth 32.
                          stddev=0.1,
                          seed=SEED, dtype=data_type()))
  conv1_biases = tf.Variable(tf.zeros([32], dtype=data_type()))

We do the same thing with the second convolutional layer. This time there is no number of channels showing up in the size.

  conv2_weights = tf.Variable(tf.truncated_normal(
      [5, 5, 32, 64], stddev=0.1,
      seed=SEED, dtype=data_type()))
  conv2_biases = tf.Variable(tf.constant(0.1, shape=[64], dtype=data_type()))

The truncated_normal variable types refer to how the variables are initialized - they are initialized with values drawn from a (truncated) normal distribution. Truncated simply means that they cannot go above 1 or below 0, which makes it easier for the machine learning algorithm.

Link to truncated_normal documentation: https://www.tensorflow.org/api_docs/python/tf/truncated_normal

Variables for Fully Connected Layers

Next come the two fully connected (FC) layers, which re-assemble the feature maps that were detected in the convolutional layers. The fully connected layer creates 64 neurons for every 4 x 4 square of pixels in the original image, with a depth of 512 neurons each.

(NOTE: This doesn't make sense. There's waaaaaay too many neurons here.)

(It may be that this is the number of inputs, then the number of outputs - as in, there are (size/4)*(size/4)*64 inputs on one side, 512 outputs on the other side.)

  fc1_weights = tf.Variable(  # fully connected, depth 512.
      tf.truncated_normal([IMAGE_SIZE // 4 * IMAGE_SIZE // 4 * 64, 512],
                          stddev=0.1,
                          seed=SEED,
                          dtype=data_type()))

The second (last) fully connected layer is a 512-wide layer that has a depth equal to the number of labels - in this case, 10 corresponding to 10 digits.

(Again, this looks like 512 inputs connecting to NUM_LABELS outputs.)

  fc2_weights = tf.Variable(tf.truncated_normal([512, NUM_LABELS],
                                                stddev=0.1,
                                                seed=SEED,
                                                dtype=data_type()))
  fc2_biases = tf.Variable(tf.constant(
      0.1, shape=[NUM_LABELS], dtype=data_type()))

Neural Network Layers

The next step in the script is to use the variables defined above to construct the neural network layers themselves.

Starting with function header:

  def model(data, train=False):
    """The Model definition."""

We first construct the first convolutional layer of the network:

    # 2D convolution, with 'SAME' padding (i.e. the output feature map has
https://www.tensorflow.org/api_docs/python/tf/truncated_normal    # the same size as the input). Note that {strides} is a 4D array whose
    # shape matches the data layout: [image index, y, x, depth].
    conv = tf.nn.conv2d(data,
                        conv1_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')

We pass in the convolution layer 1 weights variable created above.

After the convolution layer comes a rectifier layer:

    # Bias and rectified linear non-linearity.
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))

then a max pooling layer to apply the biases:

    # Max pooling. The kernel size spec {ksize} also follows the layout of
    # the data. Here we have a pooling window of 2, and a stride of 2.
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')

Then comes the second convolutional layer:

    conv = tf.nn.conv2d(pool,
                        conv2_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')

Now the pool layer is a 4D tensor. We have to reshape it into a 2D matrix so that its shape matches the shape of the fully connected layer. Just take the 2nd, 3rd, and 4th dimensions and stretch them all out into a single dimension.

    # Reshape the feature map cuboid into a 2D matrix to feed it to the
    # fully connected layers.
    pool_shape = pool.get_shape().as_list()
    reshape = tf.reshape(
        pool,
        [pool_shape[0], pool_shape[1] * pool_shape[2] * pool_shape[3]])

The final step is the last fully connected layer, which will output 10 signals to 10 neurons, each corresponding to our 10 classes/digits.

    # Fully connected layer. Note that the '+' operation automatically
    # broadcasts the biases.
    hidden = tf.nn.relu(tf.matmul(reshape, fc1_weights) + fc1_biases)
    # Add a 50% dropout during training only. Dropout also scales
    # activations such that no rescaling is needed at evaluation time.
    if train:
      hidden = tf.nn.dropout(hidden, 0.5, seed=SEED)

    return tf.matmul(hidden, fc2_weights) + fc2_biases

Why Separate Variable Definitions and Model Creation

The reason this network separates the variable creation, which initializes TensorFlow variables representing weights and biases, and the model creation, which is contained in a function that assembles a neural net layer by layer, is because the model will be used in two different contexts: training, and prediction.

The model construction always uses the same variables, no matter what the model is going to be used for. By separating variable creation from model construction, we allow the training and evaluation steps to use the same network architecture and the same weights.