Keras

Neural network package in Python.

See https://github.com/charlesreid1/circe - specifically the examples using the NIST handwriting digit classification data set. This shows how to utilize Keras to train a neural network to perform dimensionality reduction, and further explores the manifold that the neural network identified for the different digits to better understand the neural network model.

Errors

Notes on errors with Keras

Convolutional Neural Networks

Max Pooling 2D Dimension Application

In a convolutional neural network, the network architecture generally looks like this:

• Convolution
• Convolution
• Pool
• Dropout
• Flatten
• Dense
• Dropout
• Dense

The problem was with how the Pool layer was working, and which dimensions it was pooling. I was constructing the neural network as follows, starting with two Convolution2D layers. The input data was of shape (6, 32, 32) - it consisted of TWO 3-channel images (hence, 6 channels), and a 32 x 32 pixel resolution. Thus, the Convolution2D layers looked like this:

modelA.add(Conv2D(32, 
                  (3, 3), 
                  input_shape=(6, 32, 32),
                  padding='same', 
                  activation='relu'))

modelA.add(Conv2D(32, (3, 3), 
                  input_shape=(6, 32, 32),
                  padding='same', 
                  activation='relu'))

modelA.summary()

This resulted in the following model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 6, 32, 32)         9248      
=================================================================
Total params: 18,496
Trainable params: 18,496
Non-trainable params: 0

But then, when I created a new MaxPooling2D layer, it was incorrectly applying the pooling operation to the wrong dimensions - it was pooling the channels!

modelA.add(MaxPooling2D(pool_size=(4, 4))

This was resulting in the MaxPooling2D layer having an output shape of (4, 8, 32) - the pooling layer was pooling the channels, plus the first dimension of each photo, while leaving the second dimension alone.

I eventually figured out how to fix this by studying the MaxPooling2D documentation page: https://keras.io/layers/pooling/#maxpooling2d

This revealed a channels_first or channels_last option that I had not seen or set. Once I added it, I got what I was after:

modelA.add(MaxPooling2D(pool_size=(4, 4),
                       data_format='channels_first'))

modelA.summary()

This led to an output shape of (6, 8, 8), as desired.

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 6, 8, 8)           0         
=================================================================
Total params: 18,496
Trainable params: 18,496
Non-trainable params: 0

Binary Categorization

Accuracy stays at exactly 50% and does not change

Was running into the issue with Keras that, when running a binary categorization model, I was seeing predictions of exactly 50%, and nothing was changing at all.

This turned out to be due to a couple of reasons:

• Poor choice of activation function
• Poor choice of optimizer

Check out the first example here: https://keras.io/getting-started/sequential-model-guide/#training

This gives an example of a simple binary categorization network:

# For a single-input model with 2 classes (binary classification):

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Generate dummy data
import numpy as np
data = np.random.random((1000, 100))
labels = np.random.randint(2, size=(1000, 1))

# Train the model, iterating on the data in batches of 32 samples
model.fit(data, labels, epochs=10, batch_size=32)

note a few things being done here:

• Optimizer being used is "rmsprop"
• Activation function of last layer, Dense(1), is sigmoid
• The last layer is a dense layer with a single neuron, which (when on) represents yes, and (when off) represents no

What I was doing wrong:

• I had set the activation function of the last Dense layer as "softmax", which meant that I was ALWAYS predicting "yes" (everything was always rounded up). Because my training set had a 50/50 mix of yes/no cases, I was getting predictions of exactly 50% because I was always guessing "yes".
• (Even earlier) I had initially been using TWO Dense neurons, e.g., Dense(2), to get yes/no. This is not correct!
• I was trying to use an SGD optimizer... something fancy-pants that I should not have been using