Keras

From charlesreid1

Revision as of 11:37, 14 October 2017 by Admin (talk | contribs) (→‎Github Repos)

Neural network package in Python.

Code

Github Repos

charlesreid1

  • https://github.com/charlesreid1/lfw_fuel - fork of lfw_fuel repo, which applies kerosene and fuel (Python libraries) for organizing and packaging LFW data in a nice format that makes it easy to load and hand off to Keras.
  • https://github.com/charlesreid1/circe - specifically the examples using the NIST handwriting digit classification data set. This shows how to utilize Keras to train a neural network to perform dimensionality reduction, and further explores the manifold that the neural network identified for the different digits to better understand the neural network model.


Keras

Paper/Network Implementations

Related Utilities

Errors

Notes on errors with Keras

Convolutional Neural Networks

Max Pooling 2D Dimension Application

In a convolutional neural network, the network architecture generally looks like this:

  • Convolution
  • Convolution
  • Pool
  • Dropout
  • Flatten
  • Dense
  • Dropout
  • Dense

The problem was with how the Pool layer was working, and which dimensions it was pooling. I was constructing the neural network as follows, starting with two Convolution2D layers. The input data was of shape (6, 32, 32) - it consisted of TWO 3-channel images (hence, 6 channels), and a 32 x 32 pixel resolution. Thus, the Convolution2D layers looked like this: 

modelA.add(Conv2D(32, 
                  (3, 3), 
                  input_shape=(6, 32, 32),
                  padding='same', 
                  activation='relu'))

modelA.add(Conv2D(32, (3, 3), 
                  input_shape=(6, 32, 32),
                  padding='same', 
                  activation='relu'))

modelA.summary()

This resulted in the following model summary: 

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 6, 32, 32)         9248      
=================================================================
Total params: 18,496
Trainable params: 18,496
Non-trainable params: 0

But then, when I created a new MaxPooling2D layer, it was incorrectly applying the pooling operation to the wrong dimensions - it was pooling the channels! 

modelA.add(MaxPooling2D(pool_size=(4, 4))

This was resulting in the MaxPooling2D layer having an output shape of (4, 8, 32) - the pooling layer was pooling the channels, plus the first dimension of each photo, while leaving the second dimension alone.

I eventually figured out how to fix this by studying the MaxPooling2D documentation page: https://keras.io/layers/pooling/#maxpooling2d

This revealed a channels_first or channels_last option that I had not seen or set. Once I added it, I got what I was after: 

modelA.add(MaxPooling2D(pool_size=(4, 4),
                       data_format='channels_first'))

modelA.summary()

This led to an output shape of (6, 8, 8), as desired. 

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 6, 8, 8)           0         
=================================================================
Total params: 18,496
Trainable params: 18,496
Non-trainable params: 0

Binary Categorization

Accuracy stays at exactly 50% and does not change

Was running into the issue with Keras that, when running a binary categorization model, I was seeing predictions of exactly 50%, and nothing was changing at all.

This turned out to be due to a couple of reasons:

  • Poor choice of activation function
  • Poor choice of optimizer

Check out the first example here: https://keras.io/getting-started/sequential-model-guide/#training

This gives an example of a simple binary categorization network: 

# For a single-input model with 2 classes (binary classification):

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Generate dummy data
import numpy as np
data = np.random.random((1000, 100))
labels = np.random.randint(2, size=(1000, 1))

# Train the model, iterating on the data in batches of 32 samples
model.fit(data, labels, epochs=10, batch_size=32)

note a few things being done here:

  • Optimizer being used is "rmsprop"
  • Activation function of last layer, Dense(1), is sigmoid
  • The last layer is a dense layer with a single neuron, which (when on) represents yes, and (when off) represents no

What I was doing wrong:

  • I had set the activation function of the last Dense layer as "softmax", which meant that I was ALWAYS predicting "yes" (everything was always rounded up). Because my training set had a 50/50 mix of yes/no cases, I was getting predictions of exactly 50% because I was always guessing "yes".
  • (Even earlier) I had initially been using TWO Dense neurons, e.g., Dense(2), to get yes/no. This is not correct!
  • I was trying to use an SGD optimizer... something fancy-pants that I should not have been using

Flags

Retrieved from "https://charlesreid1.com/w/index.php?title=Keras&oldid=21627"