Keras

Neural network package in Python.

Code

Github Repos

charlesreid1

• https://github.com/charlesreid1/in-your-face - examples of fitting Keras neural networks to the LFW (labeled faces in the wild) dataset.

• https://github.com/charlesreid1/lfw_fuel - fork of lfw_fuel repo, which applies kerosene and fuel (Python libraries) for organizing and packaging LFW data in a nice format that makes it easy to load and hand off to Keras.

• https://github.com/charlesreid1/circe - specifically the examples using the NIST handwriting digit classification data set. This shows how to utilize Keras to train a neural network to perform dimensionality reduction, and further explores the manifold that the neural network identified for the different digits to better understand the neural network model.

Keras

• https://github.com/fchollet/keras - the main Keras repo

• https://github.com/farizrahman4u/keras-contrib - keras community contributions

• https://github.com/fchollet/deep-learning-models - pre-trained models for Keras, now available in the main Keras repo/package

• https://github.com/fchollet/hualos - Keras total visualization project (Keras RemoteMonitor callbacks - JSON - Flask - C3)

Paper/Network Implementations

• https://github.com/titu1994/Neural-Style-Transfer - neural network style transfer network (implementation from paper) via Keras

• https://github.com/ellisvalentiner/credit-card-fraud - analysis of credit card fraud (uses custom Keras layer)

• https://github.com/maciejkula/triplet_recommendations_keras - movie recommendation with triplet loss function in Keras

• https://github.com/bstriner/keras-adversarial - GANs (generative adversarial networks) using keras

• https://github.com/farizrahman4u/recurrentshop - Keras framework for building complex RNNs

• https://github.com/usernaamee/keras-wavenet - Keras implementation of Deep Mind's Wavenet paper

• https://github.com/kentsommer/keras-inceptionV4 - Keras implementation of Inception V4 architecture

• https://github.com/pengpaiSH/Kaggle_NCFM - Keras to solve Kaggle Nature Conservancy Fisheries Monitoring dataset leaderboard

• https://github.com/kylemcdonald/SmileCNN - CNN to detect smiles with Keras

• https://github.com/snf/keras-fractalnet - FractalNet ultra deep neural networks w/o residuals (M$ paper)
  • https://github.com/gustavla/fractalnet - original version by paper's author

• https://github.com/alexander-rakhlin/CNN-for-Sentence-Classification-in-Keras - CNN for sentence classification in Keras

• https://github.com/udibr/headlines - generation of short headlines for articles using Keras, NLP, and RNN

• https://github.com/jinfagang/LSTM_learn - Keras for LSTM and time series prediction

• https://github.com/kengz/openai_lab - reinforcement learning with Keras

• https://github.com/jisungk/deepjazz - deep learning generative jazz music

• https://github.com/bstriner/keras-tqdm - Keras plus TQDN for nice progress bars

Related Utilities

• https://github.com/yusugomori/deeplearning-tensorflow-keras - extensive notes/notebooks on different architectures, divided into chapters/sections

• https://github.com/bstriner/bayesian_dense - bayesian weight uncertainty dense layer for Keras

• https://github.com/bstriner/dense_tensor - dense Tensor layer for Keras

• https://github.com/sandeep-krishnamurthy/keras-mxnet-benchmarks - Examples for profiling performance of Keras/MXnet

• https://github.com/keplr-io/quiver - convolutional neural net visualization for Keras

• https://github.com/joeddav/devol - automating hyperparameter tuning with genetic algorithm

Errors

Notes on errors with Keras

Convolutional Neural Networks

Max Pooling 2D Dimension Application

In a convolutional neural network, the network architecture generally looks like this:

• Convolution
• Convolution
• Pool
• Dropout
• Flatten
• Dense
• Dropout
• Dense

The problem was with how the Pool layer was working, and which dimensions it was pooling. I was constructing the neural network as follows, starting with two Convolution2D layers. The input data was of shape (6, 32, 32) - it consisted of TWO 3-channel images (hence, 6 channels), and a 32 x 32 pixel resolution. Thus, the Convolution2D layers looked like this:

modelA.add(Conv2D(32, 
                  (3, 3), 
                  input_shape=(6, 32, 32),
                  padding='same', 
                  activation='relu'))

modelA.add(Conv2D(32, (3, 3), 
                  input_shape=(6, 32, 32),
                  padding='same', 
                  activation='relu'))

modelA.summary()

This resulted in the following model summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 6, 32, 32)         9248      
=================================================================
Total params: 18,496
Trainable params: 18,496
Non-trainable params: 0

But then, when I created a new MaxPooling2D layer, it was incorrectly applying the pooling operation to the wrong dimensions - it was pooling the channels!

modelA.add(MaxPooling2D(pool_size=(4, 4))

This was resulting in the MaxPooling2D layer having an output shape of (4, 8, 32) - the pooling layer was pooling the channels, plus the first dimension of each photo, while leaving the second dimension alone.

I eventually figured out how to fix this by studying the MaxPooling2D documentation page: https://keras.io/layers/pooling/#maxpooling2d

This revealed a channels_first or channels_last option that I had not seen or set. Once I added it, I got what I was after:

modelA.add(MaxPooling2D(pool_size=(4, 4),
                       data_format='channels_first'))

modelA.summary()

This led to an output shape of (6, 8, 8), as desired.

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 6, 32, 32)         9248      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 6, 8, 8)           0         
=================================================================
Total params: 18,496
Trainable params: 18,496
Non-trainable params: 0

Binary Categorization

Accuracy stays at exactly 50% and does not change

Was running into the issue with Keras that, when running a binary categorization model, I was seeing predictions of exactly 50%, and nothing was changing at all.

This turned out to be due to a couple of reasons:

• Poor choice of activation function
• Poor choice of optimizer

Check out the first example here: https://keras.io/getting-started/sequential-model-guide/#training

This gives an example of a simple binary categorization network:

# For a single-input model with 2 classes (binary classification):

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Generate dummy data
import numpy as np
data = np.random.random((1000, 100))
labels = np.random.randint(2, size=(1000, 1))

# Train the model, iterating on the data in batches of 32 samples
model.fit(data, labels, epochs=10, batch_size=32)

note a few things being done here:

• Optimizer being used is "rmsprop"
• Activation function of last layer, Dense(1), is sigmoid
• The last layer is a dense layer with a single neuron, which (when on) represents yes, and (when off) represents no

What I was doing wrong:

• I had set the activation function of the last Dense layer as "softmax", which meant that I was ALWAYS predicting "yes" (everything was always rounded up). Because my training set had a 50/50 mix of yes/no cases, I was getting predictions of exactly 50% because I was always guessing "yes".
• (Even earlier) I had initially been using TWO Dense neurons, e.g., Dense(2), to get yes/no. This is not correct!
• I was trying to use an SGD optimizer... something fancy-pants that I should not have been using