Neural network package in Python.
- 1 Code
- 2 Notes
- 3 Errors
- 4 Resources
- 5 Flags
- https://github.com/charlesreid1/in-your-face - examples of fitting Keras neural networks to the LFW (labeled faces in the wild) dataset.
- https://github.com/charlesreid1/lfw_fuel - fork of lfw_fuel repo, which applies kerosene and fuel (Python libraries) for organizing and packaging LFW data in a nice format that makes it easy to load and hand off to Keras.
- https://github.com/charlesreid1/circe - specifically the examples using the NIST handwriting digit classification data set. This shows how to utilize Keras to train a neural network to perform dimensionality reduction, and further explores the manifold that the neural network identified for the different digits to better understand the neural network model.
- https://github.com/fchollet/keras - the main Keras repo
- https://github.com/farizrahman4u/keras-contrib - keras community contributions
- https://github.com/fchollet/deep-learning-models - pre-trained models for Keras, now available in the main Keras repo/package
- https://github.com/fchollet/hualos - Keras total visualization project (Keras RemoteMonitor callbacks - JSON - Flask - C3)
- https://github.com/MarcBS/keras - Keras fork with additional functionality
- https://github.com/MarcBS/multimodal_keras_wrapper - wrapper for MarcBS' Keras fork (see prior entry)
- https://github.com/dmlc/keras - Fork of Keras that supports an MXNet backend
- https://github.com/titu1994/Neural-Style-Transfer - neural network style transfer network (implementation from paper) via Keras
- https://github.com/ellisvalentiner/credit-card-fraud - analysis of credit card fraud (uses custom Keras layer)
- https://github.com/maciejkula/triplet_recommendations_keras - movie recommendation with triplet loss function in Keras
- https://github.com/bstriner/keras-adversarial - GANs (generative adversarial networks) using keras
- https://github.com/farizrahman4u/recurrentshop - Keras framework for building complex RNNs
- https://github.com/usernaamee/keras-wavenet - Keras implementation of Deep Mind's Wavenet paper
- https://github.com/kentsommer/keras-inceptionV4 - Keras implementation of Inception V4 architecture
- https://github.com/flyyufelix/DenseNet-Keras - Keras implementation of DenseNet + ImageNet
- https://github.com/pengpaiSH/Kaggle_NCFM - Keras to solve Kaggle Nature Conservancy Fisheries Monitoring dataset leaderboard
- https://github.com/kylemcdonald/SmileCNN - CNN to detect smiles with Keras
- https://github.com/titu1994/Super-Resolution-using-Generative-Adversarial-Networks - super resolution with generative adversarial networks
- https://github.com/snf/keras-fractalnet - FractalNet ultra deep neural networks w/o residuals (M$ paper)
- https://github.com/gustavla/fractalnet - original version by paper's author
- https://github.com/alexander-rakhlin/CNN-for-Sentence-Classification-in-Keras - CNN for sentence classification in Keras
- https://github.com/udibr/headlines - generation of short headlines for articles using Keras, NLP, and RNN
- https://github.com/jinfagang/LSTM_learn - Keras for LSTM and time series prediction
- https://github.com/kengz/openai_lab - reinforcement learning with Keras
- https://github.com/jisungk/deepjazz - deep learning generative jazz music
- https://github.com/bstriner/keras-tqdm - Keras plus TQDN for nice progress bars
- https://github.com/yusugomori/deeplearning-tensorflow-keras - extensive notes/notebooks on different architectures, divided into chapters/sections
- https://github.com/bstriner/bayesian_dense - bayesian weight uncertainty dense layer for Keras
- https://github.com/bstriner/dense_tensor - dense Tensor layer for Keras
- https://github.com/sandeep-krishnamurthy/keras-mxnet-benchmarks - Examples for profiling performance of Keras/MXnet
- https://github.com/keplr-io/quiver - convolutional neural net visualization for Keras
- https://github.com/joeddav/devol - automating hyperparameter tuning with genetic algorithm
I think my in-your-face repository (https://github.com/charlesreid1/in-your-face) does a pretty good job of making notes on how to use Keras as we go.
Check out the iPython notebooks in that repository.
Notes on errors with Keras
Convolutional Neural Networks
Max Pooling 2D Applied to Wrong Dimensions
In a convolutional neural network, the network architecture generally looks like this:
The problem was with how the Pool layer was working, and which dimensions it was pooling. I was constructing the neural network as follows, starting with two Convolution2D layers. The input data was of shape (6, 32, 32) - it consisted of TWO 3-channel images (hence, 6 channels), and a 32 x 32 pixel resolution. Thus, the Convolution2D layers looked like this:
modelA.add(Conv2D(32, (3, 3), input_shape=(6, 32, 32), padding='same', activation='relu')) modelA.add(Conv2D(32, (3, 3), input_shape=(6, 32, 32), padding='same', activation='relu')) modelA.summary()
This resulted in the following model summary:
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_1 (Conv2D) (None, 6, 32, 32) 9248 _________________________________________________________________ conv2d_2 (Conv2D) (None, 6, 32, 32) 9248 ================================================================= Total params: 18,496 Trainable params: 18,496 Non-trainable params: 0
But then, when I created a new MaxPooling2D layer, it was incorrectly applying the pooling operation to the wrong dimensions - it was pooling the channels!
This was resulting in the MaxPooling2D layer having an output shape of
(4, 8, 32) - the pooling layer was pooling the channels, plus the first dimension of each photo, while leaving the second dimension alone.
I eventually figured out how to fix this by studying the MaxPooling2D documentation page: https://keras.io/layers/pooling/#maxpooling2d
This revealed a channels_first or channels_last option that I had not seen or set. Once I added it, I got what I was after:
modelA.add(MaxPooling2D(pool_size=(4, 4), data_format='channels_first')) modelA.summary()
This led to an output shape of (6, 8, 8), as desired.
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_1 (Conv2D) (None, 6, 32, 32) 9248 _________________________________________________________________ conv2d_2 (Conv2D) (None, 6, 32, 32) 9248 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 6, 8, 8) 0 ================================================================= Total params: 18,496 Trainable params: 18,496 Non-trainable params: 0
Accuracy stays at exactly 50% and does not change
Was running into the issue with Keras that, when running a binary categorization model, I was seeing predictions of exactly 50%, and nothing was changing at all.
This turned out to be due to a couple of reasons:
- Poor choice of activation function
- Poor choice of optimizer
Check out the first example here: https://keras.io/getting-started/sequential-model-guide/#training
This gives an example of a simple binary categorization network:
# For a single-input model with 2 classes (binary classification): model = Sequential() model.add(Dense(32, activation='relu', input_dim=100)) model.add(Dense(1, activation='sigmoid')) model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy']) # Generate dummy data import numpy as np data = np.random.random((1000, 100)) labels = np.random.randint(2, size=(1000, 1)) # Train the model, iterating on the data in batches of 32 samples model.fit(data, labels, epochs=10, batch_size=32)
note a few things being done here:
- Optimizer being used is "rmsprop"
- Activation function of last layer, Dense(1), is sigmoid
- The last layer is a dense layer with a single neuron, which (when on) represents yes, and (when off) represents no
What I was doing wrong:
- I had set the activation function of the last Dense layer as "softmax", which meant that I was ALWAYS predicting "yes" (everything was always rounded up). Because my training set had a 50/50 mix of yes/no cases, I was getting predictions of exactly 50% because I was always guessing "yes".
- (Even earlier) I had initially been using TWO Dense neurons, e.g., Dense(2), to get yes/no. This is not correct!
- I was trying to use an SGD optimizer... something fancy-pants that I should not have been using
Ugh, a big ugly giant hairy list of links. I'll do my best to keep this maintained.
Building a very, very simple neural network with Keras - https://www.pyimagesearch.com/2016/09/26/a-simple-neural-network-with-python-and-keras/