Fixed malformed code in RNN example

Experiments/Tensorflow/RNN/belling_the_cat.txt

@@ -1 +1 @@
-long ago , the mice had a general council to consider what measures they could take to outwit their common enemy , the cat . some said this , and some said that but at last a young mouse got up and said he had a proposal to make , which he thought would meet the case . you will all agree , said he , that our chief danger consists in the sly and treacherous manner in which the enemy approaches us . now , if we could receive some signal of her approach , we could easily escape from her . i venture , therefore , to propose that a small bell be procured , and attached by a ribbon round the neck of the cat . by this means we should always know when she was about , and could easily retire while she was in the neighbourhood . this proposal met with general applause , until an old mouse got up and said  that is all very well , but who is to bell the cat ? the mice looked at one another and nobody spoke . then the old mouse said it is easy to propose impossible remedies . 
+long ago , the mice had a general council to consider what measures they could take to outwit their common enemy , the cat . some said this , and some said that but at last a young mouse got up and said he had a proposal to make , which he thought would meet the case . you will all agree , said he , that our chief danger consists in the sly and treacherous manner in which the enemy approaches us . now , if we could receive some signal of her approach , we could easily escape from her . i venture , therefore , to propose that a small bell be procured , and attached by a ribbon round the neck of the cat . by this means we should always know when she was about , and could easily retire while she was in the neighbourhood . this proposal met with general applause , until an old mouse got up and said  that is all very well , but who is to bell the cat ? the mice looked at one another and nobody spoke . then the old mouse said it is easy to propose impossible remedies .

Experiments/Tensorflow/RNN/rnn_words.py

@@ -37,20 +37,17 @@ def read_data(fname):
     with open(fname) as f:
         content = f.readlines()
     content = [x.strip() for x in content]
-    content = [content[i].split() for i in range(len(content))]
+    content = [word for i in range(len(content)) for word in content[i].split()]
     content = np.array(content)
-    content = np.reshape(content, [-1, ])
     return content
 
 training_data = read_data(training_file)
 print("Loaded training data...")
 
 def build_dataset(words):
     count = collections.Counter(words).most_common()
-    dictionary = dict()
-    for word, _ in count:
-        dictionary[word] = len(dictionary)
-    reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
+    dictionary = {word: rank for rank, (word, _) in enumerate(count)}
+    reverse_dictionary = {v: k for k, v in dictionary.items()}
     return dictionary, reverse_dictionary
 
 dictionary, reverse_dictionary = build_dataset(training_data)
@@ -178,4 +175,3 @@ def RNN(x, weights, biases):
             print(sentence)
         except:
             print("Word not in dictionary")
-

README.md

@@ -1,6 +1,6 @@
-# Deep Learning: Theory and Experiments
-
-## Notes
+# Deep Learning Lecture Notes and Experiments
+### Code samples have links to other repo that I maintain ([Advanced Deep Learning with Keras](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras) book) or contribute ([Keras](https://github.com/keras-team/keras))
+## Lecture Notes
 0. Course Roadmap
   - [Deep Learning](https://docs.google.com/presentation/d/1JSmMV3SICXJ3SQOSj5WgJipssOpBYZKNZm1YnU5HO3g/edit?usp=sharing)
   - [Limits of Deep Learning](https://docs.google.com/presentation/d/13nsjiEjpiUpidxThT6hoCg19o0TMG-ho4tzgvuPkWV8/edit?usp=sharing)
@@ -16,13 +16,66 @@
 3. Deep Neural Networks
   - [Deep Feedforward Neural Networks, Cost, Output, Hidden Units](https://docs.google.com/presentation/d/1woHBsNgnwzjJndMcXXznaBKlLvWywuA6T7BFi0K7Yhg/edit?usp=sharing)
   - [Back Propagation](https://docs.google.com/presentation/d/1XD0tA6oxOETfFn1DTGJByhhyH3MF586OCN06WvAP22E/edit?usp=sharing)
+- PyTorch Sample Code
+  - [Backprop on a single unit MLP](backprop/backprop.ipynb)
+- Keras Sample Code
+  - [Overview](https://docs.google.com/presentation/d/15Y1snbE73g8vw16RN6uehVHyDFxAK_b0iKcmId1j5qM/edit?usp=sharing)
+  - [MLP on Linear Model](keras/mlp/linear.ipynb)
+  - [MNIST Sampler](keras/mlp/mnist-sampler.ipynb)
+  - [MLP on MNIST](keras/mlp/mlp-mnist.ipynb)
 4. [Regularization](https://docs.google.com/presentation/d/1lg4oxRDvfUIEtzMJ7E-Lqv1cDNiwoNeT1r5T-XnFIQI/edit?usp=sharing)
+- Keras Sample Code
+  - [MLP on MNIST no Regularizer](keras/regularization/mlp-mnist-noreg.ipynb)
+  - [MLP on MNIST with L2](keras/regularization/mlp-mnist-l2.ipynb)
+  - [MLP on MNIST with Dropout](keras/regularization/mlp-mnist-dropout.ipynb)
+  - [MLP on MNIST with Data Augmentation](keras/regularization/mlp-mnist-data_augment.ipynb) 
+
 5. [Optimization](https://docs.google.com/presentation/d/1wt53ds5dywq3WUm-jkdKFUjiHayBAV6-CSFAJg76Clg/edit?usp=sharing)
-6. [Convolutional Neural Networks](https://docs.google.com/presentation/d/1vxCMwjbssYKisIWt2UYiuOFMsJaFv-5-I6mYvtJ6Hr8/edit?usp=sharing)
-7. [Embeddings](https://docs.google.com/presentation/d/1YtKWA53T2NqXoL0vnk8jWl1WCBWCLhbh5OWz_1JrGdU/edit?usp=sharing) 
-8. [Recurrent Neural Networks, LSTM, GRU](https://docs.google.com/presentation/d/1qjQkUwnr2V--7JPz0H_wkzRyTYX3UtJsYrB3MQPGKLE/edit?usp=sharing)
+
+6. [Convolutional Neural Networks (CNN)](https://docs.google.com/presentation/d/1vxCMwjbssYKisIWt2UYiuOFMsJaFv-5-I6mYvtJ6Hr8/edit?usp=sharing)
+- Keras Sample Code
+  - [CNN on MNIST](keras/cnn/cnn-mnist.ipynb)
+  - [CNN on MNIST using Functional API](keras/cnn/cnn-functional.ipynb)
+  - [CNN on MNIST Siamese Network](keras/cnn/cnn-siamese.ipynb)
+
+7. [Deep Networks](https://docs.google.com/presentation/d/14aFawAa4zNqvPRkhmS5YATVxSlS01fig7qnstecRgG0/edit?usp=sharing)
+- Keras Sample Code
+  - [DenseNet](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter2-deep-networks/densenet-cifar10-2.4.1.py)  
+  - [ResNet](https://github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py)
+8. [Embeddings](https://docs.google.com/presentation/d/1YtKWA53T2NqXoL0vnk8jWl1WCBWCLhbh5OWz_1JrGdU/edit?usp=sharing) 
+- Keras Sample Code
+  - [Simple Embedding and Sentiment Analysis](keras/embedding/sentiment_analysis.ipynb)
+  - [Glove Embedding](keras/embedding/glove_embedding.ipynb)
+9. [Recurrent Neural Networks, LSTM, GRU](https://docs.google.com/presentation/d/1qjQkUwnr2V--7JPz0H_wkzRyTYX3UtJsYrB3MQPGKLE/edit?usp=sharing)
+- Keras Sample Code
+  - [SimpleRNN on MNIST](keras/rnn/simple-rnn-mnist.ipynb)
+  - [CuDNNLSTM on MNIST](keras/rnn/cudnnlstm-mnist.ipynb)
+10. [AutoEncoders](https://docs.google.com/presentation/d/1gXWl0luuDe1qoQLSKdOUrzoq51WhhNLeq7x3PxQXYkA/edit?usp=sharing)
+- Keras Sample Code
+  - [AutoEncoder](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter3-autoencoders/autoencoder-mnist-3.2.1.py)
+  - [Denoising AutoEncoder](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter3-autoencoders/denoising-autoencoder-mnist-3.3.1.py)
+  - [Colorization AutoEncoder](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter3-autoencoders/colorization-autoencoder-cifar10-3.4.1.py)
+11. [Generative Adversarial Networks (GAN)](https://docs.google.com/presentation/d/13fiFibqjl9ps_CktJzMNAvoZXOlzHQDu8eRSb3a227g/edit?usp=sharing)
+- Keras Sample Code
+  - [DCGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter4-gan/dcgan-mnist-4.2.1.py)
+  - [CGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter4-gan/cgan-mnist-4.3.1.py)
+
+11a. [Improved GANs](https://docs.google.com/presentation/d/1tATpY1gzJo8x6Ziceln-KjcgEeqX-fnapL6IFdc9Wbk/edit?usp=sharing)
+- Keras Sample Code
+  - [WGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter5-improved-gan/wgan-mnist-5.1.2.py)
+12. [Variational Autoencoder (VAE)](https://docs.google.com/presentation/d/1ORVwhh5PgWEehcUQYL9t8nBCk9cj4TgXmUJIl6WMkpo/edit?usp=sharing)
+- Keras Sample Code
+  - [VAE MLP](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter8-vae/vae-mlp-mnist-8.1.1.py)
+  - [VAE CNN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter8-vae/vae-cnn-mnist-8.1.2.py)
+  - [CVAE](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter8-vae/cvae-cnn-mnist-8.2.1.py)
+13. [Deep Reinforcement Learning (DRL)](https://docs.google.com/presentation/d/1oZC5qGofbx-dlPcnr00_fPBK0GEtt6FnMdpsnA6XYX8/edit?usp=sharing)
+- Keras Sample Code
+  - [Q-Learning](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter9-drl/q-learning-9.3.1.py)
+  - [Q-Learning on FrozenLake-v0](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter9-drl/q-frozenlake-9.5.1.py)
+  - [DQN and DDQN on CartPole-v0](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter9-drl/dqn-cartpole-9.6.1.py)
 
-## [Tensorflow](https://www.tensorflow.org/) (tf) Experiments
+### Warning: The following are old experiments that are longer updated and maintained
+### [Tensorflow](https://www.tensorflow.org/) Experiments
 1. [Hello World!](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/Intro/hello.py) 
 2. [Linear Algebra](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/Math/linear_algebra.py)
 3. [Matrix Decomposition](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/Math/decomposition.py)

backprop/backprop.ipynb

@@ -0,0 +1,188 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Backprop demo on a single unit linear MLP\n",
+    "This is a demonstration of how backpropagation works on a single unit MLP"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 96,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "import torch.nn as nn\n",
+    "import torch.nn.functional as F\n",
+    "import torch.optim as optim"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A single neuron/unit MLP with linear activation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 97,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Net(nn.Module):\n",
+    "    def __init__(self):\n",
+    "        super(Net, self).__init__()\n",
+    "        self.fc1 = nn.Linear(1, 1)\n",
+    "    \n",
+    "    def forward(self, x):\n",
+    "        x = self.fc1(x)\n",
+    "        return x"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Build a network. Model a function: y = 2*x + 1\n",
+    "\n",
+    "Perform supervised learning using the following dataset:\n",
+    "\n",
+    "\n",
+    "| Step |  x   |  y  |\n",
+    "| :-- | ---: | ---:|\n",
+    "|  0   | 0.   |  1. |\n",
+    "|  1   | 1.   |  3. |\n",
+    "\n",
+    "At each step, perform backprop and print the gradients."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 107,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "---------------------0-----------------------\n",
+      "0. Bias grad:  None\n",
+      "0. Weights grad:  None\n",
+      "0. Bias:  tensor(0., grad_fn=<SelectBackward>)\n",
+      "0. Weights:  tensor([1.], grad_fn=<SelectBackward>)\n",
+      "0. Input:  tensor([0.])\n",
+      "0. Predicted Output:  tensor([0.], grad_fn=<AddBackward0>)\n",
+      "0. Loss:  tensor(1., grad_fn=<MseLossBackward>)\n",
+      "\n",
+      "---------------------1-----------------------\n",
+      "1. Bias grad:  tensor([-2.])\n",
+      "1. Weights grad:  tensor([[0.]])\n",
+      "1. Bias:  tensor(0.2000, grad_fn=<SelectBackward>)\n",
+      "1. Weights:  tensor([1.], grad_fn=<SelectBackward>)\n",
+      "1. Input:  tensor([1.])\n",
+      "1. Predicted Output:  tensor([1.2000], grad_fn=<AddBackward0>)\n",
+      "1. Loss:  tensor(3.2400, grad_fn=<MseLossBackward>)\n",
+      "\n",
+      "---------------------2-----------------------\n",
+      "2. Bias grad:  tensor([-3.6000])\n",
+      "2. Weights grad:  tensor([[-3.6000]])\n",
+      "2. Bias:  tensor(0.5600, grad_fn=<SelectBackward>)\n",
+      "2. Weights:  tensor([1.3600], grad_fn=<SelectBackward>)\n"
+     ]
+    }
+   ],
+   "source": [
+    "net = Net()\n",
+    "\n",
+    "# for ease of backprop demo, we set weight=1.0 and bias=0.0\n",
+    "net.fc1.bias = torch.nn.Parameter(torch.tensor([0.]))\n",
+    "net.fc1.weight = torch.nn.Parameter(torch.tensor([[1.]]))\n",
+    "print(\"---------------------0-----------------------\")\n",
+    "print(\"0. Bias grad: \", net.fc1.bias.grad)\n",
+    "print(\"0. Weights grad: \", net.fc1.weight.grad)\n",
+    "print(\"0. Bias: \", net.fc1.bias[0])\n",
+    "print(\"0. Weights: \", net.fc1.weight[0])\n",
+    "\n",
+    "# x=0, y=1.\n",
+    "input = torch.tensor([0.])\n",
+    "print(\"0. Input: \", input)\n",
+    "input.unsqueeze(0)\n",
+    "output = net(input)\n",
+    "print(\"0. Predicted Output: \", output)\n",
+    "target = torch.tensor([1.])\n",
+    "target = target.view(1, -1)\n",
+    "\n",
+    "# Use MSE Loss\n",
+    "criterion = nn.MSELoss()\n",
+    "loss = criterion(output, target)\n",
+    "print(\"0. Loss: \", loss)\n",
+    "\n",
+    "# Use SGD optimizer with learning rate of 0.1\n",
+    "optimizer = optim.SGD(net.parameters(), lr=0.1)\n",
+    "# Clear optimizer gradient\n",
+    "optimizer.zero_grad()\n",
+    "# Perform backprop\n",
+    "loss.backward(retain_graph=True)\n",
+    "# Update weight and bias\n",
+    "optimizer.step()\n",
+    "print(\"\\n---------------------1-----------------------\")\n",
+    "print(\"1. Bias grad: \", net.fc1.bias.grad)\n",
+    "print(\"1. Weights grad: \",net.fc1.weight.grad)\n",
+    "print(\"1. Bias: \", net.fc1.bias[0])\n",
+    "print(\"1. Weights: \",net.fc1.weight[0])\n",
+    "\n",
+    "# x=1.0, y=3.0\n",
+    "input = torch.tensor([1.])\n",
+    "print(\"1. Input: \", input)\n",
+    "input.unsqueeze(0)\n",
+    "output = net(input)\n",
+    "print(\"1. Predicted Output: \", output)\n",
+    "target = torch.tensor([3.])\n",
+    "target = target.view(1, -1)\n",
+    "\n",
+    "optimizer.zero_grad()\n",
+    "loss = criterion(output, target)\n",
+    "print(\"1. Loss: \", loss)\n",
+    "loss.backward()\n",
+    "optimizer.step()\n",
+    "print(\"\\n---------------------2-----------------------\")\n",
+    "print(\"2. Bias grad: \", net.fc1.bias.grad)\n",
+    "print(\"2. Weights grad: \",net.fc1.weight.grad)\n",
+    "print(\"2. Bias: \", net.fc1.bias[0])\n",
+    "print(\"2. Weights: \",net.fc1.weight[0])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

keras/cnn/cnn-functional-2.1.1.py

@@ -0,0 +1,78 @@
+''' Using Functional API to build CNN
+
+~99.3% test accuracy
+'''
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+from keras.layers import Dense, Dropout, Input
+from keras.layers import Conv2D, MaxPooling2D, Flatten
+from keras.models import Model
+from keras.datasets import mnist
+from keras.utils import to_categorical
+
+
+# load MNIST dataset
+(x_train, y_train), (x_test, y_test) = mnist.load_data()
+
+# from sparse label to categorical
+num_labels = len(np.unique(y_train))
+y_train = to_categorical(y_train)
+y_test = to_categorical(y_test)
+
+# reshape and normalize input images
+image_size = x_train.shape[1]
+x_train = np.reshape(x_train,[-1, image_size, image_size, 1])
+x_test = np.reshape(x_test,[-1, image_size, image_size, 1])
+x_train = x_train.astype('float32') / 255
+x_test = x_test.astype('float32') / 255
+
+# network parameters
+input_shape = (image_size, image_size, 1)
+batch_size = 128
+kernel_size = 3
+filters = 64
+dropout = 0.3
+
+# use functional API to build cnn layers
+inputs = Input(shape=input_shape)
+y = Conv2D(filters=filters,
+           kernel_size=kernel_size,
+           activation='relu')(inputs)
+y = MaxPooling2D()(y)
+y = Conv2D(filters=filters,
+           kernel_size=kernel_size,
+           activation='relu')(y)
+y = MaxPooling2D()(y)
+y = Conv2D(filters=filters,
+           kernel_size=kernel_size,
+           activation='relu')(y)
+# image to vector before connecting to dense layer
+y = Flatten()(y)
+# dropout regularization
+y = Dropout(dropout)(y)
+outputs = Dense(num_labels, activation='softmax')(y)
+
+# build the model by supplying inputs/outputs
+model = Model(inputs=inputs, outputs=outputs)
+# network model in text
+model.summary()
+
+# classifier loss, Adam optimizer, classifier accuracy
+model.compile(loss='categorical_crossentropy',
+              optimizer='adam',
+              metrics=['accuracy'])
+
+# train the model with input images and labels
+model.fit(x_train,
+          y_train,
+          validation_data=(x_test, y_test),
+          epochs=20,
+          batch_size=batch_size)
+
+# model accuracy on test dataset
+score = model.evaluate(x_test, y_test, batch_size=batch_size)
+print("\nTest accuracy: %.1f%%" % (100.0 * score[1]))