u-net-notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes and thoughts\n",
    "\n",
    "#### Current best ideas / runs that work / etc.\n",
    "- Batch size = ~1/6th of training data size seems to make all the difference\n",
    "    - Smaller batch sizes = noisier graphs but more weight updates\n",
    "    - Larger batch sizes = smoother graphs with fewer weight updates\n",
    "    - Unclear which helps here\n",
    "- nfilters is only 4 for a single class (perhaps 8 works better? Takes longer to train. Trade-offs...)\n",
    "    - It seems that 16 produces better jaccard indices when dropout is in play\n",
    "- Perhaps the best method is to train 10 models, one for each class\n",
    "- Dropout after each maxpooling step in the down-path and after each upconv/conv/conv step in the up-path.\n",
    "    - Initially used p=0.2 for all layers\n",
    "    - Changed to `[0.1,0.2,0.3,0.4,0.5,0.4,0.3,0.2,0.1]` which seems to be working well\n",
    "- Activation = LeakyReLu\n",
    "    - What about going back to regular relu?\n",
    "- Batchnorm is *after* the activation\n",
    "- L2 reg is applied to each convolution with a lambda of 0.00001\n",
    "    - Question remains: is this even doing anything?\n",
    "    - I set the reg paramter to zero and observed no major difference\n",
    "- lr is 0.001\n",
    "    - This is the default for Adam. Would a higher value train faster?\n",
    "- Currently there are 400 training images\n",
    "    - Could this be reduced?\n",
    "        - Reducing lowers the validation scores. More data is better.\n",
    "    - Currently the data augmentation is applied to the entire data set, then it is randomly split for validation\n",
    "    - Should the test/val sets be split early and *then* apply data augmentation *seperately*?\n",
    "    - **NB** be aware that some classes only have a single image. Thus, the training/val sets should each contain examples of this class but it should be heavily augmented so as to be treated as different)\n",
    "\n",
    "#### Ideas\n",
    "- Weight map: sum the total area of all classes in y, then calculate each class' proportion of the total and use `1-value` in place of 1 in the binary mask. This will cause low frequency classes to contribute more to the total loss, i.e. penalizing the model when it fails to predict low frequency classes.\n",
    "\n",
    "#### Data augmentation / image manipulation\n",
    "- [Histogram Equalization](http://scikit-image.org/docs/dev/auto_examples/color_exposure/plot_equalize.html#sphx-glr-auto-examples-color-exposure-plot-equalize-py) (also see [here](https://www.kaggle.com/gabrielaltay/dstl-satellite-imagery-feature-detection/exploring-color-scaling-images/discussion), [here](http://adilmoujahid.com/posts/2016/06/introduction-deep-learning-python-caffe/) and [here](https://medium.com/@vivek.yadav/improved-performance-of-deep-learning-neural-network-models-on-traffic-sign-classification-using-6355346da2dc#.x9nidcsh6))\n",
    "- Rotation (with reflection): see data-augmentation.ipynb\n",
    "- Image normalization: see image-preprocessing-new.ipynb\n",
    "\n",
    "#### Overfitting solutions\n",
    "- CNNs are supposed to be more robust to this because of the shared weight matrix of each filter\n",
    "- **Data augmentation!**\n",
    "    - Have done random rotations on data increasing total n by 16-fold\n",
    "- L2 regularization (added into the layers via `W_regularizer=l2(l=0.01)` parameter)\n",
    "    - Not seeing much improvement\n",
    "    - Currently set to 0\n",
    "- Batchnorm\n",
    "    - Move batchnorm to *before* the relu takes place (see [here](http://stackoverflow.com/questions/34716454/where-do-i-call-the-batchnormalization-function-in-keras)\n",
    "    - Add batchnorm to upconv() layer\n",
    "    - mode:\n",
    "        - Currently using mode=2\n",
    "        - Try other modes?"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}