first_scaling_up_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First Somewhat Successful Scale Up with XTCAV\n",
    "\n",
    "Attempt to classify lasing vs. nolasing.\n",
    "\n",
    "## Input\n",
    "\n",
    "* 80 xtcav images, randomly chosen from amo86815 runs 69 (no lasing) 70 and 71 (lasing), but chosen to be balanced (40 no las and 40 lasing)\n",
    "* background subtracted (from dark run 68)\n",
    "* downsampled to (284, 363) from (726, 568)\n",
    "* log transformed: $ I \\rightarrow \\log(1 + \\max(0,I)) $\n",
    "\n",
    "input tensor shape=(80, 284, 363, 1) # one channel\n",
    "\n",
    "8,247,360 pixels, 33 MB\n",
    "\n",
    "## Network\n",
    "\n",
    "* Two layers - one convolutional layer, one fully connected layer mapping to two outputs\n",
    "* softmax classifier\n",
    "\n",
    "* convolutional kernels: window=(16,16) numchannels=16, strides=(2,2)\n",
    "* bias_init=0, K_init_stddev=0.01\n",
    "\n",
    "* conv output shape=(80, 142, 182, 16)   (126 MB)\n",
    "* relu activation\n",
    "* avg pool: window=(12,12) strides=(10,10)\n",
    "* pool output shape=(80, 15, 19, 16)  1.39 MB\n",
    "\n",
    "cvn layer produces 4560 output units for fully connected output layer, with\n",
    "\n",
    "* Weights = (4560,2)  \n",
    "* bias_init=0\n",
    "* W_init_stddev=0.01\n",
    "\n",
    "convnet has 13,234 unknown variables, 4112 (31%) in convnet layers, and 9,122 (68%) in hidden layers.\n",
    "\n",
    "convnet maps 103,092 features to 2 outputs for hidden layers.\n",
    "\n",
    "## loss/optimization\n",
    "\n",
    "* average cross entropy, i.e\n",
    "``` \n",
    "reduce_mean( softmax_cross_entropy_with_logits( H_O, labels))\n",
    "```\n",
    "where `H_O` is the `(80,2)` output tensor from the linear operations of the fully connected output layer, and labels are the one hot vectors of the lasing/no lasing truth\n",
    "\n",
    "* no regularization\n",
    "* momemtum optimizer with mom=0.4\n",
    "* learning rate starts at 0.01, exponential decay rate of 0.96, decay steps=10, staircase=True\n",
    "\n",
    "## Job\n",
    "\n",
    "* told tensorflow to use 4 threads via \n",
    "```\n",
    "        sess = tf.Session(config=tf.ConfigProto(intra_op_parallelism_threads = FLAGS.intra_op_parallelism_threads))\n",
    "```\n",
    "which seemed to work, I get these messages\n",
    "```\n",
    "I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 4\n",
    "I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 24\n",
    "```\n",
    "\n",
    "After about 10 hours, the job did 30 steps - 20 minutes a step. Below is some output. Key\n",
    "\n",
    "* `m1s` is the number of 1's in the training data (40/80) this is more interesting for minibatch and stochatstic gradient descent\n",
    "* tr.acc/#1s is the training set accuracy and #1's predicted in the training set\n",
    "* tst.acc/#1s is the test set accuracy and #1's predicted in the test set - this has 60 randomly chosen samples (not overlapping the train set)\n",
    "* loss same as xentropy since no regularization\n",
    "* gr-ang is $\\cos(g_{t-1}, g)$ where $g_t$ is the gradient (a 13,234 long vector) at step $t$. \n",
    "\n",
    "```\n",
    "  step m1s  tr.acc/#1s tst.acc/#1s xentropy  loss  |grad| gr-ang  learnrate \n",
    "     1  40   0.49  75   0.52  57   0.6931   0.6931  0.260   0.00   0.0100\n",
    "     6  40   0.66  47   0.57  36   0.6882   0.6882  0.253   1.00   0.0100\n",
    "    11  40   0.59  43   0.50  30   0.6823   0.6823  0.297   1.00   0.0096\n",
    "    16  40   0.60  44   0.48  31   0.6745   0.6745  0.334   1.00   0.0096\n",
    "    21  40   0.60  46   0.47  30   0.6653   0.6653  0.350   1.00   0.0092\n",
    "    26  40   0.62  48   0.43  32   0.6558   0.6558  0.354   1.00   0.0092\n",
    "    31  40   0.64  49   0.42  31   0.6463   0.6463  0.351   1.00   0.0088\n",
    "```\n",
    "It does seem to be learning the training data, one has a 1/161 chance of getting .64 accuracy at random ($Z$ score of 2.5), but it is not learning anything to help it with the test data\n",
    "\n",
    "\n",
    "### Job status\n",
    "Doing `bjobs -l` I see\n",
    "```\n",
    "Mon Feb 22 22:46:01: Started 1 Task(s) on Host(s) <psana1411>, Allocated 1 Slot\n",
    "                     (s) on Host(s) <psana1411>, \n",
    "\n",
    "Tue Feb 23 08:56:29: Resource usage collected.\n",
    "                     The CPU time used is 86926 seconds.\n",
    "                     MEM: 4.1 Gbytes;  SWAP: 4 Mbytes;  NTHREAD: 36\n",
    "                     PGID: 904;  PIDs: 904 977 979 \n",
    "\n",
    "\n",
    " MEMORY USAGE:\n",
    " MAX MEM: 6.7 Gbytes;  AVG MEM: 3.6 Gbytes\n",
    "```\n",
    "When I do top on psana1411, I see other jobs running, so I am not reserving slots correctly. I should add -n X. Here is a ganglia plot:\n",
    "\n",
    "![ganglia](psana1411_ganglia_first_scale_up.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.5043961348\n"
     ]
    }
   ],
   "source": [
    "# calculate a Z score\n",
    "acc = .64 #.42\n",
    "mu=40\n",
    "X=acc*80\n",
    "std=(.5*.5*80)**0.5\n",
    "Z = (X-mu)/std\n",
    "print Z\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsuccessful Scale Up Issues with XTCAV data\n",
    "\n",
    "Before dropping down to the small network, I was testing with \n",
    "\n",
    "* 2-3 convent layers\n",
    "* 2 fully connected layers\n",
    "* 'whitening', or basicaly GCN - global contrast normalization, each pixel over minibatch - mean=0, stddev=1\n",
    "* not whitening\n",
    "* high momentum, .9, .95, .99\n",
    "* different learning rates, up to .1, down to .0005 (I think)\n",
    "* different bias inits\n",
    "\n",
    "## minibatch and SGD\n",
    "\n",
    "* Tried to read 128 random images for each step\n",
    "* worried that all time was going to reading the images\n",
    "* dropped down to minibatch of 8\n",
    "* smaller minibatch - maybe not a good idea to whiten\n",
    "* accuracy was not improving\n",
    "* train steps still long\n",
    "\n",
    "## Swing between all 1s or 0s, image normalization\n",
    "\n",
    "* My networks often swing from predicting all class 0 or all class 1\n",
    "* this seems weird, proper random initalization and small learning rate should give me a balanced amount of 0's vs. 1's in the loss function?\n",
    "* Fiddled a lot with bias/weight initalization, learning rate, ect\n",
    "* after reading ch 13 of deeplearningbook.net, I think some image preprocessing is important - dynamic range of [-1,1] or [0,1], log transform, whitening, not sure if we should normalize by per pixel stddev\n",
    "* after this, inference function predicted more balanced 0s vs 1s during training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulated Data Scale Up\n",
    "\n",
    "Here we report on simulated data scale up\n",
    "\n",
    "## simulated data\n",
    "\n",
    "This is signal vs. noise\n",
    "\n",
    "### miniBatch\n",
    "```\n",
    "size=128 images \n",
    "img I, shape=(40,40)\n",
    "signal: 10+ vertical line - mean=5, stddev=.1\n",
    "noise: stddev=.1\n",
    "```\n",
    "\n",
    "## network\n",
    "a 4 layer feed forward network\n",
    "\n",
    "* whiten: `WI=ApproxWhiten(I)`   each pixel in miniBatch\n",
    "* architecture/loss\n",
    "\n",
    "```\n",
    "Layer    kernel   kstrides  bias nonlinear  lrn maxpool-ksize strides\n",
    "CVN01   (5,5,1,4)  (2,2)    Yes   relu     False  (3,3)      (2,2)\n",
    "CVN02   (5,5,4,3)  (2,2)    Yes   relu     False  (3,3)      (2,2)\n",
    "\n",
    "CVN01: (40,40) -> (20,20)\n",
    "CVN02(20,20).shape=(10,10)\n",
    "\n",
    "H03     W-shape=(100,5)\n",
    "H04     W-shape=(5,2)\n",
    "\n",
    "loss = avg(xentropy)  + 0.005 L1-norm(CVN01_K, CVN02_K, H03_W)\n",
    "```\n",
    "\n",
    "## scale up - more paremters\n",
    "\n",
    "Still 40 x 40 images, but more parameters:\n",
    "\n",
    "* momentum .9\n",
    "* learnrate.01\n",
    "* CVN01 kernel window (12,12) channels 20, strides (4,4)\n",
    "* CVN02 kernel window (8,8) channels 16   strides (3,3)\n",
    "* hidden units 10\n",
    "\n",
    "got to 100% accuracy in 130 steps.\n",
    "\n",
    "## scale up - larger image - 100 x 100\n",
    "\n",
    "* go from (40,40) image to (100,100) image\n",
    "* initialze biases as: \n",
    "\n",
    "```\n",
    "CVN01_B=.1\n",
    "CVN02_B=.1\n",
    "H03_B=.1\n",
    "H04_B=0\n",
    "```\n",
    "got 100% acc in 310 steps\n",
    "\n",
    "## scale up - larger image 200 x 200, more parameters\n",
    "\n",
    "* 307k parameters, 160k image (200,200)\n",
    "* CVN01: kernel window (32,32) channels 16, strides (2,2), relu, maxpool window=(4,4) strides=(3,3)\n",
    "* CVN01 output is (34,34)\n",
    "* CVN02 output is (6,6) with 16 channels\n",
    "* H03: 576 inputs, 50 outupts\n",
    "* H04: 2 outputs\n",
    "\n",
    "trained at 12 sec/step, got 100% on validation in 5 minutes (12 steps)\n",
    "\n",
    "Seemed to use 100GB mem on ganglia?\n",
    "\n",
    "## scale up to big - 1000 x 1000 image\n",
    "\n",
    "Went to 905,912 parameters, and 4MB images (1,000,000 pixels). Created 7GB of simulated data in 2 minutes, however ganglia reported memory usage went to swap. Details of arch:\n",
    "\n",
    "\n",
    "```\n",
    "tensorflow)psana1612: ~/condaDev/xtcav-mlearn/convnet $ bsub -q psnehq -x -I python convnet_app.py -c convnet_flags_big.py -t 32 -d 1000\n",
    "Warning: job being submitted without an AFS token.\n",
    "Job <7322> is submitted to queue <psnehq>.\n",
    "<<Waiting for dispatch ...>>\n",
    "<<Starting on psana1503.pcdsn>>\n",
    "SimpleImgData - about to produce 1780 (1000 x 1000) images (6790.16 MB)\n",
    "  made data in 100.02 sec\n",
    "I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 32\n",
    "I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 32\n",
    "('CVN01', 'CVN02')\n",
    "('H03', 'H04')\n",
    "evolving learning rate\n",
    "whitenedInput.shape=(128, 1000, 1000, 1)\n",
    "CVN01:\n",
    "             CVN_K.shape=(32, 32, 1, 16)\n",
    "          CVN_conv.shape=(128, 500, 500, 16)\n",
    "             CVN_B.shape=(16,)\n",
    "     CVN_nonlinear.shape=(128, 500, 500, 16)\n",
    "         CVN_pool.shape=(128, 167, 167, 16)\n",
    "             CVN_U.shape=(128, 167, 167, 16)\n",
    "CVN02:\n",
    "             CVN_K.shape=(32, 32, 16, 16)\n",
    "          CVN_conv.shape=(128, 84, 84, 16)\n",
    "             CVN_B.shape=(16,)\n",
    "     CVN_nonlinear.shape=(128, 84, 84, 16)\n",
    "         CVN_pool.shape=(128, 28, 28, 16)\n",
    "             CVN_U.shape=(128, 28, 28, 16)\n",
    "H03:\n",
    "   H_W.shape=(12544,50)\n",
    "   H_B.shape=(50,)\n",
    "H04:\n",
    "   H_W.shape=(50,2)\n",
    "   H_B.shape=(2,)\n",
    "convnet has 905912 unknown variables, 278560 (30%) in convnet layers, and 627352 (69%) in hidden layers.\n",
    "convnet maps 1000000 features to 2 outputs for hidden layers.\n",
    "initial loss=0.70\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}