updated validation page

build/html/_sources/human.rst.txt

@@ -35,7 +35,3 @@ After going through every clip, a csv will be updated of the events, which will
 The ``ManualReview`` column is then saves whether the event was correctly detected as ``True`` or ``False`` for further analysis.
 
 
-
-
-
-

build/html/_sources/intro.rst.txt

@@ -15,7 +15,7 @@ Before starting, there are **3 main steps** to follow:
 
 2. You need to clone/ download the `SORT repository <https://github.com/abewley/sort>`_, and put the folder inside the ``Repositories/`` folder.
 
-3. If you would like to work alongside the examples, also consider downloading the `Example Data <>`_ and putting it under ``Data/`` folder.
+3. If you would like to work alongside the examples, also consider downloading the `Example Data <https://doi.org/10.17617/3.EZNKYV>`_ and putting it under ``Data/`` folder.
 
 
 |start-h1| Installation |end-h1|

build/html/_sources/validation.rst.txt

@@ -1,17 +1,116 @@
 Validation and grid search
-==================
+===========================
 
-.. Here, I will introduce the validation and gridseach workflows, which was used to calculate all the evaluation metrics and optimize hyper parameters for the final inference. These steps highly relates to the type of data/ manual annotation you have, so will differ between each system.
+Here, I will introduce the validation and grid search workflows. 
+This will be different for each study system, and will depend on the format of the manual annotation you have. 
 
+This page would be slightly different from the other pages in the documentation, as I will give general guidance on generating evaluation metrics within python, so that you can readily adopt this to your own system.
+As with other pages, I will provide sample code for the Siberian Jay system.
 
-|start-h1| PAGE UNDER CONSTRUCTION |end-h1|
+|start-h1| Running the sample code |end-h1|
 
-Here, I will introduce the validation and grid search workflows. This will be different for each study system, and will depend on the format of the manual annotation you have. 
-I will expand this page more in the future, but for now, you can refer to ``Code/6_SampleValidation.py`` and ``./Code/7_SampleGridSearch.py`` for sample code for running validation and the grid search algorithms in the Siberian Jay system.
+If you would just like to try out the sample code, here is how you can run it for the Siberian jay system:
 
-If you are in need of urgent advice, you can shoot me an email :\)
+.. code-block:: python
 
-This page will be written out in the coming weeks.
+    python Code/6_SampleValidation.py
+
+
+Now, I will try to breakdown the validation procedure using the sample script for the Siberian Jay dataset.
+The overall aim is to create two python lists, one as the ground truth, and one as the predicted classes. 
+Then, we can use the `sklearn` library to calculate all kinds of evaluation metrics.
+How the ground truth and predicted classes are matched up will depend on the format of the manual annotation you have, that's why this step will require some data wrangling.
+
+
+Here, I defined the two empty lists before looping through the videos to get predictions.
+Throughout the script, I will then populate these lists with the ground truth and predicted classes.
+
+.. literalinclude:: 6_SampleValidation.py
+   :lines: 24-34
+   :linenos:
+   :lineno-start: 24
+   :language: python
+   :emphasize-lines: 5-6
+
+Then for each video, I loaded the detection pickle file (See :ref:`inference` for more details), then used the SORT tracker to get behavioural events.
+
+.. literalinclude:: 6_SampleValidation.py
+   :lines: 35-71
+   :linenos:
+   :lineno-start: 35
+   :language: python
+
+This results in a list of bounding boxes and tracking IDs for each frame in the video. We will use this list to match up with the ground truth later.
+
+Next, we will load the manual annotation from BORIS. 
+The BORIS data is annotated as absolute time, so we need to multiply it by the frame rate (25 here) to get back frame number.
+We then populate a list with the same length as the predicted list, then count the number of individuals "feeding" for a given frame
+
+.. literalinclude:: 6_SampleValidation.py
+   :lines: 74-86
+   :linenos:
+   :lineno-start: 74
+
+
+Next, we will match up the ground truth and predicted classes, but with a certain time window.
+This would be different for different kind of data, we refer to the original publication for discussion on this.
+For the typical cases, perhaps you only have to match up whether a predicted class was correctly identifying an event, but here because of the slight mismatch in manual annotation and the automated method, we need to summarize datapoints into time windows.
+
+.. literalinclude:: 6_SampleValidation.py
+   :lines: 88-96
+   :linenos:
+   :lineno-start: 88
+
+Finally, we will loop through the time window lists and populate the global prediction and ground truth lists.
+
+.. literalinclude:: 6_SampleValidation.py
+   :lines: 97-114
+   :linenos:
+   :lineno-start: 97
+
+
+Now with the ground truth and predicted classes, we can use the `sklearn` library to calculate all kinds of evaluation metrics.
+
+.. literalinclude:: 6_SampleValidation.py
+   :lines: 116-150
+   :linenos:
+   :lineno-start: 116
+
+The code above will calculate the precision, recall, F1 score, and confusion matrix for the two lists of predicted and ground truth classes.
+
+
+|start-h1| Grid search |end-h1|
+
+After figuring out the validation pipeline and obtaining metrics, you can then proceed to grid search to optimize hyperparameters of the SORT tracker.
+This step is not strictly necessary, but is just a standardize way of making sure everything is optimized for your system.
+
+You can run this in the terminal for the Siberian jay dataset:
+
+.. code-block:: python
+
+    python Code/7_SampleGridSearch.py
+
+
+The core of the script is essentially the same as the validation script, but the difference is that here, we loop through the validation a lot of times to test out all cominbations of hyperparameters.
+
+.. literalinclude:: 7_SampleGridSearch.py
+   :lines: 151-158
+   :linenos:
+   :lineno-start: 151
+
+Here is where we define the range of parameters we want to explore in the ``Code/7_SampleGridSearch.py`` script.
+The script will then loop through the combinations of these paramters and save it as a csv.
+
+We do note that the validation function can be essentially the same as the one used for validation above, except for a small change at the end to save the validation metrics instead of printing it.
+
+.. literalinclude:: 7_SampleGridSearch.py
+   :lines: 124-129
+   :linenos:
+   :lineno-start: 124
+
+After running the script, the results will be saved in a csv file, then you can choose your best hyper parameters!
+
+Hope this whole workflow was clear, and here is the whole YOLO-Behaviour pipeline! If you have any questions, feel free to contact me at ``hoi-hang.chan[at]uni-konstanz.de``
 
 
 .. |start-h1| raw:: html

build/html/intro.html

@@ -91,7 +91,7 @@
 </div></blockquote>
 </li>
 <li><p>You need to clone/ download the <a class="reference external" href="https://github.com/abewley/sort">SORT repository</a>, and put the folder inside the <code class="docutils literal notranslate"><span class="pre">Repositories/</span></code> folder.</p></li>
-<li><p>If you would like to work alongside the examples, also consider downloading the <a href="#id1"><span class="problematic" id="id2">`Example Data &lt;&gt;`_</span></a> and putting it under <code class="docutils literal notranslate"><span class="pre">Data/</span></code> folder.</p></li>
+<li><p>If you would like to work alongside the examples, also consider downloading the <a class="reference external" href="https://doi.org/10.17617/3.EZNKYV">Example Data</a> and putting it under <code class="docutils literal notranslate"><span class="pre">Data/</span></code> folder.</p></li>
 </ol>
 <p><h1> Installation </h1></p>
 <p>Before starting, I recommend downloading  <a class="reference external" href="https://www.anaconda.com/download/success">Anaconda</a> to create a virtual environment for the project. After downloading anaconda, you will need launch anaconda prompt (if you are on windows) or just your terminal (for mac and linux). This will be where you run all of the code required in the rest of the demo.</p>