Merge pull request #1 from cvjena/develop

Merge development changes into main branch

.gitignore

@@ -54,7 +54,6 @@ MANIFEST
 
 #jupyter
 .ipynb_checkpoints
-*.ipynb
 
 # app/data
 temp_results

README.md

@@ -1,89 +1,92 @@
 # electromyogram
 
-This is a small python package to create a Electromyogram (EMG) Intensity plots for facial muscles with facial structure.
-The current focus is on the solely facial muscles but it could be extended to other muscles, but this is not planned yet.
+![Teaser](files/teaser.jpg)
 
-We currently support the two following schematics for acquiring the EMG data:
+This Python package provides a convenient way to create an Electromyogram (EMG) Intensity plot specifically designed for facial muscles with facial structure. With this tool, you can visualize and analyze the intensity of EMG data collected from various facial muscles.
 
-- Fridlund and Cappacio () []
-- Kuramoto et al. () []
+A small demo is hosted [here](https://semg.inf-cv.uni-jena.de/), together with the tool [face-projection](https://github.com/cvjena/face-projection) for a projection onto the face.
 
-## Installation
+## Why use sEMG Intensity Plot?
+
+- **Easy to use**: The package provides a straightforward interface, making it accessible for users of all levels of expertise.
+- **Visualize muscle activity**: The EMG Intensity plot allows you to visualize the intensity of muscle activity over the face, providing insights into patterns and variations.
+- **Designed explicitly for facial** muscles**: The tool focuses on facial muscles, enabling you to study and understand muscle activity in the face, which can be particularly useful in fields like facial expression analysis, neuroscience, and rehabilitation.
+- **Potential for extension**: While the current focus is on facial muscles, this tool could potentially be extended to analyze other muscle groups.
+- **Beyond muscles**: The tool can also be used to plot additional facial information, such as oxygen saturation, but this is not officially supported yet.
 
-Currently the package is not available on PyPI.
-Thus, you have to install it from the source code.
-If you want to use the package in a virtual environment, you have to activate it first.
-Clone this repository and install it with pip:
+## Installation
 
+The package is available on [PyPI](https://pypi.org/project/electromyogram/) and can be installed with `pip`:
+
 ```bash
-git clone <link to this repository>
-cd electromyogram
-pip install .
+pip install electromyogram
 ```
 
-It is then available in your python environment, and you can import it with:
+If you want to install it locally to add changes, please close the repository and install it with `pip` in development mode.
+We assume you already created a virtual environment and activated it :)
 
-```python
-import electromyogram
+```bash
+git clone <link to this repository>
+cd electromyogram
+pip install -e .
 ```
 
 ## Usage
 
-We predefined the two schematics (Fridlund and Cappacio and Kuramoto et al.) on a 2D canvas.
-This canvas is based on the canonical face model used by Google in several of their projects (dominantly in *mediapipe*).
-All EMG sensor coordinates are given relatively to this canvas thus arbitrary canvas scaling is possible.
-We default to a canvas size of 4096x4096 pixels.
-
-The current process for electromyogram visualization is based on a 2 step method.
-First, we create the interpolation of the given EMG values on a 2D canvas for the chosen schematic.
-Second, we allow the application of different color maps to the interpolation to create the final plot.
+This tool is intended to simplify the creation of only the spatial intensity map for surface EMG data.
+All the required preprocessing of the data is not part of this package and is likely project-specific.
+We assume that the data is given in a dictionary (or pandas table) and the keys are the sensor locations.
 
-### Example 1: Fridlund and Cappacio
+Then, the correct physical interpolation between the sensors is done, and the result is a 2D array of the interpolated values on the canonical face model.
+You can then apply different color maps to the interpolation to create the final plot.
+Detailed examples with test data can be found in `examples/`.
 
-Our first example is based on the Fridlund and Cappacio schematic.
-We assume that the data is given in a dictionary and the keys are the sensor locations.
-Note: We only support a subset of the sensors in the Fridlund and Cappacio schematic. (TODO add list of supported sensors)
-
-The following locations are expected and then interpolated:
-![Locations ](files/locations_fridlund.png)
 
 ```python
 import electromyogram as emg
 
 # we assume that the data is given in a dictionary and the keys are the sensor locations
 data_values = dict(...)
 
+scheme = emg.Fridlund()  # or emg.Kuramoto()
 # create the interpolation
-interpo = emg.interpolate(emg.Fridlund, data_values)
-myogram = emg.colorize(interpo, cmap='viridis')
+powermap = emg.interpolate(scheme, data_values, shape=(1024, 1024))
+powermap = emg.colorize(powermap, cmap='viridis')
 ```
 
-### Example 2: Kuramoto et al.
-Our second example is based on the Kuramoto et al. schematic.
-We assume that the data is given in a dictionary and the keys are the sensor locations.
-Note: We only support a subset of the sensors in the Kuramoto et al. schematic. (TODO add list of supported sensors)
+For the colorization, the users can use any color map from [matplotlib](https://matplotlib.org/stable/tutorials/colors/colormaps.html) or [pallettable](https://jiffyclub.github.io/palettable/) (e.g., `pallettable.scientific.sequential.Imola_20`)
+![Colors](files/colorization.jpg)
 
-The following locations are expected and then interpolated:
-![Locations ](files/locations_kuramoto.png)
+## Surface EMG Schematics
 
-```python
-import electromyogram as emg
+We currently support the two following schematics for acquiring the EMG data.
+If you want to have your own, please open an issue or create a pull request, and we will be happy to add it.
 
-# we assume that the data is given in a dictionary and the keys are the sensor locations
-data_values = dict(...)
-# create the interpolation
-interpo = emg.interpolate(emg.Kuramoto, data_values)
-myogram = emg.colorize(interpo, cmap='viridis')
-```
+| [Fridlund and Cappacio, 1986](https://pubmed.ncbi.nlm.nih.gov/3809364/) | [Kuramoto et al., 2019](https://onlinelibrary.wiley.com/doi/10.1002/npr2.12059) |
+|---|---|
+| ![Locations ](files/locations_fridlund.jpg)  |  ![Locations ](files/locations_kuramoto.jpg) |
+
+If you want to define your own scheme, just create a new class that inherits from `emg.Schematic` and implement the `get_sensor_locations` function.
+Then use it in the `interpolate` function, and you are good to go.
 
 ## Todos
 
-- [ ] Handle if not all values are given for a schematic better
-- [ ] Add result images
+- [ ] Handle if not all values are given for a better schematic
+- [X] Add result images
+- [X] Add a function to draw triangulation onto the 2D canvas
+- [ ] Add a function to draw sensor locations onto the 2D canvas
+- [X] Add the option to remove the area outside the canonical face model
+- [ ] Make a better interface for the channel names
+- [ ] Add function to create the according colorbar for matplotlib in the correct size
+
 ## License
 
+MIT License
 
-## References
+## Citation
 
+If you use our work, please cite us:
 
-## Acknowledgements
+```bibtex
+under publication, please wait a bit :)
+```

examples/data/fridlund.csv

@@ -0,0 +1,7 @@
+            ,DAO li ,OrbOr li ,Ment li ,Mass li ,Zyg li ,Llsup li ,OrbOc li ,lat Front li ,med Front li ,Corr li ,Deprsup li ,DAO re ,OrbOr re ,Ment re ,Mass re ,Zyg re ,Llsup re ,OrbOc re ,lat Front re ,med Front re ,Corr re ,Deprsup re
+angry       ,  9.87 ,   21.85 ,  16.80 ,   1.41 ,  1.10 ,    0.00 ,    6.89 ,        1.31 ,        0.93 ,   2.44 ,      2.74 ,  0.00 ,   39.11 ,  17.33 ,   2.04 ,  1.63 ,    0.00 ,    9.51 ,        0.71 ,        0.82 ,   2.27 ,      2.07
+suprised    ,  0.85 ,    0.79 ,   1.21 ,   0.46 ,  0.45 ,    0.61 ,    1.56 ,        4.71 ,        3.51 ,   2.26 ,      2.93 ,  0.73 ,    0.80 ,   1.12 ,   0.51 ,  0.52 ,    0.92 ,    3.01 ,        1.67 ,        3.95 ,   2.65 ,      2.46
+sad         ,  0.98 ,    1.23 ,  28.19 ,   0.63 ,  0.72 ,    3.49 ,    2.22 ,        3.09 ,        6.00 ,   7.91 ,      7.16 ,  1.00 ,    1.47 ,  28.97 ,   0.76 ,  0.80 ,    2.51 ,    3.41 ,        1.08 ,        6.72 ,   8.42 ,      7.04
+fearful     ,  8.68 ,    0.00 ,  11.20 ,   0.86 ,  0.92 ,    1.83 ,    1.58 ,        0.00 ,        5.65 ,   3.11 ,      2.99 ,  4.60 ,   11.33 ,   6.70 ,   1.11 ,  0.96 ,    2.05 ,    4.13 ,        1.60 ,        6.46 ,   3.68 ,      3.85
+happy       ,  3.14 ,   18.30 ,   8.37 ,   0.70 ,  0.00 ,    3.71 ,    6.05 ,        0.72 ,        0.65 ,   0.52 ,      0.33 ,  0.00 ,   15.04 ,  10.23 ,   0.92 ,  0.00 ,    3.31 ,    7.41 ,        0.60 ,        0.62 ,   0.58 ,      0.55
+distugusted ,  2.46 ,    8.31 ,   4.05 ,   0.77 ,  0.95 ,    1.79 ,    7.78 ,        3.65 ,        2.55 ,  10.57 ,      8.50 ,  1.90 ,    6.01 ,   3.92 ,   0.72 ,  0.81 ,    1.83 ,    8.97 ,        0.97 ,        2.39 ,  10.00 ,      8.13

examples/data/kuramoto.csv

@@ -0,0 +1,8 @@
+
+,E1,E3,E5,E7,E9,E13,E15,E17,E2,E4,E6,E8,E10,E14,E16,E18,E19,E20,E24
+angry,4.10,6.05,5.98,4.73,11.44,2.57,2.98,2.76,3.81,4.21,12.49,5.63,10.74,4.04,3.59,3.46,7.97,0.00,5.99
+suprised,6.94,6.13,3.45,3.64,5.18,3.27,3.39,3.70,6.29,5.71,3.20,3.41,4.47,3.38,3.36,3.80,4.53,3.75,3.48
+sad,5.17,8.45,3.97,4.33,17.21,0.00,4.33,4.54,4.74,5.57,3.67,4.62,17.74,9.39,3.76,4.14,7.13,4.90,5.83
+fearful,9.13,9.07,3.77,4.78,0.00,6.87,4.24,6.63,10.26,7.80,3.38,4.67,24.93,6.23,3.90,5.92,5.19,5.09,3.10
+happy,1.82,2.55,3.28,3.35,0.00,3.77,3.16,6.41,1.66,2.12,4.39,3.32,0.00,3.07,4.48,7.58,2.25,4.30,2.27
+distugusted,8.89,8.90,6.04,5.28,6.39,6.53,4.20,3.76,10.34,7.18,12.29,5.84,5.65,10.30,5.06,4.10,10.73,5.75,9.14

setup.cfg

@@ -12,10 +12,10 @@ license = MIT
 license_files = LICENSE.txt
 long_description = file: README.md
 long_description_content_type = text/markdown; charset=UTF-8; variant=GFM
-url = https://github.com/pyscaffold/pyscaffold/
+url = https://inf-cv.uni-jena.de/home/research/learning3d/facial-paresis-analysis/
 project_urls =
     Source = https://github.com/cvjena/electromyogram
-    Tracker = https://github.com/cvjena/electromyogramissues
+    Tracker = https://github.com/cvjena/electromyogram/issues
 
 # Change if running only on Windows, Mac or Linux (comma-separated)
 platforms = any
@@ -47,7 +47,7 @@ install_requires =
     scipy>=1.8,<2
     palettable>3.3
     matplotlib>=3.5,<4
-
+    h5py>=3.10
 [options.packages.find]
 where = src
 exclude =

src/electromyogram/__init__.py

@@ -7,6 +7,7 @@
     "colorize",
     "get_colormap",
     "annotate_locations",
+    "postprocess"
 ]
 
 from electromyogram.plot import (
@@ -15,6 +16,7 @@
     get_colormap,
     interpolate,
     plot_locations,
+    postprocess,
 )
 
 from electromyogram.schemes import Fridlund, Kuramoto, Scheme

src/electromyogram/plot.py

@@ -1,8 +1,12 @@
-__all__ = ["interpolate", "plot_locations", "colorize", "get_colormap", "annotate_locations"]
+__all__ = ["interpolate", "plot_locations", "colorize", "get_colormap", "annotate_locations", "postprocess"]
 
+
+from dataclasses import dataclass
 from typing import Optional, Type, Union
+from pathlib import Path
 
 import cv2
+import h5py
 import numpy as np
 from matplotlib import pyplot as plt
 from scipy import interpolate as interp
@@ -11,6 +15,24 @@
 from .schemes import DEFAULT_SIZE_PX, Scheme
 from .utils import rel_to_abs
 
+@dataclass
+class FaceModel:
+    points: np.ndarray
+    triangles: np.ndarray
+    facets: np.ndarray
+    masking: np.ndarray
+
+face_model_data = h5py.File(Path(__file__).parent / "face_model.h5", "r")
+
+face_model = FaceModel(
+    points=np.array(face_model_data["points"]),
+    triangles=np.array(face_model_data["triangles"]),
+    facets=np.array(face_model_data["facets"]),
+    masking=np.array(face_model_data["masking_canonical"]),
+)
+
+face_model_data.close()
+
 def annotate_locations(
     ax: plt.Axes,
     scheme: Scheme,
@@ -177,8 +199,7 @@ def __interpolate(
         The maximum value of the EMG values. Defaults to None and will be set to the maximum value of the EMG values.
     """
 
-    if not scheme.valid(emg_values):
-        raise ValueError("Either missing or invalid EMG keys/values in dict")
+    scheme.valid(emg_values) # this raises a ValueError if the values are not valid
 
     canvas = np.zeros(shape, dtype=np.float32)
     keys_sorted_semg = sorted(scheme.locations.keys())
@@ -297,10 +318,26 @@ def colorize(
     # scale the values to the range [0, 255]
     interpolation = (interpolation * 255).astype(np.uint8)
     colored = apply_colormap(interpolation, get_colormap(cmap))
-
-    # if white_background:
-    #     size = colored.shape[:2]
-    #     coords_scaled = ((np.array(consts.FACE_COORDS) / 4096) * size[0]).astype(np.int32)
-    #     mask = cv2.fillConvexPoly(np.zeros(size), cv2.convexHull(coords_scaled), 1)
-    #     colored = np.where(mask[..., None] == 0, np.full_like(colored, fill_value=[255, 255, 255]), colored)
     return colored
+
+def postprocess(
+    powermap: np.ndarray,
+    remove_outer: bool = True,
+    draw_triangle: bool = True,
+    triangles_alpha: float = 0.2,
+) -> np.ndarray:
+    # scale the points to the current shape
+    points = (face_model.points * powermap.shape[0]).astype(np.int32)
+
+    if draw_triangle:
+        color = 255 if powermap.ndim != 3 else (255, 255, 255)
+        lines = np.zeros_like(powermap)
+        lines = cv2.polylines(lines, [points[tri] for tri in face_model.triangles], True, color, thickness=1)
+        powermap = cv2.addWeighted(powermap, 1-triangles_alpha, lines, triangles_alpha, 0)
+
+    if remove_outer:
+        hull = cv2.convexHull(points, returnPoints=True)
+        mask = cv2.drawContours(np.zeros(powermap.shape[:2]), [hull], 0, 1, -1)
+        powermap[mask == 0] = 255 if powermap.ndim != 3 else [255, 255, 255]
+
+    return powermap

src/electromyogram/schemes.py

@@ -42,12 +42,10 @@ def valid(self, emg_values: dict[str, float]) -> bool:
         temp_locations = self.locations.copy()
         for emg_name, emg_value in emg_values.items():
             if emg_name not in temp_locations:
-                return False
+                raise ValueError(f"EMG name {emg_name} not in locations.")
             if not self._check_value(emg_value):
-                return False
+                raise ValueError(f"EMG value {emg_value} for {emg_name} is not valid.")
             del temp_locations[emg_name]
-        return True
-
     def _check_value(self, emg_value: float) -> bool:
         return emg_value >= 0