Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks by Bohdan Ivaniuk-Skulskyi, Andrew Garrett Kurbis, Alex Mihailidis, and Brokoslaw Laschowski

News

10/07/2024

• Our absrtact accepted to Toronto Robotics Conference

10/06/2024

• Our paper accepted to IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob) 2024

05/05/2023

• Our absrtact accepted to ICRA 2023 Computer Vision for Wearable Robotics Workshop

19/04/2023

• Our poster accepted to ICAIR 2023

Installation

Create virtual environment and run requirements.txt

python3 -m venv venv
source venv/bin/activate
pip install git+https://github.com/Atze00/MoViNet-pytorch.git
pip install -r requirements.txt

Install StairNet dataset and run preprocessing file

python data_preprocessing/dataset_preprocessing.py --data_folder /path-to-dataset-dir/

Define os environment variable pointing to it

export DATASET=/path-to-preprocessed-dataset-dir/

• unzip data_splits/train.txt file.

Model Zoo

Name	Parameters	GFLOPs	Resolution	Accuracy	F1-score	Download
MoViNet	4.03M	2.5	5x3x224x224	0.983	0.982	model	config
MobileViT-LSTM	3.36M	9.84	5x3x224x224	0.970	0.968	model	config
MobileNet-LSTM	6.08M	53.96	5x3x224x224	0.973	0.970	model	config
MobileNet-LSTM (seq2seq)	5.93M	50.97	5x3x224x224	0.707	0.799	model	config
Baseline (Kurbis et al.)	2.26M	0.61	3x224x224	0.972	0.972	-	-

Inference

Download one checkpoint with its configuration file and run the following command

python test.py --experiment_cfg CONFIG.yaml \
  --dataset_folder $DATASET \
  --val_samples_file data_splits/validation.txt \
  --test_samples_file data_splits/test.txt \ 
  --checkpoint_path CHECKPOINT.pth

Train

python train.py --experiment_cfg CONFIG.yaml \
  --dataset_folder $DATASET \
  --train_samples_file data_splits/train.txt \
  --val_samples_file data_splits/validation.txt \
  --test_samples_file data_splits/test.txt

Confusion matrices

BibTex

Sequential image classification of human-robot walking environments using temporal neural networks

@article {Ivanyuk-Skulskiy2023.11.10.566555,
  author       = {Ivanyuk-Skulskiy, Bogdan and Kurbis, Andrew Garrett and Mihailidis, Alex and Laschowski, Brokoslaw},
  title        = {Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks},
  elocation-id = {2023.11.10.566555},
  year         = {2023},
  doi          = {10.1101/2023.11.10.566555},
  publisher    = {Cold Spring Harbor Laboratory},
  URL          = {https://www.biorxiv.org/content/early/2023/11/13/2023.11.10.566555},
  eprint       = {https://www.biorxiv.org/content/early/2023/11/13/2023.11.10.566555.full.pdf},
  journal      = {bioRxiv}
}

StairNet: visual recognition of stairs for human–robot locomotion

@article{Kurbis2024,
  author       = {Andrew Garrett Kurbis and Dmytro Kuzmenko and Bogdan Ivanyuk-Skulskiy and Alex Mihailidis and Brokoslaw Laschowski},
  title        = {StairNet: visual recognition of stairs for human–robot locomotion},
  journal      = {BioMedical Engineering OnLine},
  year         = {2024},
  volume       = {23},
  number       = {1},
  pages        = {20},
  doi          = {10.1186/s12938-024-01216-0},
  url          = {https://doi.org/10.1186/s12938-024-01216-0},
  issn         = {1475-925X}
}

Acknowledgements

The visual encoder models are taken from timm library and the MoViNet is based on this implementation