MiVOLO: Multi-input Transformer for Age and Gender Estimation

MiVOLO: Multi-input Transformer for Age and Gender Estimation, Maksim Kuprashevich, Irina Tolstykh, 2023 arXiv 2307.04616

Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation, Maksim Kuprashevich, Grigorii Alekseenko, Irina Tolstykh 2024 arXiv 2403.02302

[Paper 2023] [Paper 2024] [Demo] [Telegram Bot] [BibTex] [Data]

MiVOLO pretrained models

Gender & Age recognition performance.

Model	Type	Dataset (train and test)	Age MAE	Age CS@5	Gender Accuracy	download
volo_d1	face_only, age	IMDB-cleaned	4.29	67.71	-	checkpoint
volo_d1	face_only, age, gender	IMDB-cleaned	4.22	68.68	99.38	checkpoint
mivolo_d1	face_body, age, gender	IMDB-cleaned	4.24 [face+body] 6.87 [body]	68.32 [face+body] 46.32 [body]	99.46 [face+body] 96.48 [body]	model_imdb_cross_person_4.24_99.46.pth.tar
volo_d1	face_only, age	UTKFace	4.23	69.72	-	checkpoint
volo_d1	face_only, age, gender	UTKFace	4.23	69.78	97.69	checkpoint
mivolo_d1	face_body, age, gender	Lagenda	3.99 [face+body]	71.27 [face+body]	97.36 [face+body]	demo
mivolov2_d1_384x384	face_body, age, gender	Lagenda	3.65 [face+body]	74.48 [face+body]	97.99 [face+body]	telegram bot

MiVOLO regression benchmarks

Gender & Age recognition performance.

Use valid_age_gender.sh to reproduce results with our checkpoints.

Model	Type	Train Dataset	Test Dataset	Age MAE	Age CS@5	Gender Accuracy	download
mivolo_d1	face_body, age, gender	Lagenda	AgeDB	5.55 [face]	55.08 [face]	98.3 [face]	demo
mivolo_d1	face_body, age, gender	IMDB-cleaned	AgeDB	5.58 [face]	55.54 [face]	97.93 [face]	model_imdb_cross_person_4.24_99.46.pth.tar

MiVOLO classification benchmarks

Gender & Age recognition performance.

Model	Type	Train Dataset	Test Dataset	Age Accuracy	Gender Accuracy
mivolo_d1	face_body, age, gender	Lagenda	FairFace	61.07 [face+body]	95.73 [face+body]
mivolo_d1	face_body, age, gender	Lagenda	Adience	68.69 [face]	96.51[face]
mivolov2_d1_384	face_body, age, gender	Lagenda	Adience	69.43 [face]	97.39[face]

Dataset

Please, cite our papers if you use any this data!

• Lagenda dataset: images and annotation.

• IMDB-clean: follow these instructions to get images and download our annotations.

• UTK dataset: origin full images and our annotation: split from the article, random full split.

• Adience dataset: follow these instructions to get images and download our annotations.

  Click to expand!

  After downloading them, your data directory should look something like this:

  data
  └── Adience
      ├── annotations  (folder with our annotations)
      ├── aligned      (will not be used)
      ├── faces
      ├── fold_0_data.txt
      ├── fold_1_data.txt
      ├── fold_2_data.txt
      ├── fold_3_data.txt
      └── fold_4_data.txt

  We use coarse aligned images from faces/ dir.

  Using our detector we found a face bbox for each image (see tools/prepare_adience.py).

  This dataset has five folds. The performance metric is accuracy on five-fold cross validation.

  images before removal	fold 0	fold 1	fold 2	fold 3	fold 4
  19,370	4,484	3,730	3,894	3,446	3,816

  Not complete data

  only age not found	only gender not found	SUM
  40	1170	1,210 (6.2 %)

  Removed data

  failed to process image	age and gender not found	SUM
  0	708	708 (3.6 %)

  Genders

  female	male
  9,372	8,120

  Ages (8 classes) after mapping to not intersected ages intervals

  0-2	4-6	8-12	15-20	25-32	38-43	48-53	60-100
  2,509	2,140	2,293	1,791	5,589	2,490	909	901

• FairFace dataset: follow these instructions to get images and download our annotations.

  Click to expand!

  After downloading them, your data directory should look something like this:

  data
  └── FairFace
     ├── annotations  (folder with our annotations)
     ├── fairface-img-margin025-trainval   (will not be used)
         ├── train
         ├── val
     ├── fairface-img-margin125-trainval
         ├── train
         ├── val
     ├── fairface_label_train.csv
     ├── fairface_label_val.csv
  

  We use aligned images from fairface-img-margin125-trainval/ dir.

  Using our detector we found a face bbox for each image and added a person bbox if it was possible (see tools/prepare_fairface.py).

  This dataset has 2 splits: train and val. The performance metric is accuracy on validation.

  images train	images val
  86,744	10,954

  Genders for validation

  female	male
  5,162	5,792

  Ages for validation (9 classes):

  0-2	3-9	10-19	20-29	30-39	40-49	50-59	60-69	70+
  199	1,356	1,181	3,300	2,330	1,353	796	321	118

• AgeDB dataset: follow these instructions to get images and download our annotations.

  Click to expand!

  Ages: 1 - 101

  Genders: 9788 faces of M, 6700 faces of F

  images 0	images 1	images 2	images 3	images 4	images 5	images 6	images 7	images 8	images 9
  1701	1721	1615	1619	1626	1643	1634	1596	1676	1657

  Data splits were taken from here

  !! All splits(all dataset) were used for models evaluation.

Install

Install pytorch 1.13+ and other requirements.

pip install -r requirements.txt
pip install .

Demo

1. Download body + face detector model to models/yolov8x_person_face.pt
2. Download mivolo checkpoint to models/mivolo_imbd.pth.tar

wget https://variety.com/wp-content/uploads/2023/04/MCDNOHA_SP001.jpg -O jennifer_lawrence.jpg

python3 demo.py \
--input "jennifer_lawrence.jpg" \
--output "output" \
--detector-weights "models/yolov8x_person_face.pt " \
--checkpoint "models/mivolo_imbd.pth.tar" \
--device "cuda:0" \
--with-persons \
--draw

To run demo for a youtube video:

python3 demo.py \
--input "https://www.youtube.com/shorts/pVh32k0hGEI" \
--output "output" \
--detector-weights "models/yolov8x_person_face.pt" \
--checkpoint "models/mivolo_imbd.pth.tar" \
--device "cuda:0" \
--draw \
--with-persons

Validation

To reproduce validation metrics:

1. Download prepared annotations for imbd-clean / utk / adience / lagenda / fairface.
2. Download checkpoint
3. Run validation:

python3 eval_pretrained.py \
  --dataset_images /path/to/dataset/utk/images \
  --dataset_annotations /path/to/dataset/utk/annotation \
  --dataset_name utk \
  --split valid \
  --batch-size 512 \
  --checkpoint models/mivolo_imbd.pth.tar \
  --half \
  --with-persons \
  --device "cuda:0"

Supported dataset names: "utk", "imdb", "lagenda", "fairface", "adience".

Changelog

CHANGELOG.md

ONNX and TensorRT export

As of now (11.08.2023), while ONNX export is technically feasible, it is not advisable due to the poor performance of the resulting model with batch processing. TensorRT and OpenVINO export is impossible due to its lack of support for col2im.

If you remain absolutely committed to utilizing ONNX export, you can refer to these instructions.

The most highly recommended export method at present is using TorchScript. You can achieve this with a single line of code:

torch.jit.trace(model)

This approach provides you with a model that maintains its original speed and only requires a single file for usage, eliminating the need for additional code.

License

Please, see here

Citing

If you use our models, code or dataset, we kindly request you to cite the following paper and give repository a ⭐

@article{mivolo2023,
   Author = {Maksim Kuprashevich and Irina Tolstykh},
   Title = {MiVOLO: Multi-input Transformer for Age and Gender Estimation},
   Year = {2023},
   Eprint = {arXiv:2307.04616},
}

@article{mivolo2024,
   Author = {Maksim Kuprashevich and Grigorii Alekseenko and Irina Tolstykh},
   Title = {Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation},
   Year = {2024},
   Eprint = {arXiv:2403.02302},
}