OpenGrpahAU

This repo is the OpenGprahAU tool.

demo:

Models were traiend on hybrid dataset of 2,000k images.

This hybrid dataset includes:

• BP4D
• DISFA
• RAF-AU
• AFF-Wild 2
• CK+
• CASME II

The tool can predict action units of 41 categories:

AU1	AU2	AU4	AU5	AU6	AU7	AU9	AU10	AU11	AU12	AU13	AU14	AU15	AU16
Inner brow raiser	Outer brow raiser	Brow lowerer	Upper lid raiser	Cheek raiser	Lid tightener	Nose wrinkler	Upper lip raiser	Nasolabial deepener	Lip corner puller	Sharp lip puller	Dimpler	Lip corner depressor	Lower lip depressor

AU17	AU18	AU19	AU20	AU22	AU23	AU24	AU25	AU26	AU27	AU32	AU38	AU39	-
Chin raiser	Lip pucker	Tongue show	Lip stretcher	Lip funneler	Lip tightener	Lip pressor	Lips part	Jaw drop	Mouth stretch	Lip bite	Nostril dilator	Nostril compressor	-

AUL1	AUL1	AUL2	AUR2	AUL4	AUR4	AUL6	AUR6	AUL10	AUR10	AUL12	AUR12	AUL14	AUR14
Left inner brow raiser	Right inner brow raiser	Left outer brow raiser	Right outer brow raiser	Left brow lowerer	Right brow lowerer	Left cheek raiser	Right cheek raiser	Left upper lip raiser	Right upper lip raiser	Left nasolabial deepener	Right nasolabial deepener	Left dimpler	Right dimpler

We provide tools for prepareing data in tool/. After Downloading raw data files, you can use these tools to process them, aligning with our protocals. We divide the dataset into three independent parts (i.e., train, val, test).

Pretrained models

Hybrid Dataset

Stage1:

arch_type	GoogleDrive link	Average F1-score	Average Acc.
Ours (MobileNetV3)	-	-	-
Ours (ResNet-18)	link	22.33	92.97
Ours (ResNet-50)	link	22.52	92.63
Ours (Swin-Tiny)	link	22.66	92.97
Ours (Swin-Small)	link	24.49	92.84
Ours (Swin-Base)	link	23.53	92.91

Stage2:

arch_type	GoogleDrive link	Average F1-score	Average Acc.
Ours (MobileNetV3)	-	-	-
Ours (ResNet-18)	link	22.51	93.23
Ours (ResNet-50)	link	23.24	93.31
Ours (Swin-Tiny)	link	22.74	93.37
Ours (Swin-Small)	-	-	-
Ours (Swin-Base)	-	-	-

Demo

• to detect facial action units in a facial image using our stage1 model, run:

python demo.py --arc resnet50 --stage 1 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth --input demo_imgs/1014.jpg  --draw_text

• to detect facial action units in a facial image using our stage2 model, run:

python demo.py --arc resnet50 --stage 2 --exp-name demo --resume checkpoints/OpenGprahAU-ResNet50_second_stage.pth --input demo_imgs/1014.jpg  --draw_text

Training and Testing

• to train the first stage of our approach (ResNet-50) on hybrid Dataset, run:

python train_stage1.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_first_stage -b 512 -lr 0.00002  

• to test the first stage of our approach (SwinT) on hybrid Dataset, run:

python test_stage1.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_first_stage  --resume ./results/OpenGprahAU-SwinT_first_stage/bs_64_seed_0_lr_2e-05/best_model.pth

• to train the second stage of our approach (ResNet-50) on hybrid Dataset, run:

python train_stage2.py --arc resnet50 --exp-name OpenGprahAU-ResNet50_second_stage -b 512 -lr 0.00001  --resume checkpoints/OpenGprahAU-ResNet50_first_stage.pth

• to test the second stage of our approach (SwinT) on hybrid Dataset, run:

python test_stage2.py --arc swin_transformer_tiny --exp-name test_OpenGprahAU-SwinT_second_stage  --resume ./results/OpenGprahAU-SwinT_second_stage/bs_64_seed_0_lr_1e-05/best_model.pth