All about Deepfakes

Catalogue

• 1 Deepfakes社会性发展调研报告
• 2 Deepfakes Datasets
• 3 Deepfake Creation
  • 3.1 Face-swapping methods
  • 3.2 Face reenactment methods
• 4 Deepfake Detection
  • 4.1 Fake image detection
  • 4.2 Fake video detection
    • 4.2.1 Temporal features across frames based ones
    • 4.2.2 Visual artifacts within video frame based methods

1 Deepfakes survey

• 2018 arxiv DeepFakes: a New Threat to Face Recognition? Assessment and Detection
• 2019 arXiv Deep Learning for Deepfakes Creation and Detection
• 2020 arXiv DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection
• 2020 arXiv Media Forensics and DeepFakes: an overview
• 2021 ACM Computing Surveys The Creation and Detection of Deepfakes: A Survey
• 荷兰初创公司Deeptrace发布两个年度（2018年和2019年）Deepfakes发展状况调研报告

2 Deepfakes Datasets

Datasets	Year	Ratio tampered:original	Total videos	Source	Participants Consent	Tools
UADFV	2018	1 : 1.00	98	YouTube	N	FakeAPP
FaceForensics	2018	1 : 1.00	2008	YouTube	N	Face2Face
Deepfake-TIMIT	2019	1 : 1.00	620	Vid-TIMIT	N	faceswap-GAN
FaceForensics++	2019	1 : 0.25	5000	YouTube	N	faceswap DeepFake Face2Face NeuralTextures
DeepFakeDetection (part of FaceForensics++)	2019	1 : 0.12	3363	Actors	Y
Celeb-DF	2019	1 : 0.51	1203	YouTube	N	a refined version of the DeepFake
DFDC Preview Dataset	2019	1 : 0.28	5214	Actors	Y	Unkonwn
KoDF	2021
Face Forensics in the Wild	2021 CVPR

3 Deepfake Creation

3.1 Face-swapping methods

• Open-source code

  • Deepfakes [Star27.6k]
  • Faceswap-GAN [Star2.5k]
  • FaceSwap [Star300+]
  • DeepFaceLab [Star11.1k]
  • DFaker [Star300+]
  • DeepFake-tf [Star<10]

• Paper

Year	Publications	Authors
2017 ICCV	Fast Face-Swap Using Convolutional Neural Networks	Iryna Korshunova, Wenzhe Shi, Joni Dambre, Lucas Theis
2018 IEEE FG	On Face Segmentation, Face Swapping, and Face Perception	Yuval Nirkin; Iacopo Masi; Anh Tran Tuan; Tal Hassner; Gerard Medioni
2019 ICCV	FSGAN: Subject Agnostic Face Swapping and Reenactment	Yuval Nirkin, Yosi Keller, Tal Hassner
2020 CVPR(Oral)	FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping	Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen

3.2 Face reenactment methods

Year	Publications
2014 ICPR	Face Anti-spoofing Based on General Image Quality Assessment
2015 TIFS	Face spoof detection with image distortion analysis.
2016 CVPR	Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
2018 TOG	HeadOn: Real-time Reenactment of Human Portrait Videos
2017 CVPR	Lip reading sentences in the wild
2017 TOG	Synthesizing Obama: learning lip sync from audio
2017 TOG	Bringing portraits to life
2018 EUSIPCO	Speaker inconsistency detection in tampered video
2018 TOG	Deep video portraits
2018 arxiv	Generative adversarial talking head: Bringing portraits to life with a weakly supervised neural network
2019 ICCV	Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
2019 TOG	Deferred Neural Rendering: Image Synthesis using Neural Textures
2019 AAAI Oral MMLAB-CUHK	Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

4 Deepfake Detection

4.1 Fake image detection

Year	Publications	Authors
2017 ICSIP	Automated face swapping and its detection	Ying Zhang; Lilei Zheng; Vrizlynn L. L. Thing
2018 IHMS Workshop	Fake Faces Identification via Convolutional Neural Network	Huaxiao Mo, Bolin Chen, Weiqi Luo
2018 MIPR	Detection of GAN-Generated Fake Images over Social Networks	Francesco Marra; Diego Gragnaniello; Davide Cozzolino; Luisa Verdoliva
2018 ISCCC	Learning to Detect Fake Face Images in the Wild	Chih-Chung Hsu; Chia-Yen Lee; Yi-Xiu Zhuang
2019 CCBR	On the Generalization of GAN Image Forensics
2019 aXiv	Limits of Deepfake Detection: A Robust Estimation Viewpoint	Sakshi Agarwal, Lav R. Varshney
2019 MDPI AG	Deep Fake Image Detection based on Pairwise Learning	Chih-Chung Hsu, Yi-Xiu Zhuang and Chia-Yen Lee
2019 IEEE BTAS	Multi-task learning for detecting and segmenting manipulated facial images and videos	Huy H. Nguyen; Fuming Fang; Junichi Yamagishi; Isao Echizen
2021 CVPR	Multi-attentional Deepfake Detection	Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, Nenghai Yu
2021 CVPR	Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain	Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, Nenghai Yu
2021 CVPR	Improving the Efficiency and Robustness of Deepfakes Detection through Precise Geometric Features	Zekun Sun, Yujie Han, Zeyu Hua, Na Ruan, Weijia Jia
2021 CVPR	Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection	Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, Yongdong Zhang
2021 CVPR	MagDR: Mask-guided Detection and Reconstruction for Defending Deepfakes	Zhikai Chen, Lingxi Xie, Shanmin Pang, Yong He, Bo Zhang

4.2 Fake video detection

4.2.1 Temporal features across frames based ones

Year	Publications	Classifiers/ Techniques
2018 AVSS	Intra-frame and temporal inconsistencies	CNN and LSTM
2018 WIFS	In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking	LRCN
2019 CVPR Workshop	Recurrent Convolutional Strategies for Face Manipulation Detection in Videos	RCN

4.2.2 Visual artifacts within video frame based methods

Year	Publications	Classifiers/ Techniques
2017 CVPR Workshop	Two-Stream Neural Networks for Tampered Face Detection	Two-Stream
2018 WIFS	MesoNet: a Compact Facial Video Forgery Detection Network	CNN
2018 IMVIP	Detection of Deepfake Video Manipulation	PRNU
2019 CVPR Workshop	Exposing DeepFake Videos By Detecting Face Warping Artifacts	VGG16 ResNet50, 101 or 152
2019 ICASSP	Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos	Capsule networks
2019 ICASSP	Exposing Deep Fakes Using Inconsistent Head Poses	SVM
2019 WACVW	Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations	Logistic regression and neural network
2019 IEEE Access	Combating Deepfake Videos Using Blockchain and Smart Contracts	Blockchain
2019 CVPR Workshops	Protecting World Leaders Against Deep Fakes	svm, OpenFace2