Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing

This is the PyTorch implementation of our paper:
Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing
Yan-Bo Lin, Hung-Yu Tseng, Hsin-Ying Lee, Yen-Yu Lin, and Ming-Hsuan Yang
Advances in Neural Information Processing Systems (NeurIPS), 2021

paper | dataset

📝 Preparation

1. pip3 install requirements.txt
2. Following AVVP, prepare pre-extracted features in .feats/r2plus1d_18, .feats/res152, and .feats/vggish

📚 Train and evaluate

Simply run bash run.sh

🎓 Cite

If you use this code in your research, please cite:

@article{lin2021exploring,
  title={Exploring Cross-Video and Cross-Modality Signals for Weakly-Supervised Audio-Visual Video Parsing},
  author={Lin, Yan-Bo and Tseng, Hung-Yu and Lee, Hsin-Ying and Lin, Yen-Yu and Yang, Ming-Hsuan},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}

License

This project is licensed under CC-BY-NC 4.0 License, as found in the LICENSE file.