Skip to content

Latest commit

 

History

History
204 lines (147 loc) · 5.77 KB

README.md

File metadata and controls

204 lines (147 loc) · 5.77 KB
Raw

VATE

license last-commit repo-top-language repo-language-count

Quick Links

Overview

the Video-Audio-Text for affective Evaluation dataset. VATE collects a wide variety of multimodal data exhibiting a multitude of spontaneous human affective states. It contains 21,871 raw videos together with voice recordings and text transcriptions from numerous emotion evoking interviews. VATE is specifically designed for contrastive self-supervised representation learning of human affective states; it prioritises quantity and quality of data over human labelling of emotions, which constitutes a highly subjective, often inconsistent and controversial aspect of modern affective computing. To highlight the usefulness of our proposal, we release a multimodal encoder employing a contrastive video-language-audio pre-training procedure carried out on the VATE dataset. Experimental results show that such model exhibits sensibly better few-shot generalisation abilities when compared to fully supervised baselines on different downstream tasks.

An overview of the data can be found at

VATE/output/VATE/metadata.json

Repository Structure

└── VATE/
    ├── VATE.py
    ├── README.md
    ├── audio.py
    ├── contrastive_model.py
    ├── dataset.py
    ├── dataset_utils.py
    ├── feature_extraction
    │   ├── VATE
    │   ├── collect_yb.py
    │   ├── couples.txt
    │   ├── cut_video.py
    │   ├── input.txt
    │   ├── main.py
    │   └── write_video.py
    ├── main.py
    ├── media.py
    ├── output
    │   └── VATE
    │       ├── best_model_contrastive.pt
    │       └── metadata.json
    ├── text.py
    ├── train_test.py
    ├── utils.py
    └── video.py

Getting Started

Installation

  1. Clone the VATE repository:
git clone https://github.com/FrancescoAgnelli3/VATE
  1. Change to the project directory:
cd VATE
  1. Install the dependencies:
pip install -r requirements.txt

Downloading VATE

Use the following command to download the VATE dataset:

  1. Change to the project directory:
cd feature_extraction
  1. Download the dataset:
python main.py

The dataset will be downloaded in the folder:

VATE/feature_extraction/VATE

If you want to add other YouTube playlists to the dataset, you can add them to the python file and run:

python collect_yb.py

And then again:

python main.py

Contrastive model

  1. To train the contrastive model on the dataset, change to the project directory:
cd ..
  1. Train the model:
python main.py
  1. The model will be saved in (or it can be directly downloaded, already pre-trained, from) the folder:
VATE/output/VATE/best_model_contrastive.pt

Contributing

To contribute to the project, please follow these guidelines:

  1. Fork the repository and clone it to your local machine.

  2. Create a new branch for your feature or bug fix.

  3. Make your changes and commit them with descriptive commit messages.

  4. Push your branch to your forked repository.

  5. Submit a pull request to the main repository.

License

This project is protected under the MIT LICENSE License.