ChatGPT Wide Evaluation

This repository provides the code and the splits of the datasets used in the publication below.

Please cite our work upon use, and check and abide by the respective licenses of the datasets below, and cite their respective work. The splits of the datasets for Personality, Aspect-based problems, and emotions rankings are identical to the originally provided splits in these datasets.

@article{Amin2023-Wide-Evaluation,
  title={{A Wide Evaluation of ChatGPT on Affective Computing Tasks}},
  author={Amin, Mostafa M and Mao, Rui and Cambria, Erik and Schuller, Bj{\"o}rn W},
  journal={Transactions on Affective Computing},
  publisher={IEEE},
  year={2024}
}

Code

The code consists of:

• The script run_experiment.py, which contains the code for training the LSTM models using word IDs (end-to-end), or using RoBERTa features. The datasets were assumed to have a different structure than what is currently available in this repository. The folder dataset_splits provides the outputs of the read_data(problem, part) on the format {problem}_{part}.csv.
• The training builds TFRecords with the IDs or the RoBERTa features of the text. To be used repetitively while using hyperparameter tuning using SMAC.
• Obtaining the responses of the GPT models is done using collect_gpt.py script, assuming the the environment variable for the OpenAI API key is set, using OPENAI_API_KEY.
• The pair sampling of the small world graph is done using the function generateSmallWorldGraph in core/pair_comparisons.py.

Datasets

• Aspect-based Datasets License GPL-3.0 license
• Sentiment Analysis Dataset
• Sentiment Ranking Dataset License Apache License, 2.0
• Emotions Ranking Dataset
• Suicide Detection Dataset License CC BY-SA 4.0
• Toxicity Dataset License CC0: Public Domain
• Well-being Dataset License Apache-2.0
• Engagement Dataset License CC0: Public Domain
• Personality Dataset License CC BY-NC 4.0
• Sarcasm Dataset License CC BY 4.0 DEED
• Subjectivity Dataset

@inproceedings{pontiki2014semeval,
	Author = {Pontiki, Maria and Galanis, Dimitris and Pavlopoulos, John and Papageorgiou, Harris and Androutsopoulos, Ion and Manandhar, Suresh},
	Booktitle = {{Proceedings of SemEval 2014}},
	Pages = {27--35},
	Title = {{SemEval-2014 Task 4: Aspect Based Sentiment Analysis}},
	Year = {2014}}

@inproceedings{pontiki2015semeval,
	Author = {Pontiki, Maria and Galanis, Dimitrios and Papageorgiou, Harris and Manandhar, Suresh and Androutsopoulos, Ion},
	Booktitle = {{Proceedings of SemEval 2015}},
	Pages = {486--495},
	Title = {{SemEval-2015 Task 12: Aspect Based Sentiment Analysis}},
	Year = {2015}}

@inproceedings{chen2020racl,
	Address = {Online},
	Author = {Zhuang Chen and Tieyun Qian},
	Booktitle = {{Proceedings of ACL}},
	Pages = {3685--3694},
	Title = {{Relation-Aware Collaborative Learning for Unified Aspect-Based Sentiment Analysis}},
	Year = {2020}}

@article{Go2009-Twitter,
	Author = {Go, Alec and Bhayani, Richa and Huang, Lei},
	Journal = {CS224N project report, Stanford},
	Number = {12},
	Pages = {2009},
	Title = {{Twitter Sentiment Classification using Distant Supervision}},
	Volume = {1},
	Year = {2009}}

@inproceedings{cortis2017semeval,
	Author = {Cortis, Keith and Freitas, Andr\'e and Daudert, Tobias and Huerlimann, Manuela and Zarrouk, Manel and Handschuh, Siegfried and Davis, Brian},
	Booktitle = {{Proceedings of SemEval-2017}},
	Pages = {519--535},
	Title = {{SemEval-2017 Task 5: Fine-Grained Sentiment Analysis on Financial Microblogs and News}},
	Year = {2017}}

@inproceedings{mohammad2017wassa,
	Author = {Mohammad, Saif and Bravo-Marquez, Felipe},
	Booktitle = {{Proceedings of WASSA 2017}},
	Pages = {34--49},
	Title = {{WASSA-2017 Shared Task on Emotion Intensity}},
	Year = {2017}}

@inproceedings{Desu22-Suicide,
	Address = {Singapore, Singapore},
	Author = {Desu, Vishal and Komati, Nikhileswar and Lingamaneni, Sphoorthi and Shaik, Fathimabi},
	Booktitle = {{Smart Intelligent Computing and Applications, Volume 2}},
	Pages = {263--270},
	Publisher = {Springer Nature Singapore},
	Title = {{Suicide and Depression Detection in Social Media Forums}},
	Year = {2022}}

@inproceedings{rastogi2022stress,
	Author = {Rastogi, Aryan and Liu, Qian and Cambria, Erik},
	Booktitle = {{IJCNN}},
	Pages = {1--8},
	Title = {{Stress Detection from Social Media Articles: New Dataset Benchmark and Analytical Study}},
	Year = {2022}}

@article{Escalante2020-Modeling,
	Author = {Escalante, Hugo Jair and Kaya, Heysem and Salah, Albert Ali and Escalera, Sergio and G{\"u}{\c{c}}l{\"u}t{\"u}rk, Ya{\u{g}}mur and G{\"u}{\c{c}}l{\"u}, Umut and Bar{\'o}, Xavier and Guyon, Isabelle and Junior, Julio CS Jacques and Madadi, Meysam and others},
	Journal = {Transactions on Affective Computing},
	Number = {2},
	Pages = {894--911},
	Publisher = {IEEE},
	Title = {{Modeling, Recognizing, and Explaining Apparent Personality From Videos}},
	Volume = {13},
	Year = {2020}}

@inproceedings{Ponce16-ChaLearn,
	Address = {Cham, Switzerland},
	Author = {Ponce-L\'opez, V\'\ictor and Chen, Baiyu and Oliu, Marc and Corneanu, Ciprian and Clap\'es, Albert and Guyon, Isabelle and Bar\'o, Xavier and Escalante, Hugo Jair and Escalera, Sergio},
	Booktitle = {{ECCV}},
	Pages = {400--418},
	Title = {{Chalearn lap 2016: First Round Challenge on First Impressions - Dataset and Results}},
	Year = {2016}}


@article{misra2023Sarcasm,
	Author = {Rishabh Misra and Prahal Arora},
	Journal = {AI Open},
	Pages = {13--18},
	Publisher = {Elsevier},
	Title = {{Sarcasm Detection using News Headlines Dataset}},
	Volume = {4},
	Year = {2023}}


@inproceedings{pang2004sentimental,
	Author = {Pang, Bo and Lee, Lillian},
	Booktitle = {{Proceedings of ACL}},
	Pages = {271--278},
	Title = {{A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts}},
	Year = {2004}}