Skip to content

Latest commit

 

History

History
97 lines (67 loc) · 8.9 KB

README.md

File metadata and controls

97 lines (67 loc) · 8.9 KB
Raw

ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models

If you are confused by any of the following contents or you have any suggestion, please contact us at [email protected]. 【如果有任何疑问以及建议，欢迎通过邮箱联系我们。】

In this repo, you will see 【内容如下】:

  • A corpus of 4k argumentative essays written by 7 GPT models. 【我们用7个GPT模型生成了4k议论文。】
  • Several classifiers for detecting GPT-generated argumentative essays (99% accuracy at doc-level; 93% at sentence-level). 【训练了相关模型检测议论文是否由GPT模型生成（文章检测达到99%的准确率，句子检测达到93%的准确率）。】

And we make following resources public 【我们公布了以下资源】:

  • Models: We fine-tuned RoBERTa to detect machine-generated argumentative essays. We trained both doc-level and sentence-level classifiers. 【我们微调了RoBERTa以检测机器生成的议论文，我们在文档级别和句子级别上都进行了训练。】
  • App-demo: We built an app in huggingface space which bases on the sentence-level detector. 【基于句子级别的检测模型，我们搭建了一个App，可以在网页中直接使用。】
  • Paper: We released our research paper in arxiv. 【我们的论文预印在了arxiv中。】

Introduction to ArguGPT corpus 语料清单

We compiled a 8k human-machine (4k human v.s. 4k machine) comparison argumentative essays. We also collected a 1k dataset for the out-of-distribution (OOD) test experiment, in which the essays are either generated by ood models or prompted by ood writing tasks. Sub-corpora are listed in the following table. 【ArguGPT包含了8k议论文文章，包括4k人类议论文和4k机器生成议论文。我们也另外收集了1k议论文，作为OOD测试集以验证分类器泛化能力（其中的议论文使用的作文题或生成式模型与ArguGPT中的不同），语料清单如下。】

sub-corpus # essays # tokens mean len source access
WECCL-human 1,845 450,657 244 SWECCL 2.0 SWECCL (Wen & Wang, 2008)
WECCL-machine 1,813 442,531 244 GPT models Released. See in data/argugpt folder
TOEFL-human 1,680 503,504 299 TOEFL11 Purchased in LDC
TOEFL-machine 1,635 442,963 270 GPT models Released. See in data/argugpt folder
GRE-human 590 341,495 578 GRE-prep materials No copyright to release
GRE-machine 590 268,640 455 GPT models Released. See in data/argugpt folder
OOD-human 500 132,902 265 CLEC CLEC (Gui & Yang, 2003)
OOD-machine 500 180,120 360 ChatGPT & four OOD models Released. See in data/argugpt folder

Note that four OOD models are: gpt-4, claude-instant, bloomz-7b, and flan-t5-11b. More detailed information about the ArguGPT corpus can be seen in our paper. 【有关ArguGPT的更多信息，请参考我们的论文。】

Data split and baseline 数据划分及基准模型

We first split the data into train/dev/test sets. The test split of TOEFL essays are as well evaluated by human participants in the Turing test. Then we established baselines by training detectors based on SVMs and RoBERTa. Moreover, we conducted ablation study to see the effect of reducing training data points. Finally, we evaluated two detectors on our own test set, namely GPTZero and RoBERTa trained by Guo et al. (2023). 【我们将数据集进行了划分。其中，图灵测试（人类测评）中使用的数据为测试集中的托福作文。在此数据集上，我们训练了SVM和RoBERTa模型，作为该数据集的基准模型。此外，我们在不同大小的训练集上进行了训练。】

split TOEFL WECCL GRE total
train 3,058 2,715 980 6,753
dev 300 300 100 700
test 300 300 100 700

Accuracy of human evaluators on the TOEFL split of test set is only 64.65%, far lagging behind ML detectors/classifiers. 【对测试集中的托福文章进行图灵测试，人类参与者的准确率仅有64.65%，远低于基于机器学习的检测器。】

train data test data maj. bsln RoBERTa Best SVM GPTZero Guo et al. (2023)
doc-all doc test 50 99.38 95.14 98.86 89.86
doc-50% doc test 50 99.76 94.14 - -
doc-25% doc test 50 99.14 93.86 - -
doc-10% doc test 50 97.67 92.29 - -
doc-all para test 52.62 74.58 83.61 - -
para-all para test 52.62 97.88 90.55 92.11 79.95
doc-all sent test 54.18 49.73 72.15 -
sent-all sent test 54.18 93.84 81 90.10 71.44
doc-all ood-ma test 100 97.00 72.20 53.40 59.20
doc-all ood-hu test 100 98.47 94.80 100.00 99.00

Notes: We broke down the dataset of essays into sentences and paragraphs, trained the models from document-, paragraph-, and sentence-level, and evaluated these classifiers. 【我们把文章拆分为段落和句子，并且分别从文章、段落和句子的层次训练了模型，并进行了一些测试。】

Team members 团队介绍

We are a group of students who are interested in language, linguistics, and NLP as well. Led by Hai Hu, an assistant professor from English Department of Shanghai Jiao Tong University (SJTU), whose research interest is computational linguistics, we hope to contribute something interesting to CL and NLP commuities from the perspective of language leaners. 【大家好，我们是一群热爱语言、同时也对NLP技术感兴趣的本科/研究生，团队由交大外院的胡海老师指导。我们会从语言学习者的角度，进行一些有趣的实验与研究。如果有幸的话，希望我们的研究能够对计算语言学和NLP社群做出一些小小的贡献！】

Name 姓名 Affiliation 所属机构 Status
Hai Hu 胡海 SFL, SJTU 上海交通大学外院 Assistant Professor
Yiwen Zhang 张伊文 Amazon 亚马逊 Language Engineer
Shisen Yue 岳士森 SFL, SJTU 上海交通大学外院 Undergraduate
Wanyang Zhang 章万扬 SS, PKU 北京大学软件与微电子学院 Graduate
Xiaojing Zhao 赵晓靖 SFL, SJTU 上海交通大学外院 Graduate
Xinyuan Cheng 程心远 SFL, SJTU 上海交通大学外院 Undergraduate
Yikang Liu 刘逸康 SFL, SJTU 上海交通大学外院 Graduate
Ziyin Zhang 张子殷 SEIEE, SJTU 上海交通大学电院 Graduate

Citation

Please cite our work as

@misc{liu2023argugpt,
      title={ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models}, 
      author={Yikang Liu and Ziyin Zhang and Wanyang Zhang and Shisen Yue and Xiaojing Zhao and Xinyuan Cheng and Yiwen Zhang and Hai Hu},
      year={2023},
      eprint={2304.07666},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}