Skip to content

A datasets and methods survey about task-oriented dialogue, including recent datasets and SOTA leaderboards.

Notifications You must be signed in to change notification settings

AtmaHou/Task-Oriented-Dialogue-Research-Progress-Survey

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

63 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Task-Oriented Dialogue Research Progress Survey

Content

Introduction

This repo is a dataset and methods survey for Task-oriented Dialogue.

We investigated most existing dialogue datasets and summarized their basic information, such as brief, download link and size.

We also included leader boards of popular dataset to present research progress in the task oriented dialogue fields.

A Chinese intro & news for this project is available here

Refer to this repo:

@misc{MAML_Pytorch,
  author = {Yutai Hou},
  title = {Task-Oriented Dialogue Research Progress Survey},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/AtmaHou/Task-Oriented-Dialogue-Research-Progress-Survey/}},
  commit = {master}
}

Updates

This section records big updates to ease refer (See ./release_detail.md or click links below):

Call for Contributions

Contributions are welcomed, you are encouraged to:

  • Directly pull request
  • Send me new dataset info
  • Send me new experiment results from published paper or public code implements.

Leader Boards

The ranking is depended on published results of related papers. We are trying to keep it up-to-date. The ranking may be unfair because features used and train/dev set splitting in those papers may be different. However, it shows a trend of research, and would be helpful for someone to start a project about task-oriented dialogue.

Dialogue State Tracking

Dialogue state tacking task aims to predict or give representation of dialogue state, which usually contains a goal constraint, a set of requested slots, and the user's dialogue act.

MultiWOZ 2.0 - Dialogue State Tracking

Multi-Domain Wizard-of-Oz dataset (MultiWOZ), a fully-labeled collection of human-human written conversations spanning over multiple domains and topics. At a size of 10k dialogues, it is at least one order of magnitude larger than all previous annotated task-oriented corpora.

The new, corrected versions of the dataset are available at MultiWOZ 2.1 (2019), MultiWOZ 2.2 (2020).

Notice: Models marked with * are open-vocabulary based models.`

Model Joint Acc. Slot Acc. Paper / Source
SOM-DST (BERT-large)* (Kim et al, 2020) 52.32 - Efficient Dialogue State Tracking by Selectively Overwriting Memory
SOM-DST* (Kim et al, 2020) 51.72 - Efficient Dialogue State Tracking by Selectively Overwriting Memory
SAS (Hu et al, 2020) 51.03 97.20 SAS: Dialogue State Tracking via Slot Attention and Slot Information Sharing
MERET (Huang et al, 2020) 50.91 97.07 Meta-Reinforced Multi-Domain State Generator for Dialogue Systems
NADST* (Le et al, 2020) 50.52 - Non-Autoregressive Dialog State Tracking
TRADE* (Wu et al, 2019) 48.62 96.92 Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
SUMBT (Lee et al, 2019) 46.649 96.44 SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking
HyST* (Goel et al, 2019) 44.24 - HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking
Neural Reading (Gao et al, 2019) 41.10 - Dialog State Tracking: A Neural Reading Comprehension Approach
GLAD (Zhong et al., 2018) 35.57 95.44 Global-Locally Self-Attentive Dialogue State Tracker
MDBT (Ramadan et al., 2018) 15.57 89.53 Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing

DSTC2 - Dialogue State Tracking

Clarification of dataset types:

The main results we list here are obtained from pure DSTC2 dataset (ASR n-best).

However, we don't list other kinds of DSTC2 data source results such as DSTC2-text (It formulates the dialog state tracking as a machine reading problem which read the dialog transcriptions multiple times and answer the questions about each of the slot, for more info please refer to paper) and DSTC-cleaned (It is used by the NBT paper and fixes ASR noise and typo during training and include ASR noise during testing, The cleaned version is available at here),

Model Area Food Price Joint Paper / Source
Liu et al. (2018) 90 84 92 72 Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
Neural belief tracker (Mrkšić et al., 2017) 90 84 94 72 Neural Belief Tracker: Data-Driven Dialogue State Tracking
RNN (Henderson et al., 2014) 92 86 86 69 Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised gate

NLU: Slot Filling

Slot filling task aims to recognize key entity within user utterance, such as position and time.

Snips - Slot Filling

Model F1 Paper / Source
Enc-dec (focus) + BERT 97.17 Code
Stack-Propagation + BERT (Qin et al., 2019) 97.0 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
Joint BERT (Chen et al., 2019) 97.0 BERT for Joint Intent Classification and Slot Filling
BLSTM-CRF + ELMo word embedding 96.92 Code
Stack-Propagation (Qin et al., 2019) 94.2 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
ELMo + BLSTM-CRF (Siddhant et al., 2018) 93.90 Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
Capsule Neural Networks (Zhang et al., 2018) 91.8 Joint Slot Filling and Intent Detection via Capsule Neural Networks
Slot-Gated (Full Atten.) (Goo et al., 2018) 88.8 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
BLSTM-CRF (Siddhant et al., 2018) 88.78 Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
Slot-Gated (Intent Atten.) (Goo et al., 2018) 88.3 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction

ATIS - Slot Filling

Notice: The following works have abnormal high-scores, because they are considered to exploit special pre-processing steps: Bi-model-Decoder (Wang et al., 2018), Intent Gating + Self-atten. (Li et al., 2018), Atten.-Based (Liu and Lane, 2016)

Model F1 Paper / Source
Bi-model-Decoder (Wang et al., 2018) 96.89 A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling
Intent Gating + Self-atten. (Li et al., 2018) 96.52 A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding
Stack-Propagation + BERT (Qin et al., 2019) 96.10 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
Joint BERT (Chen et al., 2019) 96.10 BERT for Joint Intent Classification and Slot Filling
Atomic Concept (Su Zhu and Kai Yu, 2018) 96.08 Concept Transfer Learning for Adaptive Language Understanding
Atten.-Base + Delexicalization (Shin et al., 2018) 96.08 Slot Filling with Delexicalized Sentence Generation
Atten.-Based (Liu and Lane, 2016) 95.98 Attention-based recurrent neural network models for joint intent detection and slot fillin
Stack-Propagation (Qin et al., 2019) 95.90 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
Encoder-Decoder-Pointer (Zhai et al., 2017) 95.86 Neural Models for Sequence Chunking
ELMo + BLSTM-CRF (Siddhant et al., 2018) 95.62 Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
Capsule Neural Networks (Zhang et al., 2018) 95.2 Joint Slot Filling and Intent Detection via Capsule Neural Networks
Slot-Gated (Intent Atten.) (Goo et al., 2018) 95.2 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Slot-Gated (Full Atten.) (Goo et al., 2018) 94.8 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction

NLU: Intent Detection

Intent detection task aims to classify user utterance into different domain or intents.

Snips - Intent Detection

Model Acc. Paper / Source
ELMo + BLSTM-CRF (Siddhant et al., 2018) 99.29 Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
Enc-dec (focus) + ELMo 99.14 Code
Stack-Propagation + BERT (Qin et al., 2019) 99.0 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
Joint BERT (Chen et al., 2019) 98.6 BERT for Joint Intent Classification and Slot Filling
Stack-Propagation (Qin et al., 2019) 98.0 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
Capsule Neural Networks (Zhang et al., 2018) 97.7 Joint Slot Filling and Intent Detection via Capsule Neural Networks
Slot-Gated (Full Atten.) (Goo et al., 2018) 97.0 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Slot-Gated (Intent Atten.) (Goo et al., 2018) 96.8 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction

ATIS - Intent Detection

Notice-1: The following works have abnormal high-scores, because they are considered to exploit special pre-processing steps: Bi-model-Decoder (Wang et al., 2018), Intent Gating + Self-atten. (Li et al., 2018), Atten.-Based (Liu and Lane, 2016), BLSTM (Zhang et al., 2016)

Model Acc. Paper / Source
BLSTM + BERT 99.10 Code
Bi-model-Decoder (Wang et al., 2018) 98.99 A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling
Intent Gating + Self-atten. (Li et al., 2018) 98.77 A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding
Atten.-Based (Liu and Lane, 2016) 98.43 Attention-based recurrent neural network models for joint intent detection and slot filling
BLSTM (Zhang et al., 2016) 98.10 A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding
Joint BERT (Chen et al., 2019) 97.9 BERT for Joint Intent Classification and Slot Filling
Stack-Propagation + BERT (Qin et al., 2019) 97.5 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
ELMo + BLSTM-CRF (Siddhant et al., 2018) 97.42 Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
Stack-Propagation (Qin et al., 2019) 96.9 A Stack-Propagation Framework with Token-level Intent Detection for Spoken Language Understanding
Capsule Neural Networks (Zhang et al., 2018) 95.0 Joint Slot Filling and Intent Detection via Capsule Neural Networks
Slot-Gated (Intent Atten.) (Goo et al., 2018) 94.1 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Slot-Gated (Full Atten.) (Goo et al., 2018) 93.6 Slot-Gated Modeling for Joint Slot Filling and Intent Prediction

Dataset Introductions

See the data details Here or in Excel File

Following information is included for each dataset:

  • Name
  • Introduction
  • Link (Download & Paper)
  • Multi or single turn
  • Task detail
  • Whether Public Accessible
  • Size & Stats
  • Included Label
  • Missing Label

Tips: The table below may not be displayed completely, scroll right to see more~

Name Introduction Links Multi/Single Turn Task Detail Public Accessible Size & Stats Included Label Missing Label
Few-shot Slot Tagging Benchmark 1. Dialogue slot tagging dataset for few-shot learning setting
2. First few-shot sequence labeling benchmark (Meta-episode style data format)
3. Also include 5 NER dataset for few-shot sequence labeling evaluation.
Download:https://atmahou.github.io/attachments/ACL2020data.zip
Paper: https://arxiv.org/pdf/2006.05702.pdf
S 7 dialogue task:
Weather,play music, search, add to list, book, moive
5 NER task
Yes For each task, it contains 100 episodes.
Each episode contains a query set (20 samples) and a support set (1-shot & 5-shot)
Slots Intent
Taskmaster-2 (2020) 1. Unlike Taskmaster-1, which includes both written "self-dialogs" and spoken two-person dialogs, Taskmaster-2 consists entirely of spoken two-person dialogs.
2. Users were led to believe they were interacting with an automated system that “spoke” using text-to-speech (TTS)
3. Intents are labeled on slots
Download: https://github.com/google-research-datasets/Taskmaster/tree/master/TM-2-2020/data
Homepage: https://github.com/google-research-datasets/Taskmaster/tree/master/TM-2-2020
M 7 domains:
restaurants, food ordering, movies, hotels, flights, music, sports
Yes 17,289 dialogs:
restaurants (3276)
food ordering (1050)
movies (3047)
hotels (2355)
flights (2481)
music (1602)
sports (3478)
NLU(Intent, Slots)
JDDC Corpus 2020 1. A large-scale Multimodal Chinese E-commerce conversation corpus.
2. Human2Human conversations
Download: https://jddc.jd.com/auth\_environment
Homepage: https://jddc.jd.com/description
M Multimodal E-commerce conversation Yes Electronic: 130k dialogues, 950k utterances, 215k images.
Clothing: 116k dialogues, 810k utterances, 200k images.
Intents (Only on images),
Database
NLU(Intent, Slots)
CrossWOZ 1. CrossWOZ, the first large-scale Chinese Cross-Domain Wizard-of-Oz taskoriented dataset.
2. Encourage natural transition across domains in conversation.
3. Provide a user simulator
4. Human2Human
Download: https://github.com/thu-coai/CrossWOZ
Paper: https://arxiv.org/pdf/2002.11893.pdf
M 5 domains, including hotel, restaurant, attraction, metro, and taxi. Yes 5,012 dialogues,
84,692 turns,
16.9 Avg. turns,

Annotation:
72 slots, 7,871 vlaues, 6 intents
User Goals,
State (Intent, Slots),
Database
API calls
JDDC Corpus 2019 1. A large-scale real scenario Chinese E-commerce conversation corpus.
2. Human2Human conversations covers: task-oriented, chitchat and question-answering.
3. Large scale: 1 million multi-turn dialogues, 20 million utterances.
4. Main task: dialogue generation
Download: http://jddc.jd.com/auth\_environment
Paper: https://arxiv.org/pdf/1911.09969.pdf
M E-commerce conversation Yes Totoal: 1 million dialogues, 20 million utterances.
Annotation: 289 different intents
Challenge1: 300 dialogues, 300 questions;
Challenge2: 15 dialogues, 168 questions;
Challenge3: 108 dialogues, 500 questions;
Intent (Machine Labeled),
Database
Slot
CAIS 1. Dialogue utterances from the Chinese Artificial Intelligence Speakers (CAIS) annotated with slot tags and intent labels. Download: https://github.com/Adaxry/CM-Net
Paper: https://www.aclweb.org/anthology/D19-1097.pdf
S Most are music related tasks. Yes Train 7995;
Dev 994;
Test 1012;
11 Intents, 75 Slots
Intent
Slots
Multimodal Dialogs (MMD) Dataset 1. Multimodal conversations in the fashion domain.
2. Human-to-human
3. Contain annotation of query type (Similar to intent)
4. Large size: 150K conversation
Download: https://amritasaha1812.github.io/MMD/
Paper:
https://arxiv.org/abs/1704.00200
M Shopping Assistant Yes 150K conversation session question-type (Intent)
State Type (17 type of dialogue state class)
Slot
Taskmaster-1 (2019) 1. A task-based dataset collect with two different procedures: Wizard of Oz and self conversation.
2. Encourage realistic and diversity by giving up restrict speaker with knowledage base.
3. Both Human2machine and Human2Human dialogues
Download: https://g.co/dataset/taskmaster-1
Paper: https://arxiv.org/pdf/1909.05358.pdf
M 6 domains: ordering pizza, creating auto repair appointments, setting up ride service, ordering movie tickets, ordering coffee drinks and making restaurant reservations. Yes Human-Human: 7,708 dialogues, 169,469 utterances
Human-Machine: 10,438 dialogues, 132,610 utterances
API calls,
Argument (Slot)
Intent
MetaLWOz 1. Dialogue dataset for developing fast adaptation methods for conversation models. (Track in DSTC 8)
2. Lots of domains and tasks: 47 domains and 227 tasks.
3. Suitable for meta learning.
4. Main task: dialogue generation
Download: https://www.microsoft.com/en-us/download/58389
Homepage: https://www.microsoft.com/en-us/research/project/metalwoz/
M 47 domains and 227 tasks. Yes 37,884 dialogues, ( >10 turns long),
47 domains and 227 tasks.
Only utterance NLU(Intent, Slots)
Minecraft Dialogue corpus 1. The goal of this project is to develop systems that can collaborate and communicate with each other to solve tasks in a 3D environment.
2. Human2Human
3. Main task: given context and 3D scenes, generating response.
Download: http://juliahmr.cs.illinois.edu/Minecraft/
Paper: https://www.aclweb.org/anthology/P19-1537.pdf
M "Architect" instruct the "Builder" to build a 3D structure. Yes 509 human-human dialogues;
15,926 utterances (train 6,548, dev 2,855, test 2,251 Architect utterances);
Golden utterance,
game log,
screenshots
NLU(Intent, Slots)
E-commerce Dialogue Corpus (EDC) 1. Real-world conversations between customers and
customer service staff from our E-commerce partners in Taobao
2. Main task: response selection
Download:
https://github.com/cooelf/
Paper: https://arxiv.org/pdf/1806.09102.pdf
M Contains 5 types of conversations: commodity consultation, logistics express, recommendation, negotiation and chitchat based on over 20 commodities. Yes Dialogues 1,020,000,
Utterance 7,500,000.
Only utterance NLU(Intent, Slots)
Schema-Guided Dialogue State Tracking(DST8) 1. Largest by now & containing over 16k multi-domain conversations spanning 16 domains
2. Present a schema-guided paradigm
3. Enable zero-shot generalization to new APIs
Download: https://github.com/google-research-datasets/dstc8-schema-guided-dialogue
Paper: https://arxiv.org/pdf/1909.05855.pdf
M 16 domains Alarm, Banks, Buses, Calendar, Events, Flights, Homes, Media, Messaging, Movies, etc. Yes Over 16k dialogues average number of turns are 20.44 multi-domain dialogues. 329,964 turns in total. Schema for each service contains:
service_name and description,
slots,
intents
_
MultiWOZ 2.0 1. Proposed by EMNLP 2018 best paper.
2. Largest by now & contain multi-domains.
3. Human2human
4. goal changes are encouraged
Download:
http://dialogue.mi.eng.cam.ac.uk/index.php/corpus/
Paper:
https://arxiv.org/pdf/1810.00278.pdf
M 7 domains
Attraction, Hospital,
Police, Hotel, Restaurant, Taxi, Train.
Yes Total 10438 dialogues
average number of turns are 8.93 and 15.39 for single and multi-domain dialogues respectively.
115, 434 turns in total.
Belief state
User Act(inform, request slots)
Agent Act(inform, request slots)
NLU(Intent, Slots)
Facebook Multilingual Task Oriented Dataset 1. (Faceboook) We release a dataset of around 57k annotated utterances
in English (43k), Spanish (8.6k) and Thai (5k) for three task oriented domains … ALARM,
REMINDER, and WEATHER.
2. For cross-lingual natural language understanding
Download: https://fb.me/multilingual\_task\_oriented\_data
Paper: https://arxiv.org/pdf/1810.13327.pdf
S 3 Domains: Alarm, Reminder, Weather

3 Languages: English, Spanish, Thai
Yes English Train: 30,521
English Dev: 4,181
English Test: 8,621

Spanish Train: 3,617
Spanish Dev: 1,983
Spanish Test: 3,043

Thai Train: 2,156
Thai Dev: 1,235
Thai Test: 1,692
Slot
Intent
Medical DS 1. Our dataset is collected from the pediatric department in a Chinese online healthcare community
2. Task-oriented Dialogue System for Automatic Diagnosis
Download:
http://www.sdspeople.fudan.edu.cn/zywei/data/acl2018-mds.zip
Paper:
http://www.sdspeople.fudan.edu.cn/zywei/paper/liu-acl2018.pdf
M Automatic Diagnosis Yes 4 Disease
67 symptoms
Slot
Action
Snips 1. Collected by Snips for model evaluation.
2. For natural language understanding
3. Homepage: https://medium.com/snips-ai/benchmarking-natural-language-understanding-systems-google-facebook-microsoft-and-snips-2b8ddcf9fb19
Download:
https://github.com/snipsco/
nlu-benchmark/tree/master/ 2017-06-custom-intent-engines
S 7 task:
Weather,play music, search, add to list, book, moive
Yes Train:13,084
Test:700
7 intent 72 slot labels
Intent
Slots
MIT Restaurant Corpus 1. The MIT Restaurant Corpus is a semantically tagged training and test corpus in BIO format.
2. For natural language understanding
Download:
https://groups.csail.mit.edu/sls/downloads/restaurant/
S Restaurant Yes Train, Dev, Test
6,894 766 1,521
Slot Intent
MIT Movie Corpus 1. The MIT Movie Corpus is a semantically tagged training and test corpus in BIO format. The eng corpus are simple queries, and the trivia10k13 corpus are more complex queries.
2. For natural language understanding
Download:
https://groups.csail.mit.edu/sls/downloads/movie/
S Movie Yes Train, Dev, Test
MIT Movie Eng 8,798 977 2,443
MIT Movie Trivia 7,035 781 1,953
Refer to: Data Augmentation for Spoken Language Understanding via Joint Variational Generation
Slot Intent
ATIS 1. The ATIS (Airline Travel Information Systems) dataset (Tur et al., 2010) is widely used in SLU research
2. For natural language understanding
Download:
1. https://github.com/AtmaHou/Bi-LSTM\_PosTagger/tree/master/data
2.https://github.com/yvchen/JointSLU/tree/master/data
S Airline Travel Information Yes Train: 4478
Test: 893
120 slot and 21 intent
Intent
Slots
Microsoft Dialogue Challenge 1. Containing human-annotated conversational data in three domains an
2. Experiment platform with built-in simulators in each domain, for training and evaluation purposes.
Paper:
https://arxiv.org/pdf/1807.11125.pdf
M Movie-Ticket Booking
Restaurant Reservation
Taxi Ordering
Yes Task Intents Slots Dialogues
Movie-Ticket Booking 11 29 2890
Restaurant Reservation 11 30 4103
Taxi Ordering 11 29 3094
Intent
Slots
Database
API-call
CamRest676 CamRest676 Human2Human dataset contains the following three json files:
1. CamRest676.json: the woz dialogue dataset, which contains the conversion from users and wizards, as well as a set of coarse labels for each user turn.
2. CamRestDB.json: the Cambridge restaurant database file, containing restaurants in the Cambridge UK area and a set of attributes.
3. The ontology file, specific all the values the three informable slots can take.
Download:
https://www.repository.cam.ac.uk/handle/1810/260970
Paper:
https://arxiv.org/abs/1604.04562
M Booking restaurant Yes Total 676 Dialogues
Total 1500 Turns
Train:Dev:Test 3:1:1 (Test set not given)
Slot
User Act(inform, request slots)
Agent Act(inform, request slots)
Intent
API call
Database
Human-human goal oriented dataset 1. Maluuba reased a travel booking dataset
2. Design for new task: frame tracking (allow comparing between history entities)
3. Homepage: https://datasets.maluuba.com/Frames
4. Human2Human
Download:
https://datasets.maluuba.com/Frames/dl
Paper:
https://arxiv.org/abs/1706.01690
https://1drv.ms/b/s!Aqj1OvgfsHB7dsg42yp2BzDUK6U
M Travel Booking Yes Dialogues 1369
Turns 19986
Average user satisfaction (from 1-5) 4.58
Frame
User agenda
User Act(inform, request slots)
Agent Act(inform, request slots)
API Call
User's satisfaction
Task successful
Database
Entity reference
Intent
Dialog bAbI tasks data 1. Facebook's 6 task-oriented dialogues data set consist of 6 different tasks.
2. Dataset for task 1-5 is constucted automaticly from bots' chat(Bot2Bot). And dataset for task 6 is simply reformated dstc2 dataset.
3. A Shared database is included.
4. This is the only task-oriented dataset among bAbI tasks.
5. The goal of it is to evaluate end2end tasks, so there is not intents and slots.
Download:
https://research.fb.com/downloads/babi/
Paper:
http://arxiv.org/abs/1605.07683
M Book a table at a restaurant Yes For each task,
training 1000
develop 1000
test 1000

For tasks 1-5,
second test set (with suffix -OOV.txt) that contains dialogs including entities not present.
API call
Full Database
Slot
Intent
User Act
Agent Act
Stanford Dialog Dataset 1. Standford NLP group's data of car autopilot agent.
2. Human2Human
3. A quick intro http://m.sohu.com/n/499803391/
Download:
http://nlp.stanford.edu/projects/kvret/kvret\_dataset\_public.zip
Paper:
https://arxiv.org/abs/1705.05414

M car autopilot agent: schedule, weather, navigation Yes Training Dialogues 2,425
Validation Dialogues 302
Test Dialogues 304
Avg. # of Utterances Per Dialogue 5.25
Dialogue level database
User Act(inform, request slots)
Agent Act(inform, request slots)
API call
Intent
Slot
Stanford Dialog Dataset LU 1. Stanford data labeled by HIT, relabel slot & intent
2. Human2Human
3. A quick intro http://m.sohu.com/n/499803391/ to stanford data
4. Annotation handbook: https://docs.google.com/document/d/1ROARKf8AJNnG2\_nPINe1Xm5Rza7V0jPnQV8io09hcFY/edit
N/A

M car autopilot agent: schedule, weather, navigation No Training Dialogues 2,425
Validation Dialogues 302
Test Dialogues 304
Avg. # of Utterances Per Dialogue 5.25
Slot
Intent
API call

Need to do sample alignment to get the following:
Dialogue level database
User Act(inform, request slots)
Agent Act(inform, request slots)
Agent Reply
DSTC-2 1. Human2Bot restaurant booking dataset
2. For usage refer to: http://camdial.org/~mh521/dstc/downloads/handbook.pdf
3. Each dialofue is stored in different folder, which contains log and label.
http://camdial.org/~mh521/dstc/ M Booking restaurant Yes Train 1612 calls
Dev 506 calls
Test 1117 dialogs
Slot
User Act(inform, request slots)
Agent Act(inform, request slots)
Intent
API call
Database
DSTC4 1. Data name as TourSG consists of 35 dialog sessions on touristic information for Singapore collected from Skype calls between three tour guides and 35 tourists
2. All the recorded dialogs with the total length of 21 hours have been manually transcribed and annotated with speech act and semantic labels for each turn level.
3. Homepage: http://www.colips.org/workshop/dstc4/data.html
4. Human2Human
N/A M Query touristic information No Train 20 dialogs
Test 15 dialogs
speech act (User & Agent)
semantic labels(Intent? User & Agent)
topic for turn (Intent?)
N/A
Movie Booking Dataset 1. (Microsoft) Raw conversational data collected via Amazon Mechanical Turk, with annotations provided by domain experts.
2. Human2Human
Download:
https://github.com/MiuLab/TC-Bot#data
Paper:
TC-bot
M Booking Movie Yes 280 dialogues
turns per dialogue is approximately 11
User Act(inform, request slots)
Agent Act(inform, request slots)
Intent
Slots
Database
API-call
Lingxi 1. The data is all single round user input divided into good words. There is more noise.
2. Completed part of speech tagging and slot labeling
3. Language: Chinese
N/A S conversational robot service user log No Utterance: 5132 Slot
POS
Agent reply
Intent
API call
Database
TOP semantic parsing 1. (Facebook) A hierarchical semantic representation for task oriented dialog systems that can model compositional and nested queries. (hierarchical intent and slot)
2. For natural language understanding
3. Human2bot
Download:
http://fb.me/semanticparsingdialog
Paper:
https://arxiv.org/pdf/1810.07942.pdf
S Navigation and event Yes Train 31279 utterances
Dev 4462 utterances
Test 9042 utterances
Hierarchical intents
Slots
SwDA 1. The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2, with turn/utterance-level dialog-act tags.
2. The tags summarize syntactic, semantic, and pragmatic information about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s.
Download: http://compprag.christopherpotts.net/swda.html
Instruction: https://web.stanford.edu/~jurafsky/ws97/manual.august1.html
S Switchboard Dialog Yes Train: 197,489 training-set utterances, 1115 conversations
Test: 40 conversations
Annotation: 42 Classes
Act Slot

Acknowledgment

Thanks for supports from my adviser Wanxiang Che.

Thanks for public contributions from: Shuai Lin, JiAnge, Su Zhu, seeledu, Tony Lin, Jason Krone, Libo Qin, HariiHe, Jelle Bosscher, .

About

A datasets and methods survey about task-oriented dialogue, including recent datasets and SOTA leaderboards.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published