Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering

ACL 2020  ·  Changmao Li, Jinho D. Choi ·

We introduce a novel approach to transformers that learns hierarchical representations in multiparty dialogue. First, three language modeling tasks are used to pre-train the transformers, token- and utterance-level language modeling and utterance order prediction, that learn both token and utterance embeddings for better understanding in dialogue contexts. Then, multi-task learning between the utterance prediction and the token span prediction is applied to fine-tune for span-based question answering (QA). Our approach is evaluated on the FriendsQA dataset and shows improvements of 3.8% and 1.4% over the two state-of-the-art transformer models, BERT and RoBERTa, respectively.

PDF Abstract ACL 2020 PDF ACL 2020 Abstract

Datasets


  Add Datasets introduced or used in this paper

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering FriendsQA Li and Choi - RoBERTa EM 53.5 # 3
F1 69.6 # 3
Question Answering FriendsQA Li and Choi - BERT EM 46.8 # 5
F1 63.1 # 6

Methods