CliCR: A Dataset of Clinical Case Reports for Machine Reading Comprehension

NAACL 2018  ·  Simon Šuster, Walter Daelemans ·

We present a new dataset for machine comprehension in the medical domain. Our dataset uses clinical case reports with around 100,000 gap-filling queries about these cases. We apply several baselines and state-of-the-art neural readers to the dataset, and observe a considerable gap in performance (20% F1) between the best human and machine readers. We analyze the skills required for successful answering and show how reader performance varies depending on the applicable skills. We find that inferences using domain knowledge and object tracking are the most frequently required skills, and that recognizing omitted information and spatio-temporal reasoning are the most difficult for the machines.

PDF Abstract NAACL 2018 PDF NAACL 2018 Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Question Answering CliCR Gated-Attention Reader F1 33.9 # 1
Question Answering CliCR Stanford Attentive Reader F1 27.2 # 2

Methods


No methods listed for this paper. Add relevant methods here