StereoSet: Measuring stereotypical bias in pretrained language models

20 Apr 2020 Moin Nadeem Anna Bethke Siva Reddy

A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT BENCHMARK
Bias Detection StereoSet XLNet (large) ICAT Score 72.03 # 2
Bias Detection StereoSet GPT-2 (medium) ICAT Score 71.73 # 3
Bias Detection StereoSet BERT (base) ICAT Score 71.21 # 4
Bias Detection StereoSet GPT-2 (large) ICAT Score 70.54 # 5
Bias Detection StereoSet BERT (large) ICAT Score 69.89 # 6
Bias Detection StereoSet RoBERTa (base) ICAT Score 67.50 # 7
Bias Detection StereoSet XLNet (base) ICAT Score 62.10 # 8
Bias Detection StereoSet GPT-2 (small) ICAT Score 72.97 # 1

Methods used in the Paper