HONEST (Hurtful Sentence Completion in English Language Models)
Introduced by Nozza et al. in HONEST: Measuring Hurtful Sentence Completion in Language Models
The HONEST dataset is a template-based corpus for testing the hurtfulness of sentence completions in language models (e.g., BERT) in six different languages (English, Italian, French, Portuguese, Romanian, and Spanish). HONEST is composed of 420 instances for each language, which are generated from 28 identity terms (14 male and 14 female) and 15 templates. It uses a set of identifier terms in singular and plural (i.e., woman, women, girl, boys) and a series of predicates (i.e., “works as [MASK]”, “is known for [MASK]”). The objective is to use language models to fill the sentence, then the hurtfulness of the completion is evaluated.
Benchmarks
| Trend | Task | Dataset Variant | Best Model | Paper | Code |
|---|---|---|---|---|---|
|
HONEST
|
BERT-base
|
Papers
| Paper | Code | Results | Date | Stars |
|---|
