TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Entity Linking	WiC-TSV	Human	Task 3 Accuracy: all	85.3	# 1
Entity Linking	WiC-TSV	Human	Task 3 Accuracy: general purpose	82.1	# 1
Entity Linking	WiC-TSV	Human	Task 3 Accuracy: domain specific	89.2	# 1
Word Sense Disambiguation	WiC-TSV	FastText	Task 1 Accuracy: all	53.7	# 6
Word Sense Disambiguation	WiC-TSV	FastText	Task 1 Accuracy: general purpose	56.2	# 5
Word Sense Disambiguation	WiC-TSV	FastText	Task 1 Accuracy: domain specific	50.6	# 6
Word Sense Disambiguation	WiC-TSV	FastText	Task 2 Accuracy: all	52.7	# 4
Word Sense Disambiguation	WiC-TSV	FastText	Task 2 Accuracy: general purpose	56.8	# 4
Word Sense Disambiguation	WiC-TSV	FastText	Task 2 Accuracy: domain specific	47.7	# 4
Word Sense Disambiguation	WiC-TSV	FastText	Task 3 Accuracy: all	53.4	# 6
Word Sense Disambiguation	WiC-TSV	FastText	Task 3 Accuracy: general purpose	57.1	# 5
Word Sense Disambiguation	WiC-TSV	FastText	Task 3 Accuracy: domain specific	49.0	# 6
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: all	54.4	# 5
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: general purpose	49.2	# 7
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: domain specific	60.6	# 5
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 2 Accuracy: all	62.8	# 3
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 2 Accuracy: general purpose	57.6	# 3
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 2 Accuracy: domain specific	69.1	# 3
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 3 Accuracy: all	60.5	# 5
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 3 Accuracy: general purpose	54.4	# 6
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 3 Accuracy: domain specific	67.9	# 4
Word Sense Disambiguation	WiC-TSV	All true	Task 1 Accuracy: all	50.8	# 7
Word Sense Disambiguation	WiC-TSV	All true	Task 1 Accuracy: general purpose	53.8	# 6
Word Sense Disambiguation	WiC-TSV	All true	Task 1 Accuracy: domain specific	47.0	# 7
Word Sense Disambiguation	WiC-TSV	All true	Task 2 Accuracy: all	50.8	# 5
Word Sense Disambiguation	WiC-TSV	All true	Task 2 Accuracy: general purpose	53.8	# 5
Word Sense Disambiguation	WiC-TSV	All true	Task 2 Accuracy: domain specific	47.0	# 5
Word Sense Disambiguation	WiC-TSV	All true	Task 3 Accuracy: all	50.8	# 7
Word Sense Disambiguation	WiC-TSV	All true	Task 3 Accuracy: general purpose	53.8	# 7
Word Sense Disambiguation	WiC-TSV	All true	Task 3 Accuracy: domain specific	47.0	# 7
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 1 Accuracy: all	75.3	# 4
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 1 Accuracy: general purpose	73.3	# 4
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 1 Accuracy: domain specific	77.9	# 3
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 2 Accuracy: all	71.7	# 2
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 2 Accuracy: general purpose	68.6	# 1
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 2 Accuracy: domain specific	74.7	# 2
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 3 Accuracy: all	76.6	# 3
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 3 Accuracy: general purpose	73.5	# 3
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 3 Accuracy: domain specific	80.4	# 3
Word Sense Disambiguation	WiC-TSV	Human	Task 3 Accuracy: all	85.3	# 1
Word Sense Disambiguation	WiC-TSV	Human	Task 3 Accuracy: general purpose	82.1	# 1
Word Sense Disambiguation	WiC-TSV	Human	Task 3 Accuracy: domain specific	89.2	# 1
Entity Linking	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: all	54.4	# 5
Entity Linking	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: general purpose	49.2	# 7
Entity Linking	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: domain specific	60.6	# 5
Entity Linking	WiC-TSV	Unsupervised Bert	Task 2 Accuracy: all	62.8	# 3
Entity Linking	WiC-TSV	Unsupervised Bert	Task 2 Accuracy: general purpose	57.6	# 3
Entity Linking	WiC-TSV	Unsupervised Bert	Task 2 Accuracy: domain specific	69.1	# 3
Entity Linking	WiC-TSV	Unsupervised Bert	Task 3 Accuracy: all	60.5	# 5
Entity Linking	WiC-TSV	Unsupervised Bert	Task 3 Accuracy: general purpose	54.4	# 6
Entity Linking	WiC-TSV	Unsupervised Bert	Task 3 Accuracy: domain specific	67.9	# 4
Entity Linking	WiC-TSV	All true	Task 1 Accuracy: all	50.8	# 7
Entity Linking	WiC-TSV	All true	Task 1 Accuracy: general purpose	53.8	# 6
Entity Linking	WiC-TSV	All true	Task 1 Accuracy: domain specific	47.0	# 7
Entity Linking	WiC-TSV	All true	Task 2 Accuracy: all	50.8	# 5
Entity Linking	WiC-TSV	All true	Task 2 Accuracy: general purpose	53.8	# 5
Entity Linking	WiC-TSV	All true	Task 2 Accuracy: domain specific	47.0	# 5
Entity Linking	WiC-TSV	All true	Task 3 Accuracy: all	50.8	# 7
Entity Linking	WiC-TSV	All true	Task 3 Accuracy: general purpose	53.8	# 7
Entity Linking	WiC-TSV	All true	Task 3 Accuracy: domain specific	47.0	# 7
Entity Linking	WiC-TSV	FastText	Task 1 Accuracy: all	53.7	# 6
Entity Linking	WiC-TSV	FastText	Task 1 Accuracy: general purpose	56.2	# 5
Entity Linking	WiC-TSV	FastText	Task 1 Accuracy: domain specific	50.6	# 6
Entity Linking	WiC-TSV	FastText	Task 2 Accuracy: all	52.7	# 4
Entity Linking	WiC-TSV	FastText	Task 2 Accuracy: general purpose	56.8	# 4
Entity Linking	WiC-TSV	FastText	Task 2 Accuracy: domain specific	47.7	# 4
Entity Linking	WiC-TSV	FastText	Task 3 Accuracy: all	53.4	# 6
Entity Linking	WiC-TSV	FastText	Task 3 Accuracy: general purpose	57.1	# 5
Entity Linking	WiC-TSV	FastText	Task 3 Accuracy: domain specific	49.0	# 6
Entity Linking	WiC-TSV	Bert-base	Task 1 Accuracy: all	75.3	# 4
Entity Linking	WiC-TSV	Bert-base	Task 1 Accuracy: general purpose	73.3	# 4
Entity Linking	WiC-TSV	Bert-base	Task 1 Accuracy: domain specific	77.9	# 3
Entity Linking	WiC-TSV	Bert-base	Task 2 Accuracy: all	71.7	# 2
Entity Linking	WiC-TSV	Bert-base	Task 2 Accuracy: general purpose	68.6	# 1
Entity Linking	WiC-TSV	Bert-base	Task 2 Accuracy: domain specific	74.7	# 2
Entity Linking	WiC-TSV	Bert-base	Task 3 Accuracy: all	76.6	# 3
Entity Linking	WiC-TSV	Bert-base	Task 3 Accuracy: general purpose	73.5	# 3
Entity Linking	WiC-TSV	Bert-base	Task 3 Accuracy: domain specific	80.4	# 3

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/wic-tsv-an-evaluation-benchmark-for-target/entity-linking-on-wic-tsv)](https://paperswithcode.com/sota/entity-linking-on-wic-tsv?p=wic-tsv-an-evaluation-benchmark-for-target)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/wic-tsv-an-evaluation-benchmark-for-target/word-sense-disambiguation-on-wic-tsv)](https://paperswithcode.com/sota/word-sense-disambiguation-on-wic-tsv?p=wic-tsv-an-evaluation-benchmark-for-target)`

WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

EACL 2021 · Anna Breit, Artem Revenko, Kiamehr Rezaee, Mohammad Taher Pilehvar, Jose Camacho-Collados ·

We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense Disambiguation. More specifically, we introduce a framework for Target Sense Verification of Words in Context which grounds its uniqueness in the formulation as a binary classification task thus being independent of external sense inventories, and the coverage of various domains. This makes the dataset highly flexible for the evaluation of a diverse set of models and systems in and across domains. WiC-TSV provides three different evaluation settings, depending on the input signals provided to the model. We set baseline performance on the dataset using state-of-the-art language models. Experimental results show that even though these models can perform decently on the task, there remains a gap between machine and human performance, especially in out-of-domain settings. WiC-TSV data is available at https://competitions.codalab.org/competitions/23683

PDF Abstract EACL 2021 PDF EACL 2021 Abstract

Code

Add Remove Mark official

semantic-web-company/wic-tsv official

Tasks

Add Remove

Binary Classification

Entity Linking

Word Sense Disambiguation

Datasets

Introduced in the Paper:

WiC-TSV

Used in the Paper:

SuperGLUE WiC

Results from the Paper

Edit

Ranked #1 on Entity Linking on WiC-TSV (Task 3 Accuracy: all metric)

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Entity Linking	WiC-TSV	Human	Task 3 Accuracy: all	85.3	# 1	Compare
			Task 3 Accuracy: general purpose	82.1	# 1	Compare
			Task 3 Accuracy: domain specific	89.2	# 1	Compare
Word Sense Disambiguation	WiC-TSV	FastText	Task 1 Accuracy: all	53.7	# 6	Compare
			Task 1 Accuracy: general purpose	56.2	# 5	Compare
			Task 1 Accuracy: domain specific	50.6	# 6	Compare
			Task 2 Accuracy: all	52.7	# 4	Compare
			Task 2 Accuracy: general purpose	56.8	# 4	Compare
			Task 2 Accuracy: domain specific	47.7	# 4	Compare
			Task 3 Accuracy: all	53.4	# 6	Compare
			Task 3 Accuracy: general purpose	57.1	# 5	Compare
			Task 3 Accuracy: domain specific	49.0	# 6	Compare
Word Sense Disambiguation	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: all	54.4	# 5	Compare
			Task 1 Accuracy: general purpose	49.2	# 7	Compare
			Task 1 Accuracy: domain specific	60.6	# 5	Compare
			Task 2 Accuracy: all	62.8	# 3	Compare
			Task 2 Accuracy: general purpose	57.6	# 3	Compare
			Task 2 Accuracy: domain specific	69.1	# 3	Compare
			Task 3 Accuracy: all	60.5	# 5	Compare
			Task 3 Accuracy: general purpose	54.4	# 6	Compare
			Task 3 Accuracy: domain specific	67.9	# 4	Compare
Word Sense Disambiguation	WiC-TSV	All true	Task 1 Accuracy: all	50.8	# 7	Compare
			Task 1 Accuracy: general purpose	53.8	# 6	Compare
			Task 1 Accuracy: domain specific	47.0	# 7	Compare
			Task 2 Accuracy: all	50.8	# 5	Compare
			Task 2 Accuracy: general purpose	53.8	# 5	Compare
			Task 2 Accuracy: domain specific	47.0	# 5	Compare
			Task 3 Accuracy: all	50.8	# 7	Compare
			Task 3 Accuracy: general purpose	53.8	# 7	Compare
			Task 3 Accuracy: domain specific	47.0	# 7	Compare
Word Sense Disambiguation	WiC-TSV	Bert-base	Task 1 Accuracy: all	75.3	# 4	Compare
			Task 1 Accuracy: general purpose	73.3	# 4	Compare
			Task 1 Accuracy: domain specific	77.9	# 3	Compare
			Task 2 Accuracy: all	71.7	# 2	Compare
			Task 2 Accuracy: general purpose	68.6	# 1	Compare
			Task 2 Accuracy: domain specific	74.7	# 2	Compare
			Task 3 Accuracy: all	76.6	# 3	Compare
			Task 3 Accuracy: general purpose	73.5	# 3	Compare
			Task 3 Accuracy: domain specific	80.4	# 3	Compare
Word Sense Disambiguation	WiC-TSV	Human	Task 3 Accuracy: all	85.3	# 1	Compare
			Task 3 Accuracy: general purpose	82.1	# 1	Compare
			Task 3 Accuracy: domain specific	89.2	# 1	Compare
Entity Linking	WiC-TSV	Unsupervised Bert	Task 1 Accuracy: all	54.4	# 5	Compare
			Task 1 Accuracy: general purpose	49.2	# 7	Compare
			Task 1 Accuracy: domain specific	60.6	# 5	Compare
			Task 2 Accuracy: all	62.8	# 3	Compare
			Task 2 Accuracy: general purpose	57.6	# 3	Compare
			Task 2 Accuracy: domain specific	69.1	# 3	Compare
			Task 3 Accuracy: all	60.5	# 5	Compare
			Task 3 Accuracy: general purpose	54.4	# 6	Compare
			Task 3 Accuracy: domain specific	67.9	# 4	Compare
Entity Linking	WiC-TSV	All true	Task 1 Accuracy: all	50.8	# 7	Compare
			Task 1 Accuracy: general purpose	53.8	# 6	Compare
			Task 1 Accuracy: domain specific	47.0	# 7	Compare
			Task 2 Accuracy: all	50.8	# 5	Compare
			Task 2 Accuracy: general purpose	53.8	# 5	Compare
			Task 2 Accuracy: domain specific	47.0	# 5	Compare
			Task 3 Accuracy: all	50.8	# 7	Compare
			Task 3 Accuracy: general purpose	53.8	# 7	Compare
			Task 3 Accuracy: domain specific	47.0	# 7	Compare
Entity Linking	WiC-TSV	FastText	Task 1 Accuracy: all	53.7	# 6	Compare
			Task 1 Accuracy: general purpose	56.2	# 5	Compare
			Task 1 Accuracy: domain specific	50.6	# 6	Compare
			Task 2 Accuracy: all	52.7	# 4	Compare
			Task 2 Accuracy: general purpose	56.8	# 4	Compare
			Task 2 Accuracy: domain specific	47.7	# 4	Compare
			Task 3 Accuracy: all	53.4	# 6	Compare
			Task 3 Accuracy: general purpose	57.1	# 5	Compare
			Task 3 Accuracy: domain specific	49.0	# 6	Compare
Entity Linking	WiC-TSV	Bert-base	Task 1 Accuracy: all	75.3	# 4	Compare
			Task 1 Accuracy: general purpose	73.3	# 4	Compare
			Task 1 Accuracy: domain specific	77.9	# 3	Compare
			Task 2 Accuracy: all	71.7	# 2	Compare
			Task 2 Accuracy: general purpose	68.6	# 1	Compare
			Task 2 Accuracy: domain specific	74.7	# 2	Compare
			Task 3 Accuracy: all	76.6	# 3	Compare
			Task 3 Accuracy: general purpose	73.5	# 3	Compare
			Task 3 Accuracy: domain specific	80.4	# 3	Compare

Methods

Add Remove

Adam • Attention Dropout • BERT • Dense Connections • Dropout • GELU • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Weight Decay • WordPiece

Edit Social Preview

WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove