Morph Call
Introduced by Mikhailov et al. in Morph Call: Probing Morphosyntactic Content of Multilingual Transformers
Morph Call is a suite of 46 probing tasks for four Indo-European languages that fall under different morphology: Russian, French, English, and German. The tasks are designed to explore the morphosyntactic content of multilingual transformers which is a less studied aspect at the moment.
The tasks are divided into four groups:
- Morphosyntactic Features: probe the encoder for the occurrence of the morphosyntactic properties.
- Masked Token: analogous to Morphosyntactic Features with the exception that the target word is replaced with a tokenizer-specific mask token.
- Morphosyntactic Values: is a group of k-way classification tasks for each feature where k is the number of values that the feature can take.
- Perturbations: tasks test the encoder sensitivity to syntactic and inflectional sentence perturbations.
Probing Methods
- Supervised probing involves training a Logistic Regression classifier to predict a property. The performance is used as a proxy to evaluate the model knowledge.
- Neuron-level Analysis [Durrani et al., 2020] allows retrieving a group of individual neurons that are most relevant to predict a linguistic property.
- Contextual Correlation Analysis [Wu et al., 2020] is a representation-level similarity measure that allows identifying pairs of layers of similar behavior.
Usage
We provide an example of the experiment on Masked Token task (Case, German).
bash
me@my-laptop:~$ python3 probe.py --help
INFO: Showing help with the command 'probe.py -- --help'.
NAME
probe.py - configure the experiment AND perform probing
SYNOPSIS
probe.py <flags>
DESCRIPTION
configure the experiment AND perform probing
FLAGS
--results_path=RESULTS_PATH
Type: Optional[str]
Default: None
path to a folder to store the probing results and the model intermediate activations
--model_architecture=MODEL_ARCHITECTURE
Type: typ...
Default: 'bert multilingual'
--model_is_finetuned=MODEL_IS_FINETUNED
Type: bool
Default: False
if to perform the experiment on the fine-tuned model
--model_finetuned_path=MODEL_FINETUNED_PATH
Type: Optional[str]
Default: None
(only if model_is_finetuned is True) path to store the fine-tuned model
--model_finetuned_config_google_url=MODEL_FINETUNED_CONFIG_GOOGLE_URL
Type: Optional[]
Default: None
(only if model_is_finetuned is True) the url of the fine-tuned model config if to be downloaded
--model_finetuned_model_google_url=MODEL_FINETUNED_MODEL_GOOGLE_URL
Type: Optional[]
Default: None
(only if model_is_finetuned is True) the url of the fine-tuned model weights if to be downloaded
--model_is_random=MODEL_IS_RANDOM
Type: bool
Default: False
if to perform the random initialization of the model
--layers_to_probe=LAYERS_TO_PROBE
Type: List
Default: 'all'
(either "all" or list w. possible numbers from 0 to 11) -- model layers to probe. e.g.: [1, 3, 11], or "all"
--train_n_sentences=TRAIN_N_SENTENCES
Type: int
Default: 1500
number of sentences used to train the probing classifier
--test_n_sentences=TEST_N_SENTENCES
Type: int
Default: 1000
number of sentences used to evaluate the probing classifier
--dev_n_sentences=DEV_N_SENTENCES
Type: int
Default: 0
DEPRECATED
Papers
| Paper | Code | Results | Date | Stars |
|---|
Dataset Loaders
No data loaders found. You can
submit your data loader here.
No data loaders found. You can
submit your data loader here.
